Community
Participate
Working Groups
This is a fork of bug 45767. The problem is this: When a classpath container changes its value, the .project file is modified. Since classpaths are now dynamic but project references are not, this means end users constantly have outgoing changes on their .project files. Example (from PDE): - Plugin B requires Plugin A - Plugin B is being developed in a workspace, compiling against Plugin A as external JARs Observe: .classpath has the PDE container, .project has no project references - User decides to checkout Plugin A from repository to hack it Observe: .project now has a new project reference. Since different users on a team will have different self-hosting styles, they now have incompatible .project files.
I discussed this problem with Jeem yesterday and he had the following idea: Classpath containers essentially delegate responsibility for computation of a portion of a project's classpath to a third party. However, JDT has not delegated the responsibility of updating the project references to this third party. This means that JDT has to "guess" based on the resolved classpath container values what the appropriate project references are. In other words, JDT is guessing what the project references are based on incomplete knowledge of that project's dependencies. A solution to this problem would be to defer responsibility for updating project references to the provider of the classpath container. This breaks down as follows: - JDT continues to add project references for all classpath entries *except* classpath containers. It makes the assumption that whoever provided the container also has responsibility to add any necessary project references. - PDE is now responsible for adding project references when classpath containers are used. It does this quite simply by adding a project reference for every plugin import in the plugin manifest. The set of plugin imports is "static" regardless of a user's self-hosting style, so the set of project references will not change either. When plug-in imports are added or removed, it updates the project references accordingly (just as I assume it is currently updating the classpath container values). - This will work because project references are "soft" by design. The platform does not care of a referenced project actually exists in the workspace. If the reference exists, it will be added to the build order appropriately. If the reference does not exist in the workspace, it has no impact on the build order. The build order is automatically updated whenever projects are created, deleted, opened, or closed. How does that sound?
How would this change the net result ? Requiring classpath container providers to be aware of the proper build order is likely not an obvious thing for them, and clients could argue that it should be done in JDT Core as it is today. The project references are really an implementation detail, I'd rather hide to clients. I would rather be in favor of a dynamic build order obtained by asking builder contributors.
I just had another realization. PDE knows the complete set of project references for a plugin from the plugin manifest. JDT can infer some dynamic subset of these references by looking at the values of the container. Since what JDT sees is simply a subset, then both PDE and JDT can add project references. Here's a revision of my previous proposal. - PDE always adds project references for all required plugins. It does this quite simply by adding a project reference for every plugin import in the plugin manifest. This set is essentially static so the .project file will never change. - JDT continues to add project references for all classpath entries including classpath containers. I assume it already checks if a project reference exists before adding it. This means the .project will never be dirtied when the user changes self-hosting styles (causing classpath container values to change).
I need a clarification: >PDE always adds project references for all required plugins. It does this >quite simply by adding a project reference for every plugin import in the >plugin manifest. I'm assuming you are talking about adding project references to the .project file. As you know, adding project references to the classpath will cause an error if the project does not exist in the workspace. Also direct imports is not the only thing we have to worry about here, we also have to account for reexported imports. For example, plug-in A imports org.eclipse.ui and is getting org.eclipse.ui.workbench for free because org.eclipse.ui reexports it. If org.eclipse.ui is external and org.eclipse.ui.workbench is in the workspace, a project reference to org.eclipse.ui.workbench would have to be made. Either way, I have to echo Philippe's sentiment that I don't understand why all of a sudden PDE has to worry about .project files. I was unaware of their use for a year and a half and I prefer if we kept it that way ;-)
I have to echo Wassim's sentiment who echoed Philippe's sentiment that project references is not something in the JDT contract from PDE's point of view. As far as we know, we are modifying project classpath. We simply state that this project depends on a list of other projects. We have no idea what project references are used for. Yes, thanks to you we now know that they are used for build order :-) but that is not what we are doing in a classpath container. We are simply computing classpath entries dynamically and the fact that that changes .project file is an implementation side-effect that we learned about only recently. We would hate to all of the sudden be responsible for build order. Philippe hates it as well because he will be on the hook every time the build fails due to the wrong build order even though PDE is to blame.
Answering comment #3, JDT is never touching the project description is the required set of project references (from JDT standpoint) is a subset of existing ones. So whatever occurs upstream, JDT should be resilient. I think most of the troubles in touching .project files is related to the fact PDE container slightly changed the project reference set by including fragments in certain cases (bug 45767). If bug 45767 is addressed, then there is no need for PDE to compute project references if JDT does it underneath, with the same net result. What is the problem with a notion of a dynamic build order ?
Philippe, JDT will overwrite the .project even when bug 45767 gets fixed. Here is a scenario: plug-in X requires org.eclipse.ui, which in turns imports and re-exports org.elipse.ui.workbench, org.eclipse.jface and org.eclipse.swt. When all these plug-ins are projects in the workspace: 1. The .project file of plug-in X contains only a reference to project org.eclipse.ui. Good. 2. The classpath of plug-in project A only contains a project reference to project org.eclipse.ui. All good! Suppose now that the developer closes projecgt org.eclipse.ui: PDE will dynamically recompute the classpath of plug-in X. This time, the classpath of X will contain a reference to the external JAR of org.eclipse.ui (since the workspace version is now closed). The classpath of X must now also contain direct references to projects org.eclipse.ui.workbench and org.eclipse.swt. These two new project references will cause JDT to overwrite the .project file. The .project file is thus very susceptible to workspace changes, while the .classpath file is unchanged. As you can see, no fragments were used to illustrate this example. While a dynamic build order is the best solution for many reasons that go beyond this defect, I was persuaded by Jim that PDE should not wait for the big build order issue to get resolved if this particular problem can be avoided with the current build mechanism unchanged. Here is the suggestion: 1. JDT Core should not update the .project file in the case of classpath containers. 2. The PDE builder will be the one responsible for updating the .project file when necessary. It is only necessary to update it when the plug-in's list of dependencies changes, and *not* every time the workspace state changes as it is the case today. 3. The list of projects in the .project of the plug-in project are references to all the direct imports of the plug-in and all the reexported imports of its direct imports. This way the list of projects is complete, and no matter what projects are added/deleted from the workspace, the .project does not need to be modified.
My comment #6 should have mentionned that the set of project references in a self-hosted workspace with all plugins as projects. I was bothered by the introduced fragments. As you correctly pointed out, further differences occurs once users start using the flexibility the PDE container provides. From a JDT standpoint, it doesn't make sense to require our clients to prepare the proper build order. What about other container contributors ? This is a breaking API change. If PDE is willing to contribute more project references than what JDT would induce, this is a fair workaround, but not a fix to the original problem. Another reason for discouraging such practice is that there is no mechanism to avoid project references from stepping on each other. This is not a happy story on orthogonal natures, but if multiple tools start playing with Java references we may be in trouble. Again dynamic references seem the only hope.
I was thinking that JDT would not be required to make any changes. This works fine when required projects are imported, since JDT attempts to add a project reference that already exists, and this case is optimized out. However, when a required project is removed from the workspace (to be replaced by an external jar), JDT is removing the project reference.
I second Philippe's sentiment. To ask classpath container providers to change project references is breaking encapsulation. JDT is updating project references is an internal implementation detail (a consequence of classpath change). As far as we as customers are concerned, we only change JDT classpath. Bottom line: 1) Use of project references is an implementation detail that classpath containers should not know about 2) Project references are used for build order, which is something that we don't care about, but JDT builder does, and should remain in its control. Even though it is agaist our short-term interests when it comes to dynamic classpaths, I have a really bad feeling about PDE going down this path and would rather sit it out until Platorm Core and JDT Core find a better (dynamic) way of managing build order.
On the other hand, if we all agree that this is a short-term solution until a better one is found, we can do it in PDE for the sake of dynamic classpaths faster adoption.
I'm all for it. The only thing that JDT needs to, as John A. points out, is to not remove project references upon the deletion of a project. I'm not sure how they feel about this.
Perhaps a backward-compatible solution would be to make the classpath initializer implement a certain interface to indicate that it will handle project references. Without it, do what you do today. Otherwise, let the initializer handle project references.
Yet another alternative would be to add a boolean attribute 'manage- references' in the classpath container intializer extension point schema. The attribute would be false by default, and PDE can set its own to true.
Out of curiosity, is it true that PDE does not care about the build order? PDE does register builders... do they not care if they are run out of order with respect to plugin dependencies?
I was thinking that this should be handled at the PDE builder level to pre- empt the Java builder, and not at the container initializer level. This would also be the best point in time to determine if the list of dependencies has changed and whether the .project file needs to be modified or not. To answer John's question, we do not care about build order.
Re: comment #10 Project build order is defined in Platform core, not by JDT. Although project references and default build order is far from ideal, it is currently *the* way we establish the build order. PDE knows about interdependencies between plug-in projects in the workspace, inferred from interdependencies between plug-ins stated in the manifest. This makes project references something that PDE is in a unique position to contribute to. Just as PDE can feed JDT the classpath entries it needs to properly compile code in the plug-in, so too it can feed Platform Core with the project references that ensure multiple plug-in projects get built in a reasonable relative order. I don't think PDE should be reluctant about going down this path. When and if we have an improved story for how to order the building of projects in workspace, PDE would naturally move to supporting that new story.
... and the reason we don't care is because our builders do not produce new resources - they are simply validators. Each manifest can be validated in isolation - validation order is not important.
Removing Dejan from cc as he is a trouble maker :-)
Whatever PDE decides so as to workaround the platform issue, JDT must honour a proper Java build order. PDE is implemented in term of Java projects, thus it cannot bypass the fact that JDT computes some reasonable build order based on Java classpath information. Re: comment #12. If a project is removed, we must remove a project references, so as not to leak references. Obsolete references may induce false platform diagnosis such as builder order cycles ? Re: comment #13. This isn't an option as we would have to live forever with this (based on previous experience). Besides, why would JDT provide a new intermediate API to cover for a missing support in the platform. There are enough evidences that static project references do not scale, for these to be replaced by a better mechanism; and then I'd be more than happy to jump on it.
If JDT continues removing project references upon project deletions, and there seems to be a good reason for it, then this workaround will never work... Should we even pursue this?
re: comment #21 - It would not be a problem if JDT never updated project references based on info obtained from classpath containers, as proposed in comment #1.
Feels inconsistent to me that JDT would not help on certain classpath entries. I would be more easily convinced to do nothing at all any longer <g>.
If JDT/Core doesn't do anything anymore will this mean that JDT/UI has to maintain the project references. This seems strange to me since JDT/UI doesn't want to know anything about the relationship between class path and project references.
Dirk, there is no suggestion for JDT UI to do anything here. We're debating whether it should be done by JDT core or by clients of the container API (PDE in this case).
Just to note that there are other implementors of class path containers as well.
WSDD has some in particular. Dirk, to clarify, I have no plan to remove any support from JDT Core with respect to project references (i.e. you don't have to revive the old project references panel in project creation wizard, yet <g>).
*** Bug 48014 has been marked as a duplicate of this bug. ***
How about a dirty solution that would solve the .project file changing, by simply moving the <projects> element (where the changes happen) out of the .project file into a .buildorder file or something. When using containers this file will get changed instead of the .project file, and users can add the .buildorder file to the .cvsignore file to stop being told of changes. In these cases, the issue would be how is this .buildorder file created when you first check out the project from CVS. Would the container be able to create them automatically in both the development workspace and the runtime workspace?
I think we should consider anything but dynamic project references contributed by registered builders. Any static approach is doomed in the same way, even if you separate concerns even more...
*** Bug 54235 has been marked as a duplicate of this bug. ***
There's a lot of deep discussion on this "bug" .... but I'm wondering if a solution to the pratical problem original mentioned would be to provide a filter on the CVS synchronize view (so .project could be filtered out)? I know by itself this wouldn't solve everything (for example, would still be told there were unsynchronized changes when tagging projects) but seeing all the "changed projects" in the synchronized view really is tiring. And, actually, its nearly caused me to miss some project's out going synchs, because I have so many (like 20) its easy to overlook one. Just thought I'd ask, since I'd hate to see the practical problem not addressed at all.
In addition to the syncrhonize view, using the label decorators in the other viewers (navigator, package explorer, etc). The simple thing would be to ignore .project if it is in the .cvsignore file. The problem with this is that then users my miss other important changes to .project, such as a nature or builder added.
Since many people have been asking lately, I will give a summary of the current position on this bug. Last fall I investigated changes that could be made to the platform core builder infrastructure that would solve this problem. The result of this investigation was recorded at: http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/platform-core-home/documents/eclipse3_builder_issues.html The only solution that attacked the root of the problem was S1, and it required breaking API changes for which there was no solid backward compatibility story. This kind of breakage to all existing builders didn't feel an acceptable tradeoff for the magnitude of the problem that it solved. S2 solved only the .project problem (by making build paths dynamically computed). Nobody was entirely happy with that solution, since it required a wad of new API that solved only the "dirty .project" problem but didn't address other long-standing concerns about the platform build infrastructure. Jeem and I made a counter-proposal that the clients who were dynamically computing the build path should also be responsible (or could optionally assume responsibility) for updating project references (comment #1 and comment #17). This was based on the observation that the project references are actually very static - a plug-in's pre-requisites don't actually change when they switch from being internal to external and vice-versa. Only the client of the classpath container (PDE in this case), knows what the static, stable set of plugin project dependencies are, so it is the only one that has any hope of computing them. Since project references currently have a strong relationship with build order, this proposal was rejected because it moved responsibility for maintaining project references away from the plug-in that cares about builder order (JDT), and because it didn't address unrelated long-standing concerns about the platform build infrastructure. Having come to this stalement, I don't know how to proceed. David's recent suggestion (comment #32), while well-intentioned, is a no-go. It defeats the purpose of putting the .project in the project content area in the first place. The .project contains important information apart from the project references, so hiding it prevents those important changes from being shared with a team. The only other suggestion I have heard is to move the project references out of the .project file, such as in the new preference store which has support for project-level preference scopes. I'm willing to investigate this, but I'm not that happy with it. This has implications for builders that rely on the platform to store project references. JDT isn't affected because they duplicate the project reference information in the .classpath file, but this might not be true for all builders. Hiding the project references could result in some builders not obtaining the desired build order.
Thanks for the summary and update, John. Just to clarify, my suggest was for a user settable filter (just like on package explorer's filter dialog) ... so, would at least be "use at your own risk" sort of thing. Frankly, once we get these files as we like them, we ignore their synch state anyway ... its just we have to ignore them "manually". Hope that doesn't burst any bubbles, but the alternative is that some on the team always get a "wrong" one, or everyone on the team has to have the exact same environment. Maybe we are not a typical team, but we are all intimately aware of these files, their contents, and don't blush to occasionally edit them by hand. Just consider this a "report from the trenches".
What about simply letting builders declare recommendation for a build order ? This is what I mentionned as dynamic references. The build manager would gather all recommendations either static or dynamic, and compute a global build order as it does today. This allows tools inferring build orders (as we do) to avoid polluting project description with generated contextual information.
I am not convinced that we have come to a stalemate. Having heard all the arguments and examined the conclusions of John's investigation, I still don't see a compelling reason or a realistic use case to not take the approach Philippe is suggesting in comments 2, 8, 30 and 36. Dynamic build order seems the only way to go and JDT is more than willing to be in control of that for Java projects. As for the other builders in the SDK, namely the PDE builders, we are more than happy with any order JDT chooses, since build order really has no effect whatsoever on the validation that we do.
I have acquiesced. Core will introduce dynamic project references that are not stored in the .project file. More details to follow shortly, I'm just writing some more tests to make sure it all works.
The only corner case I could come up with is a situation where one copy/paste a project folder into a workspace (no metadata), then creates a Java project on it. Unsure whether we will consider this as a classpath change (and thus issue dynamic references in reaction). Need to be tested.
For reference regarding comment #38, http://dictionary.reference.com/search?q=acquiesce John comes up with pretty good ones :-)
Summary of API changes in core: New methods: IProjectDescription.get/setDynamicReferences. Dynamic references behave just like ordinary references, except they are not stored in the .project file. The intent is that references that are likely to change based on local workspace conditions should be stored as dynamic references. References that are not workspace-dependent, or that need to be shared between teams for any reason, should use the old (static) project references. For example, JDT would store references resolved from dynamic classpath variables as dynamic project references, but could leave fixed classpath project dependencies as normal project references. Other API changes: IProject.getReferencedProjects() - This method will return the union of static and dynamic references, with duplicates omitted. Clients that are interested in knowing a project's references, but don't care to distinguish dynamic from non-dynamic, should use this method. IProject.getReferencingProjects() - Changed to compute in-references based on both static and dyanmic references IWorkspace.computeProjectOrder - Changed to compute topological order using both static and dynamic references. IProject.setDescription - This method is now "smart" about only writing the .project file if the contents of the .project file have actually changed. I.e., changes to the location or dynamic references will not dirty the .project file. These changes are available in HEAD.
This is a great day for the selfhosting community :-).
Now that the platform core changes have been released, it just remains for JDT core to start using dynamic references for classpath containers. Moving this bug to JDT core.
*** Bug 54851 has been marked as a duplicate of this bug. ***
Fixed SetClasspathOperation.updateProjectReferencesIfNecessary() to use dynamic project references. Changed ClasspathInitializer.testContainerInitializer6() to reflect this change (i.e. the .project file is no longer changed)
Reopen to assign
Marking as fixed (sorry for the mass mailing)
Verified for 3.0 using build I200403240800.
