Summary: | [Builder] Too many dependents found when incrementally recompiling | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Eclipse Project] JDT | Reporter: | Philipe Mulet <philippe_mulet> | ||||||||
Component: | Core | Assignee: | Kent Johnson <kent_johnson> | ||||||||
Status: | VERIFIED FIXED | QA Contact: | |||||||||
Severity: | normal | ||||||||||
Priority: | P3 | CC: | john.arthorne, John_Wiegand, tjbishop | ||||||||
Version: | 2.0 | ||||||||||
Target Milestone: | 2.1.3 | ||||||||||
Hardware: | PC | ||||||||||
OS: | Windows 2000 | ||||||||||
Whiteboard: | |||||||||||
Attachments: |
|
Description
Philipe Mulet
2002-04-18 08:55:41 EDT
This works as intended. Turn on AutoBuild on & this case does not happen. Would still be nice to support. Post 2.0 maybe We could improve the incremental workspace scenario as follows: A transient set of structurally changed files could be carried during a build action on a per project basis. This list would go away at the end of a build iteration (same kind of approach as our global problem counter). If the list is there, then its info would take precedence over a binary delta. Thus it would optimize incremental compilation of a workspace, and if only manually building one project at a time, then there would be no change since we don't want to get into the business of keeping track of the reduced set of structurally changed files. The only counter argument to this would be that we may not scale with other builders contributing separate .class file changes. In which case, instead of remembering the structurally changed file, we could instead record those which haven't structurally changed, so that only these would be ignored from the binary delta in dependent projects. This should could also benefit when autobuild is on, since an atomic operation can modify several resources at once (refactoring, checkout or some other builder running ahead). Testcase: project A +- pa/ +- X.java public class X { Y y; } +- Y.java public class Y {} project B prereqs A +- pb/ +- Z.java public class Z { pa.X x; } project C prereqs A +- pc/ +- T.java public class T { pa.X x; } Touch X.java (add comment) and add a slot to Y.java Perform an incremental build. Build trace: Starting build of A @ Fri Oct 31 12:46:48 CET 2003 Found source delta for: A Clearing last state : State for A (#7 @ Fri Oct 31 12:46:14 CET 2003) INCREMENTAL build Compile this changed source file pa/X.java Compile this changed source file pa/Y.java About to compile pa/X.java About to compile pa/Y.java Writing changed class file X.class Type has structural changes pa/Y will look for dependents of Y in pa Writing changed class file Y.class Found match in pa to Y Found match in pa to Y Recording new state : State for A (#8 @ Fri Oct 31 12:46:48 CET 2003) Finished build of A @ Fri Oct 31 12:46:48 CET 2003 Starting build of B @ Fri Oct 31 12:46:48 CET 2003 Found binary delta for: A Clearing last state : State for B (#7 @ Fri Oct 31 12:37:51 CET 2003) INCREMENTAL build Found changed class file pa/X will look for dependents of X in pa Found changed class file pa/Y will look for dependents of Y in pa Found match in pa to X adding affected source file pb/Z.java About to compile pb/Z.java Writing changed class file Z.class Recording new state : State for B (#8 @ Fri Oct 31 12:37:51 CET 2003) Finished build of B @ Fri Oct 31 12:46:48 CET 2003 Starting build of C @ Fri Oct 31 12:46:48 CET 2003 Found binary delta for: A Clearing last state : State for C (#8 @ Fri Oct 31 12:38:14 CET 2003) INCREMENTAL build Found changed class file pa/X will look for dependents of X in pa Found changed class file pa/Y will look for dependents of Y in pa Found match in pa to X adding affected source file pc/T.java About to compile pc/T.java Writing changed class file T.class Recording new state : State for C (#9 @ Fri Oct 31 12:38:14 CET 2003) Finished build of C @ Fri Oct 31 12:46:48 CET 2003 With suggested fix, build trace would look like: Starting build of A @ Fri Oct 31 12:52:29 CET 2003 Found source delta for: A Clearing last state : State for A (#8 @ Fri Oct 31 12:46:48 CET 2003) INCREMENTAL build Compile this changed source file pa/X.java Compile this changed source file pa/Y.java About to compile pa/X.java About to compile pa/Y.java Writing changed class file X.class Type has structural changes pa/Y will look for dependents of Y in pa Writing changed class file Y.class Found match in pa to Y Found match in pa to Y Recording new state : State for A (#9 @ Fri Oct 31 12:52:29 CET 2003) Finished build of A @ Fri Oct 31 12:52:29 CET 2003 Starting build of B @ Fri Oct 31 12:52:29 CET 2003 Found binary delta for: A Clearing last state : State for B (#8 @ Fri Oct 31 12:37:51 CET 2003) INCREMENTAL build Skipped over unchanged class file pa/X Found changed class file pa/Y will look for dependents of Y in pa Recording new state : State for B (#9 @ Fri Oct 31 12:37:51 CET 2003) Finished build of B @ Fri Oct 31 12:52:29 CET 2003 Starting build of C @ Fri Oct 31 12:52:29 CET 2003 Found binary delta for: A Clearing last state : State for C (#9 @ Fri Oct 31 12:38:14 CET 2003) INCREMENTAL build Skipped over unchanged class file pa/X Found changed class file pa/Y will look for dependents of Y in pa Recording new state : State for C (#10 @ Fri Oct 31 12:38:14 CET 2003) Finished build of C @ Fri Oct 31 12:52:29 CET 2003 Created attachment 6616 [details]
Suggested fix
Created attachment 6617 [details]
better patch
Better fix since it ensures flushing the list prior to adding to it, in case of several incremental loops (within a given project). e.g.: - non-structural change (loop1) - structural change (loop2) Though it doesn't deal with: - structural change (loop1) - non-structural change (loop2) Should rather remember both structural/non-structural changes to properly react. Created attachment 6618 [details]
even better patch
This version records both structural changes and non-structural changes, so as
to deal properly with incremental project loops.
Note that it will work fine in presence of build cycles, since the deltas will contain less and less changes, thus when intersected with the binary changes information, it will not cause any harm. The only situation where we may still be in trouble is the case where even though we know we did not change a file structurally, some other builder may have performed on it after we are done, and we would incorrectly think it hasn't changed, though we have no clue. This case would require to remember a timestamp (on the build state) when the incremental project builder is finished, so that when considering the binary delta, it could notice that the timestamp on the file is more recent than the build state information (BinaryChangeNatures), and thus still consider the binary delta for triggering dependents recompilation. Reopening for consideration The tricky scenario to test with is: Project A has 2 builders (the Java builder & another builder which optimizes .class files). Project B depends on A & has 1 builder. Turn off Auto-build and make structural changes to types in A, followed by a manual build of A. Now make small non-structural changes to the same types in A and hit crtl-B. Does B see the structural changes from A? Or do the types appear as non- structural changes. If they show up as structural changes, do they still if the second builder on A makes structural changes to types that had non-structural changes? I think to solve this scenario, we need to only optimize a build cycle which follows a complete project build cycle, and not manual single project builds. We would need Core/BuildManager to record the type of the last build cycle so we can query it. Also we need to do a timestamp compare on each non-structural changed .class file so we can detect if another builder has replaced it after the Java builder wrote it out. Agreed. CC'ing JohnA to get some information about feasability of this new platform API ? Side-question: we may want to backport this to 2.1.x, thus we would need the API addition there too... Can you clarify what core API you're looking for? Whether it's a project build versus a workspace build, or manual versus auto? As our (Java) builder runs, we want to be able to know whether it is a full- workspace build or not. Ideally, we would like to be able to query what was the previous build action, but the current one may do the trick, we just need to find a way to store this information consistently. Side question: do we have a hook we can use before the build starts and after the build ends ? Are the PRE_AUTO_BUILD/POST_AUTO_BUILD events addressing this ? I have the impression that these events only occur when the build manager is driving the build process (as opposed to using IProject#build(...) directly). Actually, instead of knowing the nature of a build action, we would rather need the list of projects which are going to be built together. We could figure whether this is the entire workspace contents or not. Moreover, we would want to perform our optimization each time the same set of projects is built over and over again. No, we need to know about the previous build cycle. The current one is not important. If the previous build cycle did not successfully build every project, then we can not optimize this build cycle, regardless of how many projects are being built. We must know that all dependents have seen all the changes. A list of projects is overkill. We just need a flag that tells us its possible to optimize the current build. We need the project list so as to notice we are the first or the last project being built (we cannot safely rely on the change notification since IProject#build is API), and at some point, we may need to support building working sets... Released fix for the most common case. Also backported it to the 2.1.x stream. This fix 'handles' the case of making a structural change to a type, which has several dependencies in its own project. Dependent projects will now only consider the 'real' structural change and will ignore the non-structural changes. Limitations: - all previous structural changes in this project that were locally built (ie. incremental build of the project only) must have been propagated to its dependent projects before making a new structural change - additional non-structural changes may be made in the project and locally built before propagating the structural change, as long as another structural change is not made. - if the user manually builds a few projects (ie. a scoped build) with structural changes more than once during the session, this optimization will help within those projects but will not help the rest of the dependent projects in the workspace. Still a bit curious about two things: - coexistence with other builders. Do we properly still consider changes performed by other builders in prereq projects (assuming they also contribute .class files) or are we consider these which we now are non- structural ? - what about building project cycles, does this break the optimizing scenario ? We never fully supported other builders in the past. Before this change when the Java builder built a project A and didn't make any structural changes, all dependent projects skipped the .class file deltas from A's output folder(s), regardless of any changes made by another builder. Now with this change, dependent projects will only consider the structurally changed types recorded in A's output folder(s). If anything we are more consistent now. The bottom line is if a second builder structurally changes .class files after the Java builder, then its responsible for propagating the changes. As for handling cycles, as long the second iteration did not produce structural changes, we'll be fine. If it does and causes a third iteration then this change will only be helpful to the projects involved in the cycle. I was wrong about the cycles case. Core did not 'fix' them the way we discussed. Since the entire build cycle is repeated, instead of just the projects involved in the cycle, this change will work even for dependent projects not involved in the cycle. Marking as fixed. Original problem is solved with this fix. Verified for 3.0M5 Verifeid for 2.1.3 (M20040225) |