Community
Participate
Working Groups
Customers have been reporting problems trying to build large applications (i.e. more than 35 projects). Although there are several scenarios that this problems occurs, the simpliest way to think of this is then a customer starts with a new workspace, loads the source of the large application and selects Rebuild All. The result is a few thousand errors in the task list. The same source can be built using JAVAC (i.e builds with out any errors). The difference in the build results between using JAVAC and Eclipse appears to related to circular dependencies. The reasons for having circular dependencies in these large applications varies with each customer. From the customer's point of view they have many developers using this structure and their nightly builds (using JAVAC) build without any errors. Thus they expect the IDE to be able to build the same code without errors. In addition waiting for V2 of Eclipse is not a alternative for these customers. Also building individual projects is also not alternaitve since it can take a long, long time to get all the parts build without errors using elipse. To demonstrate the problem I have created a testcase using 6 projects. Here are the steps using Eclipse: 0) Turn off Automatic build on Windows->Perferences 1) Create the projects: CycleDB CycleMain CyclePeople CyclePolicy CycleServer CycleUtil 2) Import from the file system into CycleDB the following: ivjdab.jar vajAS400ET.jar vajrmi.jar dax.jar Import from the file system into CycleServer the following: VAJSQLJ.jar The jars files are in RTJars.zip 3) Import the java source into each project from the corresponding jars files (which are in Cycle.zip). Ensure that the .classpath file is replaced during the import. 4) Select Project->Rebuild All The result will be a few hundred build errors. Here are the steps using JAVAC: 1) Ensure you have a 1.3 JDK installed 2) Place the bld.cmd (which is in Cycle.zip) in the Eclpise workspace directory 3) From the command line remove all the class files in the workspace directory tree (for example issue del *.class /s from the workspace directory) 4) Ensure JAVAC is on your path 5) Run bld.cmd The result is a successful build with no errors. The zip files Cycle and RTJars are available on request.
Created attachment 393 [details] Contains bld.cmd and source jar files
*** Bug 10264 has been marked as a duplicate of this bug. ***
It is expected that builds with circular dependencies will fail. Each project is only built once, and the projects are built in some sequential order. We plan to make this situation more obvious to users, but we have no plan to support circular dependencies.
That is unacceptable for some users. You are saying that the Workbench will not be able to compile code that javac can compile, right? Is it true that the only option we can give to customers who cannot remove circular dependencies, is to not buy products that are build one top of Eclipse? Please let me know your thoughts.
This is and will continue to be a problem for customers (espically for VAJ customers). Again from the customer point of view, the code can be built using JAVAC, thus it should build with the IDE. Need to look at if this still exists in V2 of Eclipse. One point fo clarification; Then talking about circular dependencies I awa referring to project dependencies and not file dependencies.
Please provide RTJars.zip
Created attachment 424 [details] DAX.jar
Created attachment 425 [details] ivjdab.jar
Created attachment 426 [details] vajrmi.jar
Created attachment 427 [details] VAJSQLJ.jar
I am having trouble attaching RTJar.zip. I have attached 4 of the jar files individually. However I am unable to attach vajAS400ET.jar, probably because of it's size (9 MB). Is there another way to sent this jar file?
I tried the same steps and code using the 20020214 V2 driver. I still have several hundred errors after the rebuild all. Interesting enough all of the class files are created after the first rebuildAll. Doing 2 more rebuildAll(s) will leave you with 6 errors in the task list (and a bucnch of deprecated warnings). Of the 6 errors, 5 errors are indicating that there are cyclic project dependencies and 1 is an unreachable code error. In V1 pressing RebuildALL twice eliminates all errors from the task list.
I found a way to eliminate the need for vajAS400ET.jar. You need to comment out the static section of the AS400DS class in the DBInterface packge of the CycleDB project. The AS400DS class should now look like he following before trying to do a build: package DBInterface; /** * This type was generated by a SmartGuide. * @author Linda C */ //import java.sqlutil.*; //import COM.ibm.ivj.eab.data.*; //import javax.swing.*; public class AS400DS { public static java.sql.Connection UR; // Uncommited Reads - 2nd Tier public static java.sql.Connection CR; // Commited Reads public static java.sql.Connection RR; // Repeatable Reads static { /* try { java.sql.DriverManager.registerDriver (new com.ibm.as400.access.AS400JDBCDriver()); //Connection c = DriverManager.getConnection ("jdbc:as400://TORAS180"); UR = java.sql.DriverManager.getConnection( "jdbc:as400://TORAS005;naming=sql;errors=full;transaction isolation=read uncommitted" , "tschic", "ch8com"); UR.setAutoCommit(false); CR = java.sql.DriverManager.getConnection( "jdbc:as400://TORAS005;naming=sql;errors=full;transaction isolation=read committed" , "tschic", "ch8com"); CR.setAutoCommit(false); RR = java.sql.DriverManager.getConnection( "jdbc:as400://TORAS005;naming=sql;errors=full;transaction isolation=repeatable read" , "tschic", "ch8com"); RR.setAutoCommit(false); } catch (java.sql.SQLException conExp) { System.out.println("400 connect error"); //throw new Policies.PolicyDatabaseAccessException ("Unable to connect to the AS400"); } */ } }
Regarding the comment posted by John Artone I agree with Paula that not supporting cycles between projects is unacceptable. Projects are Eclipse - or VAJ - artifacts that exist to make life easier to the user, the concept of "project" does not exist in Java so any error related to the existence of the "project" concept when building up an application is an IDE bug. Please, do not reinvent Java!
*** Bug 8667 has been marked as a duplicate of this bug. ***
I suggest this bug be resolved as WONTFIX. The reason for this (which is obvious to the Eclipse developers, but apparently not to all users) is that the current design allows any kind of dependencies within a single project. Those having problems with some projects having circular dependencies simply have to make a view of those projects as a single project, possibly with many source folders. I say a view because most seem to think they are locked by some kind of historical directory structure that "works fine with other IDEs". However, a parallel view can always be created by using symbolic links. This is most easily done on the CVS server (assuming one is used and it runs some kind of UNIX). Alternatively, with module support in Eclipse's CVS-client, modules can probably be used instead to create a second view. NOTE: Win2k and later support something similar to symlinks, see http://www.sysinternals.com/ntw2k/source/misc.shtml#junction These different views could be used in parallel forever, but that this problem occurs should be taken as a warning signal that your project partitioning is inadequate. Fixing that, not Eclipse, should be your primary concern. Usually the sooner, the better, so you should be happy Eclipse pointed this out. There are programs like Headway reView that can help with this sort of thing. (Google it.) If you choose not to do anything about it, fine, it's your mess. But stop complaining about Eclipse enforcing proper project partitioning. It is a highly desirable behavior, it is by design and certainly not a bug. Lastly I should mention that I work with a codebase consisting of almost 5000 java files. We previously used VAJ with 30-some projects. Unfortunately, due to lack of design and enforcing mechanisms, there were a huge amount of inter- project circular depencencies. When we moved to Eclipse, everything compiled fine (modulo compiler JLS ambiguity pickiness) as a single project. To improve our design to be modular rather than monolithic (something that's been long overdue) we rearranged the code into 152 projects with proper unidirectional dependencies. Unsurprisingly, that caused a few thousand compiler errors that we currently are resolving, mostly using Service Provider Interfaces (and a lot of generic code-cleaning). Meanwhile, we have an aggregated view of the 152 projects which continuously will compile without errors. Side note: Using 152 projects (with errors) in Eclipse is usually significantly slower (startup, syncing) than using the aggregated project (error free) with the same code. But that's another story. (When we're done, we probably won't use more than ten projects at a time anyway.)
Most of the time people encountering such a cycle will bypass the error simply joining the projects involved in the cycle in a single project, that is going in the opposite direction of modularity. Human laziness will always find a way to bypass the good practices enforced by a tool. So do not try to make the perfect tool. Let the user decide how he or she will use the tool. A great tool is that one that allows the user to use it in a way that the tool developers never thought. Let me insist if you want to warn the user, change the message to a warning but let eclipse work as javac - and VAJ - works.
What is the lastest status?
There is still no plan to fix this in the 2.0 timeframe.
We developed a SCADA - plant monitoring - system using Java and VAJ. Among others we have two main programs the Monitor and a Panel Editor to build and edit plant panels that represent graphically plant data in real time. We have two projects one devoted to the Monitor and the other one related to the Panel Editor. We have different people asigned to each project. The Panel Editor can test a panel in real time so it needs to access classes in the Monitor project. One day we decided that it would be nice to allow the user to launch the Panel Editor from the monitor. It was a simple thing to do we just added a menu item with a reference to the Panel Editor main method. But due to this bug now this cannot be done without finishing joining both projects. Can you say that this is our fault because a bad design? Do you think that Eclipse is better if it does not allow that kind of changes? Please, please, please fix this bug.
Anyone have any updates on this issue? Just curious. Thanks, Steve
This seems to have been debated to a standstill. ;-} It's certainly easier to support builds that can be reduced to a partial order. Unfortunately, users will always split things into subprojects based on considerations other than application architecture, e.g., developer location or project size, and cyclic dependencies between projects are often introduced as design evolves. Yes, there is always some refactoring that will eliminate cycles - you can always put all of the code in one big project - but the tool shouldn't dictate the partitioning of the code. Most professionally-written linkers support cyclic dependencies between modules; indeed, that is the fundamental problem of linking. It can't be debated away. It would be good to see some statement of intention from the developers in the bug commentary.
I agree with Bob's comments. My company would really like to see this issue resolved (for or against circular dependencies) before we consider going to App. Developer. It would just ne nice to know the direction so that we can either plan to change our code to remove those dependencies or not have to do it. Steve
This limitation (circular project dependencies) really isn't a totally new concept within Eclipse. Plugins also have not been allowed to have circular dependencies on each other since the beginning. Avoiding circular dependencies between projects can prevent further development issues down the line. Also a way to avoid a circular dependency may be to create a third project and having the initial 2 projects depend on it. Sort of like the way DBA's create a third table when they wish to remove many-to-many relationships when they attempt to seek a normalized database environment.
If I understand correctly from your comments it sounds like Eclipse will never allow circ. dependencies? I am just looking for a decision either way on this issue.
I am also interested in hearing the decsion on this one! However, I think the only person who can make that call is the owner of this bug report or someone representing OTI.
*** Bug 23683 has been marked as a duplicate of this bug. ***
If you wish, have a look at "my" bug 23683 with a suggestion somewhere in the middle between full-blown "support cyclic dependency" and "not support cyclic dependency" at all. I don't think it'd be the last wise word from a conceptual point of view, but hey, neither is Java. On the other hand, it'd be cheap to do, from OTI's point of view, and would help in many typical situations.
O, I didn't immediately see the the significance of the quote by OTI's Philippe Mulet, somewhere in the discussion in bug 23683: "The Eclipse current build infrastructure doesn't support cycles, but this is going to change (see 2.1 plans)." and "the JDT/Core cycle change is also going to be backported into a 2.0.2 update." So some decision on the issue has been reached, even though it has not been reported through this bug. Through a quick search of the web site, I found in http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/jdt-core-home/r2.1/main.html the note "* [done]If toggling the Java builder for proceeding in presence of classpath issues (e.g. cycles), classpath issues should be reported as warnings."
Sorry folks for not having updated this one PR, but as you see it is not directly in JDT/Core range. The changes we applied in JDT/Core, and which are going to be backported for 2.0.2 are allowing to build at least as well as in Eclipse 1.0, which did not care about cycles, and was silently leaving inconsistencies. Now, we will flag these inconsistencies (through cycle markers) but still preserve some reasonable build action (if settings are toggled accordingly). I still think some action should be taken on Platform/Core to fully address the cycle issue: - ensure the build order is reasonable in presence of cycles - build process should iterate until projects in cycle, incrementally until no change is made or some maximum number of iteration has been exceeded (in case some builders are modifying resources too eagerly). However, I would be interested to know how much people think the JDT/Core only fixes would provide an acceptable middle-term solution for 2.0.2 ?
Hi Philippe, Sounds good to me. However, what impact would this isolated fix for the JDT/Core only for 2.0.2 have on the proposed fix being developed by OTI (and to be applied on top of 2.0.2) that would "translate" error messages regarding circular dependencies into warning messages. Also, are there any side effects to producing an isolated fix? Any potential cascading effects? Would fixing the problem in JDT/Core and not in Platform/Core lead to confusion, that is, will something go wrong during a build that would totally confuse the user and make them unaware of what the problem is. Currently they have a problem building with circular dependencies and they know it because the error messages tell them. Would they, with your fix, have a problem and still know what the problem is? Thanks and Regards, Ali
Yes, the JDT/Core change is rather cosmetic. You can choose the severity of a classpath problem: either ERROR or WARNING (can't ignore them completely). Then you still have the toggle allowing to abort building in presence of classpath errors. Those combined allow you to proceed building in presence of cycles, without seeing cycle errors which some users were complaining about. But as I said earlier, this may not properly building your projects, since the build order is still wrong (circular projects are built at the end), and only one pass on each project is performed, which means prereqs of project in cycles may not have been built yet...
We have seen this issue addressed incrementally, (which is fine) in 2.0.1, 2.0.2 and further improvements are being worked on in 2.1. Thus, the question I have (which I hope generates some discussion) is what is the expected final behaviour for workspaces with circular dependent projects? Is it that we eventually want to get them build cleanly and produce warnings? Or is something even further than that where the user could set some preference indicating whether he wishes to be warned about this, if in fact his workspace were to have circular dependent projects. Since, some users may actually like to be warned (for clean design purposes) and others (java purists) simply may not care? Or is the final outcome on this issue some other final state? If so, what?
I like the idea of having the preference. Some people may like to be warned because they would like to avoid these dependencies. Others will have circular dependencies and may not be able to break the dependencies so they will not want to be warned about a condition that they cannot fix. The final state should be a clean build on the first attempt when circular dependencies do exist. Ideally, there would be little to no build time increase due to this condition since it is a very common scenario with specialized Java projects.
I agree with Mr. Berg: A clean build is what we want. In my opinion, projects are not the right entities to look at with respect to low level building activities. A Java IDE should have .java - files as the entities that get buildt. If I first need to compile project A's java file a1.java, then project B's java file b1.java, next project A's java file a2.java - so what? If b1.java needs a1.class to be compiled, and a2.java needs b1.class - then build first a1, then b1, then a2 is what needs to be done and that is what I want my IDE to do. If I tell the IDE, "rebuild project A", there should be a way for me to find out, in case I care, why the heck this always rebuilds B's b1, too. But mostly, I won't care. In either case, "rebuild project A" should simply mean, "do whatever is required to have everything in A in good shape". If, afterwards, b1 is in good shape, too - fine with me. However, in an ideal world, "rebuild project A" should not result in a rebuild of some unrelated b2 in B, even if b2 requires b1, unless there's something in A that needs b2. Do you people know that famous essay, "recursive make considered harmful"? http://www.tip.net.au/~millerp/rmch/recu-make-cons-harm.html
Here is some update based on our current thinking. Eclipse building story enforces to have project builders. This approach allows to sequence lots of builders on a per project basis, not just running a Java compilers. You may expect forthcoming tools to take advantage of this feature, and updates targets with generated .class files, or preprocess .java files right before the Java compiler kicks in. Also, the building strategy is to compute a project build order, and then iterate over these projects so as to run all the builders in appropriate order (depending on how builders got registered). In this model, cycles are nasty. If all you care is to compile in one pass (like you would with Javac) all the Java files, then you can achieve the exact same behavior using an Ant script (thus bypassing the Eclipse build management). This is only a workaround, and notice that when you do so, you likely have a common classpath for all these files; as opposed to a per project classpath as you currently have when using Eclipse projects. I mentionned earlier that Eclipse platform could solve cycles issues by: - change the build order to not defer projects in cycle to the end of the build process - iterate incrementally on projects involved into a cycle until no more change is generated (remember that we have incremental builders) The first issue got solved around 2.1M3, but the second isn't doable after further thoughts. In the grand picture, where any tool could contribute any number of builders on a project, you must realize that when building a Java project, you may trigger 10 incremental builders, and who knows what they are going to do... are they even able to deal with incrementality (i.e. a simple approach to an incremental builder is to simply perform a full build over and over again, not fast, but simple to write). Under these rules, we have to be really defensive and protect against these sorts of scenarii. We cannot make the assumption that the only builder which will run is the Java builder, which can perform a good incremental job and thus could succeed building these cycles. The only alternative would we to change completely the build management process to iterate over builders and pass them the list of projects to build. This way, the Java builder would get the set of Java projects to compile, and could do a better job at handling cycles. But this is a breaking change, which cannot be done for 2.1, since it would change all current assumptions that builder tools have made so far. There is one thing we can explore though. Eclipse doesn't put any constraint that builder modifies resources only inside its project being built. One could imagine that when building a project involved inside a cycle, its siblings could also be (partially) built as needed at the same time, so that it would avoid find errors where it simply hadn't yet compiled enough code. e.g. Project1 <---> Project2 when Project1 is being built by the Java builder, if Project2 hasn't been compiled yet, it would also trigger some compilation inside Project2, so that Project1 can be proceed normally. The Java building activity could spread over more than one project, when cycles are present. This would however not mean that the Java builder for Project1 would recursively a build action on Project2. This would be against the rule, and could lead to some infinite building loops. As a consequence, it means that pre/post registered builders may be fooled a little by our action. Here is an example describing this situation: Imagine that Project1 has 3 registered builders: Project1 +- preBuilder1 +- javaBuilder1 +- postBuilder1 Imagine that Project2 has 3 registered builders: Project2 +-preBuilder2 +-javaBuilder2 +- postBuilder2 (actually pre/post builder could denote an arbitrary number of builders). Now, imagine Project1 and Project2 are involved in a cycle. When performing a full build, with build order Project1->Project2, the following builder activity would occur: Project1 Build preBuilder1 javaBuilder1 -> javaBuilder2 postBuilder1 Project2 Build preBuilder2 javaBuilder2 (in case changes were done by preBuilder2) --> javaBuilder1 postBuilder2 This means that if Project2 Build/javaBuilder2 induces changes in Project1, postBuilder1 will not see them until the next Build action... On the good side, we would still associate each project with its own classpath (as opposed to using a common one). How would this sound ? It would still be a JDT/Core only solution, which in most cases would be acceptable we think. We would like to investigate it further for 2.1M5... Also, given the latest additions on flexible project structure: - linked resources (~symlinks) - exclusion patterns associated with source folders - multiple output folders (at most one per source folders) I suspect, the need for cycle support is no longer so strong...
> Eclipse building story enforces to have project builders. The point this essay "recursive makefiles considered bad" is making, is exactly: This is a sad building story. You will continue to be plagued with the consequences, messing around with a fix here and a discussion there and what have you, until you finally change that story. Interestingly, those same bad fixes that essay noticed back in the old C / Makefile days, are still alive and well. Your idea, "we will build over and over again" is a typical bad fix, something they've done back then, and, as I see, people are still doing it, even in the modern world of Java and IDE and what have you. Building is a depth-first traversal of a (huge) directed acyclic graph. Nodes being the "objects" that are being build (typically files, but could be other stuff), and an edge connects each prerequisite node required for building a target node with that target node. Do your depth-first traversal, build everything that's outdated on the way back, and you're done building. This is the one build abstraction known to work. I suggest you should seriously consider opening up Eclipse for this kind of a build abstraction. > you may trigger 10 incremental builders, > and who knows what they are going to do... That is exactly the point. This kind of knowledge is power. You could have "simple" builders. If a builder never needs the kind of stuff it itself generates as its own prerequisits (e.g., from another project), neither directly nor indirectly, Eclipse only needs to call it with an order "build this project". For such simply builders, your build story is all that's required. But if a builder needs interaction with other builders, e.g., instances of itself registered for a different project, Eclipse should offer with the neccessary infrastructure to facilitate such interaction. In other words: These builder are "intelligent builders". Intelligent builders should be both able and required to tell Eclipse "what they are going to do", that is, what the nodes and edges in the graph are. Which other objects are required to build each target object? For intelligent building, whole projects as build target are simply to coarse. Eclipse should be able to ask a builder to build particular objects. From a list that has been made up on the fly, by Eclipse's build management. Nodes from a project do belong into an equivalence class in other respects. But not as far as building activity is concerned. You'll never get that build graph cycle-free this way. Those cycles will continue to burn money and time, both CPU time and headache time. If your abstractions are finely-grained enough, the acyclic graph readily appears. The build process then becomes a depth-first traversal. Everybody knows how that works. Everybody understands it. You can make a precise report on what's going to happen, even before the fact, should anybody need or want that. No more "who knows what's going on". This is the one way to minimize both CPU and headache time.
Remaining work for M5 is in JDT/Core space. Moving bug report.
Actually, the looping will be done in the platform, until the JavaBuilder claims it has no more structural changes to apply. Back to Platform/Core (the JDT/Core side is in, we only need clearance to release it).
Fixed. We now iterate until no more builders request rebuilds. There is a new preference setting for the maximum number of build iterations. These fixes are in integration build I20030205. Users are still encouraged to amalgamate cyclic projects into one single project with several source folders. Our new support for linked resources, and for checking out from CVS directly into a sub-folder, should make this easier than it used to be for people with existing layout constraints.
I have filed an enhancement request under the title "Infrastructure for one-shot build across projects". This is now bug 31547, i.e., http://bugs.eclipse.org/bugs/show_bug.cgi?id=31547 . (Of course, you can add yourself to the CC list there if you're interested what comes out of it.)
