Community
Participate
Working Groups
This reopens bug 476083 In year 2001 a limit for java files build in one batch was set to 1000 with the initial commit to constrain memory usage. (in org.eclipse.jdt.internal.core.builder.AbstractImageBuilder.MAX_AT_ONCE) Later the default value has been increased to 2000 due to bug 106446, 126999. The underlying Problem that a limited file set can produce compilation issues was never solved. Still the errors exists and typically also appear when using JPA annotation processing. Its kinda annoying to fiddle out why compilation failed and then to increase the maxCompiledUnitsAtOnce. We got project where we increased it to 100000 and still dont run in any memory trouble. Back in 2001 ram was expensive and heap limited to 32 bit adressing and limited even more by OS to 31bit. Today (2021) ram is cheap and plenty and virtual 64 bit adressed both in OS and JVM. All source files will always fit into virtual 64 bit RAM. I vote, that maxCompiledUnitsAtOnce is virtually removed by setting it to Integer.MAX_VALUE. If anyone does have a problem with high ram usage he could still limit this (or increase max heap). Its much easier to identify high ram usage then to google why compilation or annotation processing failed.
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/176922
Value of 0 indicates to not build in chunks. Probably that value should be used instead of Integer.MAX_VALUE. We use "-DmaxCompiledUnitsAtOnce=0" in our JDT+xtext based product and have not seen problems with it (to my knowledge). Still, personally I wouldn't touch the code based on "we don't have problems with it, so nobody else will".
(In reply to Simeon Andreev from comment #2) > Value of 0 indicates to not build in chunks. Probably that value should be > used instead of Integer.MAX_VALUE. > > We use "-DmaxCompiledUnitsAtOnce=0" in our JDT+xtext based product and have > not seen problems with it (to my knowledge). Still, personally I wouldn't > touch the code based on "we don't have problems with it, so nobody else > will". +1. Agree with Simon.
I would really be wary of touching that part of the code, especially when it is working well.
It is not working well! I lost 3 days for debugging until i found why our projects do not compile. And since i found the workaround (years ago) i am called on a regular basis by colleagues asking "what was that parameter again?". I want to get rid of that. It is only working well if you a) have a small project, where the limit doesnt matter anyway, b) have no annotation processor, c) have *luck* that all annotated files are processed in one batch, d) you have an eclipse with increased limit! We only used to give up since we have a workaround which like everybody uses. A permanent workaround.
Possibly you can provide reproduction steps, if your environment has the problem? Our developers work with ~250 plug-ins in the workspace, the code base is not small either. We don't set maxCompiledUnitsAtOnce in the developers IDE (only in the product that is being developed). So the problem you are seeing might not be related to project/workspace size, but specific build steps/builders in your environment? E.g. we had bug 531382 before, with JDT+xtext. Though I guess you can always ask on the mailing lists. If multiple subscribers would agree the default 2000 makes no sense (anymore), that would be an argument in favour of the change. In our case (me/Andrey), we probably don't mind the change.
The bug is not about the code in the workspace but about number of .java files within a single project: As soon as you have more .java files in a project then the java build for that project is split into batches. As soon as you require all files in one batch (due to multiple classes in single files or annotation processing) it fails without any meanigfull error. That is well reproducibly and was already desribed and analysed in other bugs which have been closed with workaround or autoclosed. (I do not like workarounds nor autocloses)
(In reply to Jörg Kubitz from comment #7) > That is well reproducibly and was > already desribed and analysed in other bugs which have been closed with > workaround or autoclosed. (I do not like workarounds nor autocloses) You can link those bugs then?
bug 126999 workaround bug 476083 autoclosed bug 533862 autoclosed bug 561287 "fixed" but comment of reoccurances #c73 The workaround has even made it into manuals: https://gerrit.wikimedia.org/r/Documentation/dev-eclipse.html https://www.artisynth.org/manuals/index.jsp?topic=%2Forg.artisynth.doc%2Fhtml%2Finstallation%2Fwindows%2FS10.html
(In reply to Jörg Kubitz from comment #7) > [...] As soon as you require all files in one > batch (due to multiple classes in single files or annotation processing) it > fails without any meanigfull error. I believe this is the part we need to understand here. Whenever I got the chance to debug an issue in this area, a real bug in the code was found and fixed. E.g.: bug 561287 has been fixed, really. And I'm glad we noticed the bug. Any reported reoccurrences of this may share the same symptom but certainly have a different cause. If you observe that the concept of chunked compilation is broken by design then perhaps you can explain, what exactly is broken. Why, e.g., does an annotation processor require to process all files in one batch? Similarly, the case of multiple classes in a single file (JDT lingo: "secondary classes"), surely is difficult to handle, but from my understanding solutions for this are in place. Can you demonstrate that those solutions don't work? This is perhaps a recipe for a reproducer: set max at once to 2 and compile a package with several files containing secondary types, which then depend on each other. Is that your situation? Alternatively, if indeed there is a problem we cannot solve by bug fixing (either because the design is broken or because we're unable to understand the bug), I don't think it's a bad idea to discuss how configuring JDT for each project's needs can be made easier. Pragmatically, if in my company projects would need to unlimit max at once, I would write this parameter into their Oomph setups, and thus the change will be automatically rolled out to all developers :) If the symptom of this problem would be easy to classify, we could add a hint to the error message, but I don't think we are in a position to judge if a particular error is connected to this issued. As a last resort, changing the default, without removing the parameter (so people can still restore the previous default), is not unthinkable, if you ask me. Do we have sufficient experience if removing that limit works reliably? We have millions of users who are happy with the current value. How many users happily work with no limit? We don't need millions casting their vote here, but confirmation from a handful of teams would be nice. A point in favor of chunked compilation: it helps to keep internal lookup structures small (LookupEnvironment.knownPackages, I'm looking at you), thus minimizing lookup times :) - put differently, there's a connection between this question and bug 571288. Something like: if you turn off chunked compilation than you increase pressure on lookup performance.
>set max at once to 2 Isnt "max at once" just another workaround? Well, if you agree that max at once=2 (not more), maxCompiledUnitsAtOnce=1 (!) should compile anything then i will try to find a small counterexample.
(In reply to Jörg Kubitz from comment #11) > >set max at once to 2 > > Isnt "max at once" just another workaround? when looking for reproducing examples I don't look for beauty, I look for something that demonstrates a bug. Putting more pressure on the compiler/builder should make reproduction easier. That's how Till managed to provide er reproducer for bug 561287. > Well, if you agree that max at once=2 (not more), maxCompiledUnitsAtOnce=1 > (!) should compile anything then i will try to find a small counterexample. I'm not sure about 1, but yes, if you find an example that reproducibly breaks with a low chunk size, that would be helpful for understanding the problem (assuming that the small example represents the same problem as you might observe on real workspaces even with a more realistic chunk size).
Created attachment 285738 [details] java source minimal example 1. start eclipse.exe -vmargs -DmaxCompiledUnitsAtOnce=1 2. create a javaproject with the source attached. 3. project/ Clean... => "B2 cannot be resolved to a type" 4. touch A.java 5. save A.java (assuming autobuild on) => error gone. 6. project/ Clean... => "B2 cannot be resolved to a type" ...
Is someone working on (or planing to work on) a fix for the underlying issue (Stephan?)? Otherwise i ask to disable the not working batch feature.
Moving target to the next M1
(In reply to Jörg Kubitz from comment #14) > Is someone working on (or planing to work on) a fix for the underlying issue > (Stephan?)? > Otherwise i ask to disable the not working batch feature. Thanks for providing steps, but sorry, no, I didn't have the time to look at this. Without debugging your sample, it seems to say, it all boils down to inability to resolve secondary types across file boundaries during chunked builds, right? If that's the trigger condition then I imagine that simply very few people ever encounter this situation. In all my code, I cannot recall ever referencing a secondary type from another file. Of course I'm not saying this situation can be ignored, but once again I wonder, why the projects exposed to this bug cannot set up their environment such that chunking is switched off via the system property. You have that option, here and now. At the other end of the spectrum, when you propose to change the default behavior for all Eclipse users, we (Manoj, me, and perhaps other oldies, too) are reluctant not because we specifically know about the high value of that feature, but because we don't know all implications of changing the default, and please believe us, each of us has been burned more than once by a simple and safe change that has caused big grief for some users. Ergo: in JDT the notion of a "simple and safe change" frequently turns out to be an illusion. I see no reason why this bug would be an exception. While it's probably difficult to convince us, that individually setting a system property is much worse than changing a default that affects all users, there could be another way out: I would not worry if the change is made by a person with a proven track record of swiftly fixing any new bugs in that particular code area. Anybody can earn the trust, can take responsibility and eventually (informal) ownership of a particular component. But in my book, bug fixing comes first, and disabling a feature comes later. And there's a simply reason: there's no better way to acquire and demonstrate a solid understanding of how JDT works in detail than by fixing whatever bugs come flying in. I admit I'm no longer closely observing who contributes what to JDT these days; who watches the bugzilla inbox and resolves incoming bugs; in which parts of JDT? OTOH, if the above doesn't sound attractive to you, please bear with our judgement.
(In reply to Stephan Herrmann from comment #16) > bug fixing comes first, I proved that the batch has a bug. We suffer from this bug. I ask to fix this bug. I provided a possible solution. Nobody would ever enable the batch again unless proving that the batch has any value. But committers do not need to proof anything anymore because they reached a "proven track record"? That would render any factual argument pointless. The only value batching claims is to avoid OutOfMemory - a pretty obvious error message. Getting the batch/bug back is possible at runtime. So there is no risk to be in need for a source rollback.
(In reply to Stephan Herrmann from comment #16) Thanks Stephan for speaking for all of us, oldies. Inbox tracking itself has become a challenge and a request in the jdt-dev mailing list elicited no response :(. >OTOH, if the above doesn't sound attractive to you, please bear with our judgement. @Joerg: As Stephan says, please bear with our judgement.
Jörg, as an offer for peace: perhaps the discussion has focused too much on my view of things. My main point is: for every change someone has to take responsibility. For your proposed change I'm not that person (nor is Manoj). After some time in JDT that person could eventually be you. For now let me just state that I don't agree with what you said in comment 17, but perhaps it's time to just let it rest and focus on inbox tracking, and - well - I created bug 574072 for the remaining problem that still needs fixing.
(In reply to Stephan Herrmann from comment #19) We share the same goal to keep/make jdt good. > responsibility. For your proposed change I'm not that person (nor is Manoj). > After some time in JDT that person could eventually be you. I can offer responsibility for any change i am involved. No change no responsibility. I can currently not take responsibility for general inbox and do not aim for that. Therefore i accept any decision of those who do. You do. Don't you?
Bug 476083 comment 2 proposed to expose this flag as a workspace preference for JDT (instead of a hidden flag). I think it could be a solution here, but it would require a bit more work. Those who see the issue could change the preference and either persist it via project settings or via Oomph. Jörg: would it be alternative for you?
(In reply to Andrey Loskutov from comment #21) > Bug 476083 comment 2 proposed to expose this flag as a workspace preference > for JDT (instead of a hidden flag). I dislike hidden flags too. I would like to have all parameters in the UI and a public description what they do. Its not even much work - I just added a preference. Nevertheless i am less concerned about being able to set it at runtime. I am concerned about a product which is shipped with unusable default settings. It is just user unfriendly to ship a product with a known error and ask every user who stumble about it to adjust a setting (or even to use oomph..). Especially if the user has no chance to know why this error occurs. Nor giving a meaningful or even related error message. It is not intuitive for a java developer that jdt would even try to split the project into batches. Everybody expects eclipse to compile if JAVAC does. As long as no hint is given that a parameter is wrong it does not help if that parameter can be changed via UI. The preferences are already fraught with settings! The challenge is to deliver good default settings. I am fine to set the hidden feature - now. I know how todo it - now. I do not have any problem with that parameter myself - anymore. I even found more secrets reading the sourcecode and the bug database. But i do not expect a user to read those! Its just my claim to ship a product which works out of the box. > Those who see the issue Thats the main issue: You can not see the issue! Blessed are those users who have a guy like you or me who knows and solves such issues in their company. But thats only a fraction of the eclipse users.
(In reply to Jörg Kubitz from comment #22) > I am concerned about a product which is shipped with unusable default settings. please try to cool down. "unusable default settings" would imply that EVERY user needs to modify those settings. I'm still convinced that the current default is fine for the majority of users. Eclipse works out of the box for very many users. Alas, it's not bug free as we all well know.
(In reply to Stephan Herrmann from comment #23) > please try to cool down. Don't take it personally. >"unusable default settings" would imply that EVERY > user needs to modify those settings. That would imply you would also call a texteditor "usable" which starts failing after 2000 letters. I would not. But yes, i claim all of you are using maxCompiledUnitsAtOnce=0 on daily bases (because not even jdt tests can run without it). Proof? commit b0085304483984f0cc2d0511332c026b3464a6e0.
(In reply to Jörg Kubitz from comment #24) > (In reply to Stephan Herrmann from comment #23) > > please try to cool down. > > Don't take it personally. I won't. I'm leaving the discussion > But yes, i claim all of you are using maxCompiledUnitsAtOnce=0 on daily > bases (because not even jdt tests can run without it). Proof? commit > b0085304483984f0cc2d0511332c026b3464a6e0. Funny. That's the commit that introduced the system property and tests that it works.
Verified with Eclipse. Version: 2021-09 (4.21) Build id: I20210706-1800
