Bug 561287 - Incremental compilation reports spurious compilation errors with bigger workspaces
Summary: Incremental compilation reports spurious compilation errors with bigger works...
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.15   Edit
Hardware: All All
: P3 major with 8 votes (vote)
Target Milestone: 4.16 M1   Edit
Assignee: Stephan Herrmann CLA
QA Contact:
URL:
Whiteboard:
Keywords: regression
: 561417 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-03-20 04:31 EDT by Al Bundy CLA
Modified: 2021-03-02 04:12 EST (History)
15 users (show)

See Also:


Attachments
example project (5.16 KB, application/zip)
2020-03-30 14:58 EDT, Till Brychcy CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Al Bundy CLA 2020-03-20 04:31:18 EDT
We have this Problem on Windows and Linux.
After Update to Eclipse 2020-03 and cleaning a project some of our own imports are marked as error because they can not be resolved altough they are existing.

Organize Imports does not work.
We have to remove the marked imports and organize imports again.

Error messages are only "XXXXX cannot be resolved to a type".
Comment 1 Stephan Herrmann CLA 2020-03-20 04:51:33 EDT
Thanks for the report. We'll definitely need more information before we can start working on this. Are you able to reproduce the problem in a sample project that you can share?

When you say "own imports" are those within the same project? Or else, how exactly is the dependency declared? Using Maven, as an explicit project dependency in the Java Build Path?

Which version of JRE are you using? Which compliance level for compilation?
Comment 2 Al Bundy CLA 2020-03-20 05:02:08 EDT
I can try to create a test-project but not all of our imports are affected.
Affected classes are all part of the projects (plugin and java nature) sources.

compiler compliance level is set to 1.7 due to backwarts compatibility.

JRE in my tested project is 1.7.0_80 but I can try another project with java 8
Comment 3 Al Bundy CLA 2020-03-20 06:07:49 EDT
another project (same product but different branch) with java 8 is also affected.

It seems that in both projects (remember - different branches!) the imports are marked as error.

But I found no indicator, why this happens.

btw. Opening the "missing" klasses with F3 works as expected even if they are marked that they cannot be resolved.
Comment 4 Al Bundy CLA 2020-03-20 06:44:51 EDT
make a small change in an affected file (eg add a whitespace), deleting the change (do not use undo - the file must be dirty for next step) and saving the file ends up in a file without errors (until the next full build of course)
Comment 5 Al Bundy CLA 2020-03-20 08:12:18 EDT
Hmm - I'm not sure but could this be kind of an order-issue?

Here is a small (trimmed) example.

package1.DispositionHelper
shows error on
package2.IRentalCalculation
and some other classes in package2
but has no problem with
package3.MailMessageUtil

here are the creation times of the class files
package1.DispositionHelper.class ‏‎12:51:47
package2.IRentalCalculation.class ‏‎12:51:53
package3.MailMessageUtil.class ‏‎12:51:25

as you can see all class-files are created 12:51

but the class-files of the "missing" class (IRentalCalculation) was generated after the class that shows the error (DispositionHelper).

I've also checked other classes - they have all the same symptom.

This would at least explain why saving the file with the "error" fixes this issue.
Comment 6 Max Dynin CLA 2020-03-24 05:50:19 EDT
In a complex project (as ours, unfortunately, is), it is a lot of work to reorder the build path to alleviate this problem. And figuring it out is not trivial by any means.

Also, this problem has already occurred - cf. bug 150749, Eclipse 3.2, back in 2006.
Comment 7 Al Bundy CLA 2020-03-24 06:20:35 EDT
Although we are also using multiple source-foldes some of the mentioned packages are in the same source-folder.
Comment 8 Stephan Herrmann CLA 2020-03-24 07:49:35 EDT
Experience tells us, that problems like this depend on a combination of several (many?) factors. Order could be one of these factors, but certainly JDT does not *generally* fail in specific orders of classes/packages/folders. Almost everything is order independent.

I promise I will analyse this, as soon as I'm able to reproduce. This is not yet given, the flag "needinfo" remains.

So far the characteristics I asked about, don't tell me, where in the implementation I should start looking, but perhaps those bits help you in isolating the problem?

With the little we know about the problem, I suggest the following strategy of pruning:

Copy the affected projects into a fresh workspace, where you can do all kinds of experiments without affecting your actual developement.

If the problem does not show up in the copy, continue making it more and more similar to the original, until it shows up. If this doesn't succeed, smth specific in your original workspace is part of the problem, which cannot be reproduced in a clean workspace, hence chances to find the root cause are diminishing (see further below).

Once the copied workspace shows the problem, gradually remove stuff, that may be unneeded. After each deletion check if the problem still occurs. It may be tricky to remove stuff without causing too many compile errors. This is where knowledge about the architecture of your software should help to find things that can be removed without breaking the rest.

You'll need get a feeling, how big the chunks should be that you remove. Before removing big chunks make a backup (or use a tmp git repo to commit the last "working" state, i.e. a state that shows the bug). If chunks are very small, you'll spend the rest of your life deleting small portions, if chunks are too big, you'll have to revert more frequently. Here it depends: if reverting to a previous "working" state allows you to recreate the problem, then big steps are good, you could even try to simulate some kind of bisection.


Pruning the project until you reach a sharable state is one option.

Another option would be to connect a debugger, so that you can use your existing workspace for observing the failure at code level. Let me know, if you are ready to do this, I will provide some info to jump-start you into this.


Any help in either direction (isolate or debug) is highly appreciated.
Comment 9 Stephan Herrmann CLA 2020-03-24 07:52:32 EDT
I forgot to ask a standard question: we need to be sure which exact version/variant of JDT you are using. For this purpose please let us know the exact version of plugin org.eclipse.jdt.core. Also tell us, if any tooling that employs bytecode weaving is installed (examples are: AspectJ, ScalaIDE, lombok ...).
Comment 10 Al Bundy CLA 2020-03-24 08:06:04 EDT
Eclipse.org	Java Development Tools Core	3.21.0.v20200304-1404	org.eclipse.jdt.core

no AspectJ, ScalaIDE, lombok or similar tools.

I can try to create a separate project to track-down the issue.
Is there some kind of logging that I can activate?
Comment 11 Stephan Herrmann CLA 2020-03-24 09:01:42 EDT
(In reply to Al Bundy from comment #10)
> Eclipse.org	Java Development Tools Core	3.21.0.v20200304-1404
> org.eclipse.jdt.core
> 
> no AspectJ, ScalaIDE, lombok or similar tools.

thanks, looks clean.
 
> I can try to create a separate project to track-down the issue.
> Is there some kind of logging that I can activate?

I'm not sure what exactly you would be searching for, but yes, there are lots of tracing options. IIUC the problem occurs during a clean build, right?

Firstly, launch Eclipse from a console where you can observe the output (or redirect stdout/err to a file).

Then go to Preferences > General > Tracing
1. [x] Enable tracing
2. Open node JDT/Core and select
  - the toplevel org.eclipse.jdt.core/debug -> true
  - which ever sub-option looks promising,
    - /debug/builder should give an overview of what's happening during a build
    - /debug/compiler will give more fine grained info during compilation
(Unfortunately, JDT/Core doesn't seem to use the Output file settings).


Thinking more about it, there's one specific parameter, which might have an influence: 
-DmaxCompiledUnitsAtOnce=2000
Put the above in your eclipse.ini on a line after "-vmargs", then start Eclipse.

2000 is the default value. You might try a higher value on the original workspace, perhaps the problem will disappear (perhaps for the price of higher build time). If you can observe such influence, use a small value for reproducing in a reduced workspace.
Comment 12 Al Bundy CLA 2020-03-24 10:09:37 EDT
first test with enabled tracing while building

After build AbstractMessagingServerTest complains about that AbstractRepositoryClientTest was not found (AbstractMessagingServerTest extends AbstractRepositoryClientTest) even though it exists.
According to the trace AbstractRepositoryClientTest was again created after AbstractMessagingServerTest.

Next step is to try maxCompiledUnitsAtOnce
btw. my small test-project contains 4565 java files and my largest project 27273.

egrep "(AbstractMessagingServerTest|AbstractRepositoryClientTest)" trace.log 
[parsing    /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #2003/2003]
[analyzing  /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #2003/2294]
Writing new class file AbstractMessagingServerTest.class
[completed  /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #2003/2294]
About to compile #2300 : base_package/repository/client/test/AbstractRepositoryClientTest.java
[parsing    /my_test_project/base_package/repository/client/test/AbstractRepositoryClientTest.java - #67/721]
[parsing    /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #722/722]
[analyzing  /my_test_project/base_package/repository/client/test/AbstractRepositoryClientTest.java - #67/722]
Writing new class file AbstractRepositoryClientTest.class
[completed  /my_test_project/base_package/repository/client/test/AbstractRepositoryClientTest.java - #67/722]
[analyzing  /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #722/722]
[completed  /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #722/722]
[parsing    /my_test_project/base_package/messaging/server/AbstractMessagingServerTest.java - #1/1]
Comment 13 Andrey Loskutov CLA 2020-03-24 10:24:00 EDT
Note, to disable batch compilation side effects the argument must be 

-DmaxCompiledUnitsAtOnce=0
Comment 14 Al Bundy CLA 2020-03-24 10:30:13 EDT
Andrey answerd while I was testing/writing

These are my results before Andreys answer:

with no change on maxCompiledUnitsAtOnce always the same classes are not found.
(~30 in my small test-project)

with -DmaxCompiledUnitsAtOnce=10 I have now over 2000 "missing" classes.

With a value of 3000 i still have over 1000 missing classes?!?
--> but this time different classes than "usual".

With a value of 5000 my test-project can be builed without errors.

Next step is to find a value for my largest project - but this takes some time to compile. :-)
it seems that it needs more than 5000. :-/


After Andreys answer I've set -DmaxCompiledUnitsAtOnce=0
 in my small test-project and it was still without errors after build.

now I'll try my largest project but I'm confident that this will work too.
Comment 15 Al Bundy CLA 2020-03-24 10:36:22 EDT
Good news - even my largest project was builded without errors.

So thanks to all for helping.

Was there any change with 2020-03 for maxCompiledUnitsAtOnce?
It seems that this arguments exists at least sind 2015 and I had never such issues.
Comment 16 Andrey Loskutov CLA 2020-03-24 10:58:25 EDT
(In reply to Al Bundy from comment #15)
> Good news - even my largest project was builded without errors.
> 
> So thanks to all for helping.
> 
> Was there any change with 2020-03 for maxCompiledUnitsAtOnce?

The argument itself is not a problem, but most likely incremental compilation has.

Which was your *last known good* Eclipse version? If 4.14, I believe this could be related to bug 559965 changes (the only change we had in /org.eclipse.jdt.core/model/org/eclipse/jdt/internal/core/builder/IncrementalImageBuilder.java).
Comment 17 Al Bundy CLA 2020-03-24 11:01:53 EDT
last working version was indeed 2019-12 / 4.14
Comment 18 Martijn Dashorst CLA 2020-03-24 12:04:33 EDT
Just to add: this issue also presents itself on macOS (and not just Win/Lin as stated above).
Comment 19 Andrey Loskutov CLA 2020-03-24 12:10:17 EDT
*** Bug 561417 has been marked as a duplicate of this bug. ***
Comment 20 Missing name CLA 2020-03-25 14:14:09 EDT
We have the same problem, a series of "cannot be resolved as a type".
Note : full compilation fails, but after compilation, adding a space in failed source file and saving will resolve local problems. This thus looks like a class dependencies problem, eclipse wopuld compile needed classes after the classes needing them.

Option  -DmaxCompiledUnitsAtOnce=0 solves the problem.
Comment 21 Stephan Herrmann CLA 2020-03-26 13:39:12 EDT
While we have collected some relevant information about the bug, I'm still not sure how anyone in the team could possibly debug this.

Hence the primary goal is still for someone affected to create a sharable example project that exhibits the bug. As mentioned before setting a low value (greater than 0) to maxCompiledUnitsAtOnce should help to reproduce even in a smaller project.

As there's a vague hint pointing towards bug 559965, should we provide a stand-in jar file with that fix reverted to test if this fixes the bug? Note that we don't yet have a theory how exactly that fix could be causing this bug. And even if there is a connection, that fix may not be the root cause, perhaps it's just unmasking a bug that has been there before.
Comment 22 Al Bundy CLA 2020-03-26 14:00:49 EDT
I've allready tried to create a testproject to reproduce the issue but at least for now without success.

IncrememtalImageBuilder has two changes in the last few months
https://github.com/eclipse/eclipse.jdt.core/commits/master/org.eclipse.jdt.core/model/org/eclipse/jdt/internal/core/builder/IncrementalImageBuilder.java

The only thing I can say for now is that with a fixed value of maxCompiledUnitsAtOnce I have always the same "missing" classes in my project.

If you want you can provide a modified jar with the reverted changes or some logging.

Than I can try it with one of my projects.
Comment 23 Eclipse Genie CLA 2020-03-26 15:38:50 EDT
New Gerrit change created: https://git.eclipse.org/r/160065
Comment 24 Andrey Loskutov CLA 2020-03-26 15:39:53 EDT
(In reply to Eclipse Genie from comment #23)
> New Gerrit change created: https://git.eclipse.org/r/160065

This is not a fix, but we can merge that and let users disable fix for bug 559965 via system flag. 

WDYT?
Comment 26 Stephan Herrmann CLA 2020-03-26 18:10:46 EDT
(In reply to Andrey Loskutov from comment #24)
> (In reply to Eclipse Genie from comment #23)
> > New Gerrit change created: https://git.eclipse.org/r/160065
> 
> This is not a fix, but we can merge that and let users disable fix for bug
> 559965 via system flag. 
> 
> WDYT?

good idea, here you go:

(In reply to Eclipse Genie from comment #25)
> Gerrit change https://git.eclipse.org/r/160065 was merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> ?id=649f56eb1d0af8d8c57e5b7f2f6e533d2978ab71
Comment 27 Andrey Loskutov CLA 2020-03-27 03:58:14 EDT
(In reply to Andrey Loskutov from comment #24)
> (In reply to Eclipse Genie from comment #23)
> > New Gerrit change created: https://git.eclipse.org/r/160065
> 
> This is not a fix, but we can merge that and let users disable fix for bug
> 559965 via system flag. 
> 
> WDYT?

To all who are affected and can see the bug: please try to download the latest nightly 4.16 build from https://download.eclipse.org/eclipse/downloads/drops4/I20200326-1800

and start this Eclipse build with -DDISABLE_FIX_FOR_559965=true JVM flag (two DD are intended, not a typo).

After that, please do a clean build - and if the problem disappears, we are one step further with understanding the root cause.
Comment 28 Missing name CLA 2020-03-27 10:20:36 EDT
I20200326-1800 with -DDISABLE_FIX_FOR_559965=true does not change anything for us, we still get the compilation errors.
Comment 29 Andrey Loskutov CLA 2020-03-27 10:33:56 EDT
(In reply to Missing name from comment #28)
> I20200326-1800 with -DDISABLE_FIX_FOR_559965=true does not change anything
> for us, we still get the compilation errors.

Damn! It would be too easy :-)

So the only way now to continue is:

(In reply to Stephan Herrmann from comment #21)
> Hence the primary goal is still for someone affected to create a sharable
> example project that exhibits the bug. As mentioned before setting a low
> value (greater than 0) to maxCompiledUnitsAtOnce should help to reproduce
> even in a smaller project.
Comment 30 Eclipse Genie CLA 2020-03-27 10:34:32 EDT
New Gerrit change created: https://git.eclipse.org/r/160098
Comment 31 Missing name CLA 2020-03-27 10:59:24 EDT
Our code is very complex (Millions of lines), so it will be very complex to build an independant example rproducing the problem...

Perhaps a hint for you : we have a series of class crossed dependencies, and this would nicely fit with observed symptoms if the problem is really a dependency one. Perhaps you could try reproducing with corssed dependencies?

Another information: people who do not have access to the whole source do not have the problem. We think that this is due to teh fact that missing classes are found in the JARs we provide to them.
Comment 32 Andrey Loskutov CLA 2020-03-27 11:06:27 EDT
(In reply to Missing name from comment #31)
> Our code is very complex (Millions of lines), so it will be very complex to
> build an independant example rproducing the problem...
> 
> Perhaps a hint for you : we have a series of class crossed dependencies, and
> this would nicely fit with observed symptoms if the problem is really a
> dependency one. Perhaps you could try reproducing with corssed dependencies?
> 
> Another information: people who do not have access to the whole source do
> not have the problem. We think that this is due to teh fact that missing
> classes are found in the JARs we provide to them.

This is interesting. You mean, you have cycles in the compilation between classes? A needs B needs C needs A? I remember, we had some cycle related changes recently. Could you try to identify one example of such cycle, and check if any of the classes with errors were related? Could you also please try to build a simplified example of such dependencies chain? Even if simplified example would compile, may be we will see some similarities with other affected projects?
Comment 33 Missing name CLA 2020-03-27 11:13:59 EDT
For the moment, I do not have any resources to do this... But we are not the only ones with the problem, perhaps others can do that?
Comment 34 Olivier Jaquemet CLA 2020-03-27 11:23:40 EDT
Hello everyone,

After an upgrade to Eclipse 2020-03, several developers in my company observe this bug. 
It always occurs in the same project, on the same classes, for a very large code base which also has dependency cycle during compilation. 

One workaround we have identified is to edit one by one the java classes in which the error is reported, and simply add an empty line and save. It leads to a successful compilation.

We are in the process to test the workaround described before (-DmaxCompiledUnitsAtOnce=0 in eclipse.ini).

Unfortunately we cannot share our project to help in the diagnosis of this issue. We could try to reproduce the dependency cycle leading this issue, but it is a very complex task to isolate.
Comment 35 Missing name CLA 2020-03-27 11:27:39 EDT
I looked a little deeper into our code, nothing is clearly linked to circular dependencies, but instead I noticed that many cases fail when importing static inner classes:
Class A {
  static class B;
}

Failing import is
import A.B;

Perhaps this can help reproducing?
Comment 36 Missing name CLA 2020-03-27 11:33:32 EDT
And when I dig more around, I discover that in previous example, class A is declared as extends MyCollection<B> where B is a static inner class of A!

Even though this is not the only case where we have the error, I would suggest trying reproducing with such a case.
Comment 38 Al Bundy CLA 2020-03-28 09:17:25 EDT
where can we download a modified jar or a nightly version that contains the jar?
Comment 39 Stephan Herrmann CLA 2020-03-28 09:38:59 EDT
Nightly builds of Eclipse SDK can be found at https://download.eclipse.org/eclipse/downloads/index.html
The update URL for so-called I-builds is http://download.eclipse.org/eclipse/updates/I-builds

If you want to test drive the tweak mentioned in comment 27 please select either I20200327-1800 or I20200326-1800.
Comment 40 Al Bundy CLA 2020-03-30 06:27:10 EDT
same here -DDISABLE_FIX_FOR_559965=true did not help :-(

Could it be related to this change?
https://github.com/eclipse/eclipse.jdt.core/commit/62050b74fac53d50e60a594ecb3a5be6d7ba9a70#diff-8cf451fad746ce2f08d613aa58012afa
Comment 41 Till Brychcy CLA 2020-03-30 13:39:57 EDT
Good news: I found a project where I can reproduce this with -DmaxCompiledUnitsAtOnce=20.

I'll find out which change caused it.
Comment 43 Till Brychcy CLA 2020-03-30 14:58:42 EDT
Created attachment 282279 [details]
example project

Happens if a class is references that is references that isn't compiled in the same batch and is in a package that has a non-package subdirectory (i.e. containing only non-java files)
Comment 44 Till Brychcy CLA 2020-03-30 15:03:36 EDT
(In reply to Till Brychcy from comment #43)
> Created attachment 282279 [details]
> example project
> 
> Happens if a class is references that is references that isn't compiled in
> the same batch and is in a package that has a non-package subdirectory (i.e.
> containing only non-java files)

Again I wish bugzilla had an edit button for comments. I meant:

Happens if a class references a class that is compiled in a later batch and is in a package that has a non-package subdirectory (meaning it contains only non-java files).
Comment 45 Till Brychcy CLA 2020-03-30 15:04:15 EDT
I forgot: you have to use -DmaxCompiledUnitsAtOnce=2 with the example project
Comment 46 Missing name CLA 2020-03-30 15:13:13 EDT
I confirm : we have a non-package directory in at least one of failing case.

I don't check all of them, but I feel this is the right way!

I hope this is the only case, let's see after fix.
Comment 47 Stephan Herrmann CLA 2020-03-30 15:27:28 EDT
Many thanks Till, for isolating this!

I'll take a look.
Comment 48 Stephan Herrmann CLA 2020-03-30 17:24:48 EDT
got it.
Comment 49 Eclipse Genie CLA 2020-03-30 18:13:26 EDT
New Gerrit change created: https://git.eclipse.org/r/160219
Comment 50 Stephan Herrmann CLA 2020-03-30 18:32:16 EDT
Using -DmaxCompiledUnitsAtOnce=2 (not =20) here's what I observed:

We ask ClasspathDirectory.isPackage(bug561287.sub)
Since it's actually a ClasspathMultiDirectory we branch into
ClasspathMultiDirectory.listDirectory(bug561287.sub)

This method performs two attempts:
1. does <binaryFolder>/bug561287.sub contain a file that's either a .class file
   or a directory (into which we need to recurse our search)?

2. If (1) produces nothing, we search <sourceFolder> likewise.

But, in (1) we do find one entry: "directory", whereas B0.class doesn't yet exist, that class is still in the build queue. So, we skip (2), but the calling isPackage() determines that "directory" does not constitute a package. Hence we conclude that there is no package bug561287.sub.

bummer.

The gerrit contains a band aid, such that (2) is only skipped if (1) found a class file, not if the only thing we found is a directory.

I'm not sure yet, whether this solution is good, because listDirectory() unfortunately combines a few different things:
- when called from isPackage() and hasCompilationUnit() we're interested only
  in whether or not any Java file can be found inside.
- when called via doesFileExist() we are interested in one particular class file.
- the result of the initial listDirectory() is cached for later.

Up until bug 547181 listDirectory() only ever looked into binaryFolder, but with the option to look into both source and binary, it's less clear to see, if looking half here and half there could produce (and cache) inconsistent state.
Comment 51 Stephan Herrmann CLA 2020-03-30 19:01:43 EDT
(In reply to Stephan Herrmann from bug 547181 comment #37)
> - improve ClasspathMultiDirectory.directoryList for the case that .class
> files have not yet been created.


Apparently, I was fighting a somewhat similar situation back then.

Stepping back a little, what is the business about looking in source/binary folders?

Most the time we are *not* interested in particular files, only in whether or not there's some Java (.class or .java) file somewhere to be found.

The only caller interested in specifics is ClasspathDirectory.doesFileExist(), which always works on behalf of some flavor of findClass(). It looks like call paths into ClasspathDirectory.doesfileExist() are actually only interested in .class files. The call paths concerned about source files are only taken when (location instanceof ClasspathSourceDirectory). 
Now, if doesFileExist() is only interested in .class files, then additional lookup in sourceFolder cannot change the outcome of doesFileExist().

I was tempted to add a parameter to directoryList() to discriminate whether or not we are interested in specific files, or just a general yes or no, but considering that the result of the first scan is cached, that scan has to be complete anyway.

Next, I wondered, if the current round of compilation could cause additional confusion with .class files incrementally entering the picture. But if the compiler produced a given .class file during the current round, then it will never ask the NameEnvironment if it exists, because it's still known in LookupEnvironment.

Ergo, lookup in binaryFolder is only interesting for incremental compilation to be able to see class files produced by a previous compiler invocation.
Comment 52 Stephan Herrmann CLA 2020-03-30 19:57:35 EDT
@Till, I guess my hand-waving above illustrates that I'm puzzled about why a simple directory scan causes so much confusion.

Actually, your fix https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=32a3857797bf19248aee13ba9b4a02f63d66f233 touches the exact same code area for a very similar purpose.

So, I'd be more than happy, if you can give your thoughts also here.
Comment 53 Martijn Dashorst CLA 2020-03-31 02:37:18 EDT
I can confirm that our compile errors also reference a class in a package with a subpackage without Java code.
Comment 54 Eclipse Genie CLA 2020-03-31 02:42:55 EDT Comment hidden (obsolete)
Comment 55 Till Brychcy CLA 2020-03-31 02:43:58 EDT Comment hidden (obsolete)
Comment 56 Till Brychcy CLA 2020-03-31 03:28:17 EDT Comment hidden (obsolete)
Comment 57 Till Brychcy CLA 2020-03-31 03:40:11 EDT
(In reply to Stephan Herrmann from comment #51)
>  The call paths concerned about source files are only taken when
> (location instanceof ClasspathSourceDirectory). 

I think the source files are needed here for org.eclipse.jdt.internal.core.builder.ClasspathDirectory.isPackage(String, String) and your patches fixes that correctly.
Comment 59 Stephan Herrmann CLA 2020-03-31 08:16:17 EDT
(In reply to Till Brychcy from comment #57)
> (In reply to Stephan Herrmann from comment #51)
> >  The call paths concerned about source files are only taken when
> > (location instanceof ClasspathSourceDirectory). 
> 
> I think the source files are needed here for
> org.eclipse.jdt.internal.core.builder.ClasspathDirectory.isPackage(String,
> String) and your patches fixes that correctly.

Thanks, Till.

(In reply to Eclipse Genie from comment #58)
> Gerrit change https://git.eclipse.org/r/160219 was merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> ?id=a73b1a7fffb30d9caa292d9b62a6eaaac4b616ef

Released to master for 4.16 M1.

(Commit comment still says "band aid", which sounds a bit more negative than needed. Still it may remind us, that this code area is a bit fragile).
Comment 60 Al Bundy CLA 2020-03-31 09:27:05 EDT
can we test this fix with https://download.eclipse.org/eclipse/downloads/drops4/I20200330-1800/ or do we have to wait for tomorrow?

Is there any chance to get an updated 4.15?

@Till: I'm just curious - how did you find the relationship between the references and non-package subdirectories?
I've also tried different referenced packages but I would never have thought of creating a non-package-directory
Comment 61 Stephan Herrmann CLA 2020-03-31 09:44:02 EDT
(In reply to Al Bundy from comment #60)
> can we test this fix with
> https://download.eclipse.org/eclipse/downloads/drops4/I20200330-1800/ or do
> we have to wait for tomorrow?

The fix has just been merged, it will be contained in the next build.
 
> Is there any chance to get an updated 4.15?

Officially: no, management has decided that quarterly releases is the best we can do. Working on maintenance branches requires efforts that we can't afford.

That said, there have been situations, where volunteers have provided individual update sites with a patch to fix just one bug. We could give 1 or 2 hints, but please don't expect the team to drive this.

Hopefully, an I-build or perhaps the upcoming milestone 4.16 M1 is a workable solution for those affected by the bug.
 
> @Till: I'm just curious - how did you find the relationship between the
> references and non-package subdirectories?
> I've also tried different referenced packages but I would never have thought
> of creating a non-package-directory

Yep, even when considering that Till has insider knowledge: isolating this was a master piece!
Comment 62 Al Bundy CLA 2020-03-31 10:25:27 EDT
@Stephan: I really understand this but sometimes it's not possible to use milestone-builds (e.g. due to company policies).

Because I think that also others are interessted in maintenance-releases I've created a new report https://bugs.eclipse.org/bugs/show_bug.cgi?id=561613 for discussion.
If there is really no way for maintenance releases than we have to live with that decision.

@Till: I'm still curious. ;-)
Comment 63 Till Brychcy CLA 2020-03-31 11:49:08 EDT
(In reply to Al Bundy from comment #60)
> @Till: I'm just curious - how did you find the relationship between the
> references and non-package subdirectories?
> I've also tried different referenced packages but I would never have thought
> of creating a non-package-directory

As Stephan had already the right idea that this was related to maxCompiledUnitsAtOnce, I simply tried with a reduced value if I can find an example project that allows me to reproduce and I succeeded (see comment 41).

Once I had that, it was easy (I'm always working with a nested eclipse launched from sources, so going back to git commits and debugging is easy for me)

I copied it into a fresh workspace to speed up bisecting which change introduced the bug. 

Once I had that (see comment 42), I set some breakpoints with conditions that check for the involved package names, stepped a bit through the code and did some local changes to narrow down the involved code paths. Then I noticed a referenced directory had no java files and remembered that the bug for which the change was introduced was about that.

Then I enabled logging to figure out in which order the involved classes are compiled and based on that tried to synthesize the example project, which luckily worked right away.

(In reply to Stephan Herrmann from comment #61)
> Yep, even when considering that Till has insider knowledge: isolating this
> was a master piece!

It's nice that you write this, but I was just lucky to find an example. Wouldn't have happened if you hadn't thought of maxCompiledUnitsAtOnce.
Comment 64 Al Bundy CLA 2020-03-31 17:19:32 EDT
@Till: thanks for explanation - i thought you found it with enabled tracing or other mechamisms. 
knowing the sources and the ability to debug them is kind of cheating. ;-)
But in this case probably the best and fastest method.

Thanks for tracking this down.
Comment 65 Andrey Loskutov CLA 2020-04-02 14:06:37 EDT
Can anyone affected by this bug verify thst the fix is working? Just to make sure we don't have another issue here.
Comment 66 Missing name CLA 2020-04-03 03:09:09 EDT
Sorry, I missed something : where can I get the version? (I'm not used obtaining other versions than standard ones...)
Comment 67 Al Bundy CLA 2020-04-03 03:18:30 EDT
I've downloaded eclipse-SDK-I20200402-2250-macosx-cocoa-x86_64.dmg and imported my test-project in a fresh workspance.
No compile-errors after first build or a manual clean.

So bug seems to be fixed.
Comment 68 Al Bundy CLA 2020-04-03 03:19:50 EDT
@Missing name: https://download.eclipse.org/eclipse/downloads/
I've used: I20200402-2250
Comment 69 Stephan Herrmann CLA 2020-04-03 04:22:20 EDT
(In reply to Al Bundy from comment #67)
> I've downloaded eclipse-SDK-I20200402-2250-macosx-cocoa-x86_64.dmg and
> imported my test-project in a fresh workspance.
> No compile-errors after first build or a manual clean.
> 
> So bug seems to be fixed.

Thanks for confirming, marking as verified.
Comment 70 Missing name CLA 2020-04-03 06:41:06 EDT
It works for me too. But as I understand, it's not a good idea to work with this verison, and we shall wait for next stable release?
Comment 71 Al Bundy CLA 2020-04-03 06:46:10 EDT
you can still use the current release (2020-03) and add
-DmaxCompiledUnitsAtOnce=0
to your eclipse.ini

To be honest - I don't see any difference in performance between
2020-03 with and 2020-06 without this key.
Comment 72 Missing name CLA 2020-04-03 08:05:45 EDT
Neither do I!
Comment 73 Alexander Veit CLA 2020-05-20 09:51:48 EDT
Same problem here. With Eclipse 2020-03 our large workspaces do not compile anymore. Classes are not found even when they are defined in the same project.

-DmaxCompiledUnitsAtOnce=0 seems to fix the issue.

The projects in the workspaces usually have the following structure:

/lib/                 JARs (optional)
/src/main/java/       Java packages
/src/test/java/       Test classes
/src/test-util/java/  Classes that are used by tests in the same project and/or tests in dependent projects.