Bug 537327 - Various build issues migrating a big workspace to Java 9
Summary: Various build issues migrating a big workspace to Java 9
Status: NEW
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.8   Edit
Hardware: PC Windows 10
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: JDT-Core-Inbox CLA
QA Contact:
URL:
Whiteboard: stalebug
Keywords: needinfo
Depends on:
Blocks:
 
Reported: 2018-07-24 05:34 EDT by M H CLA
Modified: 2022-09-24 05:14 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description M H CLA 2018-07-24 05:34:57 EDT
I am aware of the changed module system of Java 9. I use Eclipse Photon with Java 9, configured Java 9 as the "installed JRE". Everything works fine as long as I leave the "Compiler compliance level" to "1.8".

But when I change "Compiler compliance level" to "9", Eclipse compiles the Java projects and prints errors like:

"The type javax.activation.DataHandler cannot be resolved. It is indirectly referenced from required .class files"

I also have custom ANT targets and compiling the very same project with Java 9 with an ANT target works flawlessly if I add the VM argument "--add-modules=ALL-SYSTEM". Compiling in Eclipse with Java 8 and running my apps with Java 9 VM also work with "--add-modules=ALL-SYSTEM" (set up in Eclipses run configuration Arguments).

So I tried to add "--add-modules=ALL-SYSTEM" to my Eclispe start script and also to 

Java > Installed JREs > myJRE > Edit > Default VM arguments

and restarted Eclipse - but the errors do not vanish! So, my ANT targets and Eclipse run configurations run with the additional argument "--add-modules=ALL-SYSTEM". But Eclipse compilation for compliance level 9 does not work.

What is wrong with compliance level 9 and VM arguments "--add-modules=ALL-SYSTEM"?
Comment 1 M H CLA 2018-08-06 01:19:05 EDT
Changed the title as some moght think this is a stupid Java 9 rookie question. It's a problem with the Eclipse compiler and command line at

Java > Installed JREs > [Java9or10JRE] > Edit > Default VM argeuments: "--add-modules=ALL-SYSTEM"

as this does not seem to be used. (Very same argument on Eclipse run configuration or ANT target build file works.)
Comment 2 Stephan Herrmann CLA 2018-08-06 10:42:26 EDT
I assume your code does not have module-info.java, right? Otherwise, just requiring the desired module should be enough.

If indeed, your code is in the unnamed module, then please try the following:
- in your project's Build Path configuration
  - go to the Libraries tab
    - open the "JRE System Library" node
      - select its child "Is modular" and click Edit.
        - In that dialog see that tab "Contents" is active, then
          - Find the desired module under "Available modules" 
            and move it to "Explicitly included modules" 
            (double click or right-arrow button).
- build your project

Background: while obviously the JDT batch compiler should support the same command line options as javac, this doesn't hold for the case where the compiler is invoked via the IDE, which is not a command line invocation.
Instead, we provide UI for the relevant options, where the mentioned "Contents" tab combines the --add-modules and --limit-modules options.

Passing --add-modules=ALL-SYSTEM to the JRE will only be interpreted by the JVM, not the compiler. So this would help for running your programs, but once you've configured everything in the mentioned dialog, the appropriate options should automatically be passed to the target VM.

HTH
Comment 3 M H CLA 2018-08-07 04:26:19 EDT
Yes, I just switched the JRE - my Java projects don't have any Java 9 changes and therefore also no "module-info.java".

I tried your steps, but didn't know what module is "javax.xml.soap.MimeHeaders" in? Checking "javax...." entries in the Contents tab:

Available modules: <none>
Explicitly included modules: <none>
Implicitly included modules: <none>

I searched the JAR with the "missing" class and found saaj-api.jar, continued adding JARs to the classpath for further missing type errors.

... until org.w3c.dom.Node : also added xml-apis.jar which contains this class to the project's classpath but Eclipse keeps complaining "The type org.w3c.dom.Node cannot be resolved. It is indirectly referenced from required .class files". Did this several times, also restarted Eclispe, double checked project's classpath.
Comment 4 M H CLA 2018-08-07 05:26:59 EDT
Eclipse seems unstable with compiler compliance level 9: every now and then, it complains about different types/classes. E.g.

 javax.xml.soap.AttachementPart.setBase64Content()

"The method setBase64Content(ByteArrayInputStream, String) is undefined for the type AttachmentPart".

When I Project > Clean > All projects, this error goes away ... and complains about a related project :

"The project cannot be built until its prerequisite iCAMC is built. Cleaning and building all projects is recommended"

I just _did_ clean _all_projects! When I clean the project alone, the above error reappears ... oh boy, what is happening here? This blocks us from changing to Java 9/10 because we can't get those errors fixed to compile ourr existing projects.

At least: switching back to compiler compliance level 1.8, everything works again (*phew*).
Comment 5 Stephan Herrmann CLA 2018-08-07 07:36:38 EDT
(In reply to M H from comment #3)
> Yes, I just switched the JRE - my Java projects don't have any Java 9
> changes and therefore also no "module-info.java".

OK

> I tried your steps, but didn't know what module is
> "javax.xml.soap.MimeHeaders" in?

This is what I do: Open the desired type (Ctrl+Shift+T), then enable breadcrumbs in the Java editor[1]. Within breadcrumbs hover over the jar icon (should be module icon, oops), it says "java.xml.ws".

> Checking "javax...." entries in the
> Contents tab:
> 
> Available modules: <none>
> Explicitly included modules: <none>
> Implicitly included modules: <none>

Did you manually search the given lists of modules for the prefix "javax"? If so, then it is correct that you found nothing, because none of the module names start with javax (module names don't directly correspond to package names).

> I searched the JAR with the "missing" class and found saaj-api.jar,
> continued adding JARs to the classpath for further missing type errors.

Is this still about the hunt for MimeHeaders? If so, that hunt shouldn't be immediately needed, once you add java.xml.ws to explicitly included modules, OK?

Long term, you should still go for such 3rd party implementations, because module java.xml.ws is deprecated for removal.

BTW: I suggest searching places like StackOverflow for recommended replacement strategy for your particular case.
 
> ... until org.w3c.dom.Node : also added xml-apis.jar which contains this
> class to the project's classpath but Eclipse keeps complaining "The type
> org.w3c.dom.Node cannot be resolved. It is indirectly referenced from
> required .class files". Did this several times, also restarted Eclispe,
> double checked project's classpath.

Node is part of the module java.xml, which is included by default. I'm not exactly sure what your configuration was at this point. Might be worth uploading a minimal demo project to show this scenario.

[1] http://help.eclipse.org/photon/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2Freference%2Fref-java-editor-breadcrumb.htm
Comment 6 Stephan Herrmann CLA 2018-08-07 07:38:52 EDT
(In reply to M H from comment #4)
> Eclipse seems unstable with compiler compliance level 9: every now and then,
> it complains about different types/classes. E.g.
> 
>  javax.xml.soap.AttachementPart.setBase64Content()
> 
> "The method setBase64Content(ByteArrayInputStream, String) is undefined for
> the type AttachmentPart".
> 
> When I Project > Clean > All projects, this error goes away ... and
> complains about a related project :
> 
> "The project cannot be built until its prerequisite iCAMC is built. Cleaning
> and building all projects is recommended"
> 
> I just _did_ clean _all_projects! When I clean the project alone, the above
> error reappears ... oh boy, what is happening here? This blocks us from
> changing to Java 9/10 because we can't get those errors fixed to compile
> ourr existing projects.

Oh boy, indeed. Bad news: I don't have any of these problems. So, any help you can give us in reproducing the trouble would be highly appreciated!
Comment 7 M H CLA 2018-08-08 02:01:34 EDT
As I am a developer I understand the request for "reproducable demo", but this is our production working set contains 14 Java projects (each leading to 1 JAR) with misc dependencies and several hundred classes with 60 third party JAR libraries referenced.

I develop Java since the first (beta) release back in 1997 but this is the worst change ever! As I mentioned above, I solved most type errors by searching for JARs on the internet and adding them to the misc project classpaths. This is a bad upgrade as Oracle should at least offer a migration guide that lists all projects/JARs that are not in the standard classpath anymore and gives links to the replacement JARs that should be included. (I don't know if the JARs I found are a "good" replacement for the former Java 8 internal classes.)

Anyway, to not lose track of Java and continue to get the latest versions / support, we have to pass the migration to full Java 9 including compiler compliance level 9.

I also know Eclipse for over 15 years and always had (with each version!) synchronisation problems. Every now and then Eclipse shows compiler errors (red marks in the project tree) that could be fixed by executing a "Project clean" for all projects which I find quite annoying - but I get used to it.

No with compiler compliance level 9, I get this strange errors that "jump" betwenn 2 classes when I clean the single project or all projects in the working set. And both erros are not "true" as teh JARs are in the projects classpat:


A) clean error project:

"The method newInstance() in the type MessageFactory is not applicable for the arguments (String)"

B) clean all projects:
"The project was not built since its build path is incomplete. Cannot find the class file for org.w3c.dom.Node. Fix the build path then try building this project"
"The type org.w3c.dom.Node cannot be resolved. It is indirectly referenced from required .class files"


What I can't understand: these errors toggle when I toggle project cleaning between the single project and all projects - which makes no sense to me.
Comment 8 M H CLA 2018-08-08 05:25:35 EDT
I go crazy: after my tests, I switched back to compiler compliance level 1.8 to continue working. It worked for a while but suddenly (after a real code error leading to compiler error), Eclipse went crazy: I did clean Project for all projects and now most projects produce errors:

"The project cannot be built until its prerequisite <...> is built. Cleaning and building all projects is recommended."

I built/cleaned all projects => no success!

One warning is

"Incompatible .class files version in required binaries. Project 'iCImport' is targeting a 1.8 runtime, but is compiled against 'C:Program Files/JDK_current/lib/jrt-fs.jar' (from the container 'JRE System Library [JDK_current]') which requires a 9 runtime"


So I recompiled all projects with my Eclipse ANT targets (each with fixed Java target 1.8) and all errors went away ... until I did the next Eclispe Project > Clean ... > Clean All Projects ...!!! Even though I switched compiler compliance level back to 1.8 (no project specific settings!).

I'm starting to hate Eclipse ... :-/
Comment 9 Stephan Herrmann CLA 2018-08-08 06:49:39 EDT
M H, let's please try to distinguish a few things:

Java 9 is not designed by the Eclipse team, we have our own troubles with those changes, but we do our best to help users to migrate to that version. For the most of this you and me are in the same boat here.

As for a reproducing example, I'm aware that a typical production workspace cannot be shared publicly, but most problems can be reproduced by a reduced workspace, and to spend our time efficiently we tend to focus on those bug reports that have steps for reproducing.
Obviously reducing your workspace can be quite an effort. Just to remind you of the two options: Make a copy of your workspace, and remove parts until the problem disappears, or: start with an empty workspace and *add* parts from your real workspace until the error appears - usually it will be a mix of both.

BTW: learning Java 9 by flipping the switch on a big production workspace may actually be a less-then-optimal idea, independent of tooling, because migrating to Java 9 is a huge step - for any project. May I suggest starting small with just a few projects?

As of today, I have no real idea what is going on on your machine so all I could do is close as "worksforme", which obviously is a bad conclusion. So we need to meet in the middle, you need to give us a chance to understand what is happening, then we will investigate the issue in depth.

I understand that errors jumping from one location to another is not a good user experience, but before we understand those errors in the first place, we can't really comment (or investigate) on that jumping.

As you mention ANT: may I conclude that your projects don't use Maven or Gradle?   Are your ANT scripts executed automatically during Eclipse build (via Ant Builder configured for your projects), or do you manually invoke those? Reason for asking: interaction between different builders is a typical suspect for non-deterministic build behavior.

We had reports indicating that configuring Java compliance for the entire workspace has no immediate effect or may lead to inconsistencies. Until we figure out that issue, you may get more reliable / reproducible behavior by explicitly setting Java compliance for each individual project. This in turn is less painful if you start with only a few projects ...
Comment 10 M H CLA 2018-08-09 01:27:41 EDT
Understood. No, we use the ANT script only manually (double click in ANT view; no ANT builders in project's "Builders").

So, I did what you suggested: switched only 1 project to compliance level 9:

"The method newInstance() in the type MessageFactory is not applicable for the arguments (String)"

for the code line

final MessageFactory mf = MessageFactory.newInstance(protocol);

I can't see any change in the Java 9 doc for this API. The real strange thing is: there is no error when compiling with the ANT script with "target" 9:

    <target name="-compile.debug9" depends="-classes.delete" unless="isPreJava9" description="compile all java sources">
        <echo level="info" message="CLASSPATH = ${classpath}"/>
        <javac compiler="javac${javaTargetVersion}" source="${javaTargetVersion}" target="${javaTargetVersion}" srcdir="${src}" classpath="${classpath}" bootclasspath="" encoding="${encoding}" destdir="${classes}"  includeantruntime="true" debug="on" optimize="off">
            <compilerarg value="${jvm9Arg1}"/>
            <compilerarg value="${jvmArg3}"/>
        </javac>
    </target>

Don't you agree, that here is a problem in Eclipse?
Comment 11 M H CLA 2018-08-09 02:57:52 EDT
I tried to somehow get further and found that this method call (with the String parameter) : 

MessageFactory.newInstance(protocol);

is not used in any our other projects, so I commented it out. The project then compiled with compliance level 9. Then I switched the general compliance level to 9: all projects compiled except one with 1 error:

"The method setBase64Content(ByteArrayInputStream, String) is undefined for the type AttachmentPart"

for line

ap.setBase64Content(new ByteArrayInputStream(contentXMLBase64.getBytes()), "application/octet-stream");

Unfortunately, I can't remove/comment out this line, too. And the Javadoc API also don't tell about any changes in Java 9.

Interestingly, both lines with error complain about a javax.xml.soap class. And again: the same project compiled from the ANT script (same classpath) works without error.

However, I noticed one difference: my ANT scripts set compiler arg "--add-modules=ALL-SYSTEM" with javaTargetVersion "9". So I removed this parameter, added the missing JARs to my classpath and the final last error from the ANT script remains:

error: cannot find symbol
ap.setBase64Content(new ByteArrayInputStream(contentXMLBase64.getBytes()), "application/octet-stream");

with the mark pointing to the dot after "ap" ... which is strange for the source code

final AttachmentPart ap = soapRequest.createAttachmentPart();
ap.setBase64Content(new ByteArrayInputStream(contentXMLBase64.getBytes()), "application/octet-stream");

So, removing "--add-modules=ALL-SYSTEM" from the ANT script also shows an error at the same code line but with a (totaly) different error message.

As "--add-modules=ALL-SYSTEM" is only a workaround and also won't be supported in next Java versions, I try to avoid such short term workarounds. But it also shows that Eclipse general VM args doesn't work (you explained it above which I didn't understand). And I can't find in Eclipse a config where to set project specific compiler VM arguments.

Anyway: in the current state, both compiles (Eclipse and ANT script) shows error at the same code line but with different error messages - and both messages seem wrong to me (the code works fine with compliance level 8).
Comment 12 M H CLA 2018-08-09 07:12:54 EDT
Too bad, you can't edit comments in bugzilla ... : I seem to have solved it: inspecting both error messages, they are not so different; both seem to mean the same that the method AttachmentPart.setBase64Content(InputStream, String) is invalid.

With your hint CTRL+SHIFT+T, I inspected the javax.xml.soap.AttachementPart and the window displayed misc matches. In the saaj-api.jar I noticed an older class version. Which let me thought that the former Java 8 internal class has a newer version. So I searched for a "latest" saaj-api.jar (http://download.oracle.com/otndocs/jcp/jaxm-1.3-mrel-spec-oth-JSpec/) and added this to my classpath. Then the error vanished in both Eclipse and ANT.

And the final one was harder: Eclipse complained about org.w3c.dom.Node which _was_ in the classpath with xml-apis.jar. The error vanished after I added jaxb-api-2.3.0.jar ... because compiling with ANT it wrote a clearer error message:
"class file for javax.xml.bind.annotation.XmlAccessType not found". As this is a node _attribute_ type Eclipse seems to display only a missleading node error.


Conclusion:

1. set compiler compliance level to "9".

2. avoid setting parameter "--add-modules=ALL-SYSTEM" (Eclipse and ANT build files)  <= to be future proof

3. find all additional required and correct (!) JAR replacmeent files and add them to the classpath. <= hard part!

4. build/cleaned single/all projetcs in Eclipse so many times until there is no error in Eclipse anymore. <= annoying part


The last point is important as Eclipse every now and then gets out of sync and I have to build/clean several times single projects/all projects. I wonder how you can set the ORDER of the projects within a working set. As our projects within the working set depend on each other, I probably have to single built/clean the projects in the correct order. => this would be a request for enhancement for Eclipse: offer an order of the working set content projects. ;-)

Thanks for the valuable hint CTRL-SHIFT+T (to find JARs for types in the breadcrumbs) and for your patience!

As the final conclusion for Eclipse I have these issues/requests:

1. awkward use of Installed JRE > Default VM arguments. (I still don't understand how this is process in Eclipse - a developer would expect to have the same effect as in Eclipse run configurtaion.)

2. offer order of projects in a working set, as the projects can depend on each other. (To avoid mutliple manual builts/project cleans in the correct order)

3. better error message for the case "Node" I described (Error message said org.w3c.dom.Node was missing, but the annotation class javax.xml.bind.annotation.XmlAccessType was missing - strange, but I guess it has something to do with something in the Node class).

*phew*! I hope that upgrading to Java 10 won't be that hard. ;-)
Comment 13 M H CLA 2018-08-16 03:12:48 EDT
After I got my existing Java projects to compile, I now try to convert our Java projects to "Modules" via

project > configure > create module-info.java

Unfortunately, the whole classpath/modulepath problems seem to start again. Lots of compiler errors. E.g. 

   import javax.xml.soap.SOAPException;
   "The type javax.xml.soap.SOAPException is not accessible"

Even though the javax.xml.soap-api.jar is in the project's modulepath (also tried classpath). I also added

    requires javax.xml.soap;

to module-info.java, but this brings "javax.xml.soap cannot be resolved to a module". All other "requires" entries have warning "Name of automatic module 'avalon.framework' is unstable, it is derived from the module's file name"; but these are only warnings and not errors.

And the project compiles fine without the "module-info.java". Double checked that "javax.xml.soap-api.jar" is in the modulepath.
Comment 14 Stephan Herrmann CLA 2018-08-26 13:44:04 EDT
I'm reducing the severity to normal, as the original issues seem to have been sorted out by now.

M H, if you want us to investigate further, please let's focus on one particular issue in this bug report (and possibly create other reports for other problems).

Which one should it be? What hints can you give that would allow us to reproduce the problem?
Comment 15 Eclipse Genie CLA 2020-09-23 01:18:30 EDT
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.
Comment 16 Stephan Herrmann CLA 2020-09-23 08:54:26 EDT
For posterity, I just stumbled upon this:

(In reply to M H from comment #13)
> Even though the javax.xml.soap-api.jar is in the project's modulepath (also
> tried classpath). I also added
> 
>     requires javax.xml.soap;
> 
> to module-info.java, but this brings "javax.xml.soap cannot be resolved to a
> module".

If you have a version of this jar that does not contain module-info.class, then you will be out of luck because file name javax.xml.soap-api.jar does not allow deriving a module name for an automatic module. '-' is not supported by the specified algorithm.
Comment 17 Eclipse Genie CLA 2022-09-24 05:14:14 EDT
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.