Community
Participate
Working Groups
While trying to find differences between old IBM and new eclipse.org builds, I noticed the following bundle is missing from the latest 3.8 I-build in the 3.8-I-builds repository: org.eclipse.jdt.compiler.batch
It looks like this is breaking the Orion build. We include the batch compiler in Orion because it is needed by jasper for compiling JSPs.
I'll give this my top priority. (if my hunch is correct, it'll be an easy fix ... if my hunch is wrong, I have no idea :) .
Well, my first hunch wasn't correct. I'd added some "slicing" options to one of the p2.mirrors to avoid pulling in all of EMF versions, ... but, turns out that was very specific to EMF. I do see it being "created" (existing) in the "build repo" ... not sure why its not making its way into the final one. Such as /opt/public/eclipse/eclipse4I/build/supportDir/src/reposource/plugins has org.eclipse.jdt.core.compiler.batch_3.7.0.I20120423-1323.jar org.eclipse.jdt.core.compiler.batch.source_3.7.0.I20120423-1323.jar
Interesting historical note ... the part of the build that looks like it should be doing the work, is related to a PDE "product" and has a bug entry in a comment, marked as "won't fix" ... from not that long ago: bug 330240 closed 11/8. Must be somewhere else now ... or, else that "presetup" was still supposed to run, even if end product not put in the build/repo. Just history. Probably unreleated.
More data, I see the versions of jdt.core and the batch compiler are different (and different "style") ... I wonder if releated? org.eclipse.jdt.core_3.8.1.v20120417-0927.jar org.eclipse.jdt.core.compiler.batch_3.7.0.I20120423-1323.jar But they are listed that way in the content.xml file. Odd thing, is I don't see anything that "requires" the the batch compiler, (none of the "master features" so not sure how it ever gets into the final repo ... doesn't seem to be mentioned "by name" anywhere. We have removed things like "p2.installer" and "OSGI starter kit" ... but, I'd think not related to those.
Well, the good news is, this has been in the repo for several of the last N builds, and was in a 4.2 I test build I did earlier today, so ... maybe it has "fixed itself"? :) /shared/eclipse/eclipse4I/siteDirTESTONLY/updates/4.2-I-builds $ find ./ -regex ".*batch_.*\.jar" ./I20120423-1323/plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.I20120423-1323.jar shared/eclipse/eclipse4N/siteDir/updates/4.2-N-builds $ find ./ -regex ".*batch_.*\.jar" | sort ./N20120420-2153/plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.N20120420-2153.jar ./N20120421-2021/plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.N20120421-2021.jar ./N20120422-2000/plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.N20120422-2000.jar ./N20120423-2000/plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.N20120423-2000.jar
Just to give some history, I believe this bundle is a "synthetic" bundle created entirely by the build process. I.e., there is no such bundle in source control. It is really just a packaging of the Eclipse batch compiler for consumption by other projects. It is not used by the SDK but other projects use it for JSP compilation (such as Jasper). The reason it has a different qualifier is that since it is generated by the build and not in any map files, it just gets the build timestamp every time.
(In reply to comment #6) > Well, the good news is, this has been in the repo for several of the last N > builds, and was in a 4.2 I test build I did earlier today, so ... maybe it has > "fixed itself"? :) Great! I had a workaround where I fetched that one bundle from an old repository, but if it is in this week's 4.2 I-build I will remove my workaround.
Curiouser and curiouser ... the batch compiler _artifact_ is not in tonight's 4.2 I-build repository (I20120424-1700) but it is in tonight's 4.2 N-build repository (N20120424-2000). Worse, sort of, the 4.2 I build content.jar/xml file says that is has it, but, it is not in the artifacts.jar/xml file. Makes me think its something related to signing/conditioning ... since that's about the only difference between N and I builds.
During the "final mirroring" (where we use the comparator, etc.) the batch compiler (and its source) are the only two with MD5 checksum errors: Problems downloading artifact: osgi.bundle,org.eclipse.jdt.core.compiler.batch,3.7.0.I20120424-1700. MD5 hash is not as expected. Expected: 1271e89a9358f6b88a7eb4ca0553d9b1 and found b76ae299f057059b3c3736fec914fcf7. While we have "ignore errors" I think that just means and error doesn't stop the whole process ... and it just refuses to mirror that one (though, surprised it'd end up in the content.xml file). The eclipse.inf file in the batch compiler ends up saying, which is the same thing the 3.8M6 version said: #Processed using Jarprocessor jarprocessor.exclude.children = true jarprocessor.exclude.sign = true pack200.args = -E4 pack200.conditioned = true I wonder if the recent signing process changes (with changes made for signing individual jars, making pack optional, etc.) changed at all what actually happens during the process? That is, that it is now conditioned, but maybe used to not be ...? In fact, it seems like if this jar is not going to be signed, it should also be not conditioned, and the jar processor options should be, simply, #Processed using Jarprocessor pack200.args = -E4 jarprocessor.exclude = true That would avoid signing and conditioning for the whole jar ... but, it may be that no one explicitly creates this eclipse.inf, but perhaps it is implicitly created during the signing process? If that is the case, then the "parent jar", perhaps, should say jarprocessor.exclude.children.pack jarprocessor.exclude.children.sign ?
I see in jdt.core project, besides the main eclipse.inf, which says jarprocessor.exclude.children=true I see another one in /org.eclipse.jdt.core/antadapter/META-INF/eclipse.inf which says jarprocessor.exclude.sign=true jarprocessor.exclude.children=true If this is the one being used while building the batch compiler, then it should say jarprocessor.exclude=true jarprocessor.exclude.children=true to also avoid "conditioning". I do also see, in the jdt.core customBuildCallbacks.xml method some variables with names similar to some in the build scripts ... basedir, postingDirectory ... so if you are assuming the build scripts remaining constant ... well, .... then I could have broken that assumption.
Dani, Adding you to CC for comment from JDT. Any chance you remember why /org.eclipse.jdt.core/antadapter/META-INF/eclipse.inf has jarprocessor.exclude.sign=true instead of jarprocessor.exclude=true I think the only time you'd want to exclude from signing, but not exclude from conditioning, is if you you were going to sign with some other certificate later on, which seems unlikely. I think maybe all along this batch processor has been being not signed, but conditioned, and it's been "worked around", which I have now likely broken (or, really might have been the process at the Eclipse Foundation, since, apparently, even for 3.8 M6 the batch compiler was conditioned by the jar signer, which doesn't seem right).
I should mention, this would be a hard jar to add to the overall "exclude from signing and packing" pack.properties we have to avoid resigning and conditioning jars such as from Orbit, because it's qualifier, apparently, changes each build (which isn't great either ... not sure how we ended up there?)
I should also mention ... there are "known" problems with conditioning and I have seen cases before where some relatively normal code changes has caused code to "break" going through the -repack sign pack process ... so ... it also seems possible this jar has always been "conditioned" in the past with no problems but now that part of pack200 does not work well with the current code. Just a possibility. There might be ways you could investigate that (using one of the pre-signing-process versions of the jar from build machine and running through the pack200 -repack step yourself ... but ... not sure its worth it. Since we don't sign it, our first approach should be to not have it conditioned either. IMHO
(In reply to comment #7) > Just to give some history, I believe this bundle is a "synthetic" bundle > created entirely by the build process. I.e., there is no such bundle in > source control. As David already found out, this is not 100% true. JDT Core has a custom builder that builds the batch compiler post.compile by calling: /org.eclipse.jdt.core/scripts/export-ecj.xml Inside jdt.core there's also the manifest and inf files for the source and the binary bundle. I filed bug 377598 to fix the versions. (In reply to comment #12) > Dani, > > Adding you to CC for comment from JDT. > Any chance you remember why > /org.eclipse.jdt.core/antadapter/META-INF/eclipse.inf > > has > > jarprocessor.exclude.sign=true > > instead of > > jarprocessor.exclude=true We never signed the batch compiler because it was too much work to get it done, see bug 356382 for more details . Form a JDT perspective I'm fine if we sign it, assuming you can make that happen during the build. The conditioning is needed to get the pack200 compression. The patches in bug 315844 might also be of interest.
(In reply to comment #15) > (In reply to comment #7) > > Form a JDT perspective I'm fine if we sign it, assuming you can make that > happen during the build. The conditioning is needed to get the pack200 > compression. > Probably easier for me to get it signed, than for me to get it conditioned correctly :) I'm sure there are a number of ways to fix this, but what I'm suggesting is the easiest _might_ be to not allow the jar processor to -repack it, and yes, live without the jar.pack.gz version of the file. I think as it is now it is being "packed twice" ... or ... maybe just once, but then the <p2.process.artifacts not called at the right time to "fix up" checksums after you extract it? But, from what I am seeing, this is being extracted BEFORE being sent for signing/processing via Eclipse Foundation ... and then what I suspect is happening (though would have to study your custom code some more) is that you then "re-replace it" after the signing/processing step. I can continue to look at "build changes" to fix things up ... but ... you might want/need to add some "echo" statements to your custom builder steps so its easier to understand what values its getting or what values are being assumed ... and maybe even putting in some "failure" conditions so if you don't find things where expected for your export-ecj.xml doesn't operate as expected it fails, instead of silently continue with the wrong jar. (You might already have that, I haven't looked at export-ecj.xml. [Deluxe solution would not be changing the version of the batch compiler each build ... not sure why that's done ... wouldn't seem that hard to use parents qualifier, with a strategic <replace statement ... little that I know.] So, maybe excluding from pack/-repack would not change anything and there would still be problems ... but, it'll take me a while to figure out what was being done and how I might have broken that, and what you need to have done. (So, in other words, might be worth trying to live without the jar.pack.gz version if that really did fix things ... I'd rather put the effort into not have the version qualifier change each build :)
I have just recalled something I changed in the build, that might account for the "new" MD5 checksum issue, that might be easy to fix. Previously, Kim was calling p2.process.artifacts at some point after the jars were processed and signed, in order to get the jar.pack.gz files. I removed that, since I could not get it to use Java 5 during that step and our custom build tasks now (oddly) require Java 6 to run (so Java 6 must be primary JRE for the build). I "took out" the processing step, thinking we could produce the jar.pack.gz file after the build (which, I have yet to do) but, point is, it might be that that p2.process.artifacts step, as a side effect, so to speak, was "fixing" the MD5 checksum discrepancy. This means I might be able to "put back" the p2.process.artifacts step, and simply remove the "pack" option (and, still do the pack later, when I can control the JRE to be Java 5). I'll investigate that angle, and ask for a respin, if looks like it would work.
Yes, easy to fix. the "pack" step was right before the "make final repo" step (which is where the md5 problem shows up) so I can just put that step back in, but remove the "pack" attribute ... and it will still fix up the MD5 checksums. Let's try that as a quick/easy fix.
(In reply to comment #18) > Yes, easy to fix. > > the "pack" step was right before the "make final repo" step (which is where the > md5 problem shows up) so I can just put that step back in, but remove the > "pack" attribute ... and it will still fix up the MD5 checksums. > > Let's try that as a quick/easy fix. Oh, and meant to say the original issues with Java 5/6, sign/pack, are discussed in bug 376032 comment 4 and others.
sweet! Glad I still have a little memory left. I see the batch compiler in today's noon respin: .../eclipse/updates/4.2-I-builds/I20120425-1200 (in plugins directory, and in metadata) Still says "3.7", but maybe that fix just wasn't released yet? plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.I20120425-1200.jar plugins/org.eclipse.jdt.core.compiler.batch.source_3.7.0.I20120425-1200.jar Now if we can just fix that qualifier :)
(In reply to comment #20) > plugins/org.eclipse.jdt.core.compiler.batch_3.7.0.I20120425-1200.jar > plugins/org.eclipse.jdt.core.compiler.batch.source_3.7.0.I20120425-1200.jar > > Now if we can just fix that qualifier :) I think if you do an N-build or the next I-build where JDT Core did a build input, we should get the correct qualifiers (see bug 377598).
