Community
Participate
Working Groups
I hope this is just a matter of adding sign="-Dsign=true" to masterBuild.sh script ... but ... needs to be done, checked ... and, I wouldn't be surprised if something doesn't need to be "fixed".
Got close! At least the scripts tried to, after adding sign="-Dsign=true" to masterBuild.sh From log ... compareAttribs: 133891 [exec] Previous HEAD position was 68330c0... fix feature version 133892 [exec] HEAD is now at 77343b9... Bug 355430 - change primary builder to 4.2 133893 [exec] Result: 2 133894 [echo] original: ${originalAttribs} 133895 [echo] polled: ls: cannot access /home/data/httpd/download-staging.priv/eclipse/I20120404-0051-out/eclipse-master-I20120404-0051.zip: No such file or directory From looking at the actual signing directory, the zip file was put at the "top" of the signing directory, instead of under a directory named 'I20120404-0051-out' /opt/public/download-staging.priv/eclipse/eclipse-master-I20120404-0051.zip So, should be easy to investigate and see what "variable" is missing.
(In reply to comment #1) > 133894 [echo] original: ${originalAttribs} > 133895 [echo] polled: ls: cannot access > /home/data/httpd/download-staging.priv/eclipse/I20120404-0051-out/eclipse-master-I20120404-0051.zip: > No such file or directory > > From looking at the actual signing directory, the zip file was put at the "top" > of the signing directory, instead of under a directory named > 'I20120404-0051-out' > Now that I've looked closer, this is actually working as designed ... odd design -- gee its different than the one in WTP -- which I originally copied from Kim's scripts :) ... but appears to be supposed to work this way, so in future runs I'll give it more of a chance. Add some more "echos" to see is going on, what's working, what's not.
I added enough debugging to see that executing the "sign" command fails, due to permission issues ... I've documented and ask for changes in the e4Build id; bug 374248 comment 27.
Just to document where we are at on this issue, signing "works", but, the "pre-conditioning" (and subsequent packing) does not. In short, it appears Java 6 is being used to condition the jars and for Eclipse Project, we need to make sure it is Java 5, since there are well known issues around this related to jars with no Java in them ... when "conditioned" with Java 6, they end up so that pack200 from Java 5 can not unpack200 them. This is effecting all the feature.jars and *.source jars being built by the platform. A second issue _might_ be that we are re-conditioning and re-signing jars from ECF. This seems a little unusual to me so I'll have to dig deeper to figure out if we are supposed to do that, or if we are supposed to "skip" those (Since, off hand, I would think they would already be conditioned and signed before we consume them.). This will take some time (at least rest of today) so I have disabled signing (and packing) for now, so complete "test builds" can at least be produced with auto tagging.
Findings so far (which are not related to fundamental problem), just making notes as I go ... All the Orbit bundles are correctly being excluded form packing and conditioning. Of all the bundles in "master feature zip to be signed", there are three odd cases of "third party" unsigned bundles: org.junit4_4.8.1.v20111115-0800.jar org.aspectj.runtime_1.6.7.20091231194938.jar org.aspectj.weaver_1.6.7.20091231194938.jar A. we still ship that old junit4 bundle? :) B. I'm surprised to see "org.aspectj.*" in there ... These are, btw, think that end up in the "master feature zip" to be signed ... maybe they are not actually used any repo or distribution? Second, the ecf bundles _are_ already signed, so ideally we would not be resigning them, but I'll need to look closer to see if they've been conditioned (pack200 -repack) with java 5 or 6. Currently they are in the p2.map file ... would be easier if in their own ecf.map file, I think ... since then could fit right in with the same Orbit exclusion logic. Not sure if that'd have other impacts? If I didn't say it yet, all the Eclipse project's bundles are being conditioned with Java 6, despite my best efforts to get it to use Java 5. So ... I'm probably doing something wrong, typo or something, or will have to do something fancy, like launch a separate process to force it to use Java 5? I still need to create some small "test cases" so I can sort it out after 6 or so runs, and not have to wait 2 hours for each test. That's my next step.
The ONLY way I could get this to work on my little sample was to "run the the build" with 1.5. There's two critical parts, one where <exec is used to call eclipse's "sign" program. The second where p2 ant task p2.process.artifacts which should be ran with same VM. I tried a number of ways to pass along -Dorg.eclipse.update.jarprocessor.pack200 and the java15home but nothing seemed to help. So, for now will just use 1.5 for the whole build. I know for WTP and Orbit, if I recall right, we run with many things, but when we want to run with a specific VM, we actually launch another <java task, even if it just runs the ant runner, and in that VM we specify exactly what we want, say, 1.5 when signing and packing.
In case followers of this bug didn't notice, we can not, apparently, run the whole build in Java 1.5, see bug 375807 comment 50. I'm trying one more experiment, but if that doesn't work, will try executing our own custom sign script, /org.eclipse.releng.eclipsebuilder/scripts/customsign.sh which, I hope, will give us the "control" we need to get Java 1.5 used during the conditioning step of the signer.
Ok, I think I've gotten Java 1.5 to sign (and pre-condition) our jars and they can later (not during the build, since it is running under 1.6) can be packed (pack200) and pass my verification tests. But ... we were still "contaminating" the bundles from ECF. I'm not sure why now, and why not before. The only rational theory I have is that there could be slight differences in the versions of pack200 that were/are used. We currently are set up (on build.eclipse.org) to use a 64 bit version of Java 1.5, and its pack200 reports version 1.23. e4Build@build:~> /shared/common/jdk-1.5.0-22.x86_64/bin/pack200 -V com.sun.java.util.jar.pack.Driver version 1.23, 06/04/14 I know in Orbit and Webtools we run the builds (and pack, etc.) with a 32 bit 1.5 VM and it reports a pack200 version of 1.22. e4Build@build:~> /shared/orbit/apps/ibm-java2-i386-50/jre/bin/pack200 -V com.sun.java.util.jar.pack.Driver version 1.22, 04/05/05 So ... maybe ... so ... the hard part ... how to easily "exclude" the ecf bundles from packing and resigning. I looked at several approaches, all of them (probably) taking days of effort ... but then thought maybe I could just move the ECF map entries from p2.map file to the orbit.map file. I normally don't like "quick and dirty tricks" ... but then noticed you apparently moved the jetty bundles into the orbit.map, presumably for similar reasons. :) So, as long as your "map file system" works like others I know about (that is, seldom matters where the entries are) we might get in a "signed run" this evening. Optimistically assuming this works, I've retitled bug 376281, which was specific to ECF, to simply say the generateExcludeList code should be more flexible and general purpose (rather than assume everything is in one file ... and who knows, maybe it doesn't and I just need to learn more about that ant task :).
Ian, adding you to CC, since I've been monkeying about with your p2.map file :) Only in the R4_HEAD branch, so far, but if it works we will propagate.
A few remaining problems ... the ant task doesn't like "fragments" and thinks everything should be a "plugin", I guess. I wonder if it matters? Details in 376281.
(In reply to comment #10) > I wonder if it matters? Trial and error while I sleep isn't bad :) I changed the 6 "fragment" to "plugin" and we'll see how it goes. I have a vague memory of Andrew saying once they wanted to support "bundle" and "fragment" all the way through, to be more semantically correct, but it was never finished and didn't really matter [in these map files ... still does in the manifests, etc., of course]. (Though, that was years ago, I could be wrong or mis-remembering). I figure it'll work like a charm, or crash and set the build machine on fire :) If anyone knows better, let us know. (If it doesn't work, I do have one more place I can "hard code" the values for now, I think, thought would break or be brittle every time ECF changed).
Well, that didn't work. I recall better now it didn't matter if you put "plugin" or "bundle", I think (for the main purpose of map files). But the build failed early not being able to find the fragment it needed. So, I've reverted all the changes to map files I made, put the Orbit map and the p2.map back to their original states. Next experiment, I've tried "hard coding" the to-be-excluded ECF plugins directly in the 'template' pack.properties file that is included in eclipsebuilder. Originally, that template pack.properties had pack200.default.args=-E4 pack.excludes=content.jar,artifacts.jar,@excludejars@ sign.excludes=content.jar,artifacts.jar,@excludejars@ So, I've changed it just to insert the whole list of exact ecf plugins as below (all as one line, doubt the line endings will show up correctly here in this bugzilla): Note what were "fragments" in map files still end up under "plugins" in master zip, to be excluded. pack200.default.args=-E4 pack.excludes=content.jar,artifacts.jar,plugins/org.eclipse.ecf_3.1.300.v20120319-0616.jar,plugins/org.eclipse.ecf.source_3.1.300.v20120319-0616.jar,plugin/org.eclipse.ecf.ssl_1.0.100.v20120319-0616.jar,plugin/org.eclipse.ecf.ssl.source_1.0.100.v20120319-0616.jar,plugins/org.eclipse.ecf.identity_3.1.200.v20120319-0616.jar,plugins/org.eclipse.ecf.identity.source_3.1.200.v20120319-0616.jar,plugins/org.eclipse.ecf.filetransfer_5.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.filetransfer.source_5.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer_3.2.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.source_3.2.0.v20120319-0616.jar,plugin/org.eclipse.ecf.provider.filetransfer.ssl_1.0.0.v20120319-0616.jar,plugin/org.eclipse.ecf.provider.filetransfer.ssl.source_1.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient_4.0.200.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient.source_4.0.200.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient.ssl_1.0.0.v20120319-0616.jar,/plugins/org.eclipse.ecf.provider.filetransfer.httpclient.ssl.source_1.0.0.v20120319-0616.jar,@excludejars@ sign.excludes=content.jar,artifacts.jar,plugins/org.eclipse.ecf_3.1.300.v20120319-0616.jar,plugins/org.eclipse.ecf.source_3.1.300.v20120319-0616.jar,plugins/org.eclipse.ecf.ssl_1.0.100.v20120319-0616.jar,plugins/org.eclipse.ecf.ssl.source_1.0.100.v20120319-0616.jar,plugins/org.eclipse.ecf.identity_3.1.200.v20120319-0616.jar,plugins/org.eclipse.ecf.identity.source_3.1.200.v20120319-0616.jar,plugins/org.eclipse.ecf.filetransfer_5.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.filetransfer.source_5.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer_3.2.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.source_3.2.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.ssl_1.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.ssl.source_1.0.0.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient_4.0.200.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient.source_4.0.200.v20120319-0616.jar,plugins/org.eclipse.ecf.provider.filetransfer.httpclient.ssl_1.0.0.v20120319-0616.jar,/plugins/org.eclipse.ecf.provider.filetransfer.httpclient.ssl.source_1.0.0.v20120319-0616.jar,@excludejars@ There must be an easier way!
With this "hard coding hack" signing now works "during the build". Pack200 works if done "after the build" (so we can do, easily, with java15.) IMHO, pack200 (p2.process.artifacts) should _always_ be done after the build, anyway, since only needed for "repos we distribute" ... as far as I know, there's no built in tests that depend on it, etc.). I've increased priority of bug 376281 since it will be a "pain" (and error prone) to mess with hard coding the ECF bundles each and every time they change in map files (two places to change, now) ... but ... we can make progress through the pain. This one being fixed ia a huge step in resolving bug 355430.
This was closed on the 9th of April. However, I don't see any packed jars in the p2 repository produced by recent builds. I filed bug 377974.
