[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] Notice of change to batch "signing service"

> ... we are using Tycho for our builds, but our jobs provide the following option when calling Maven
> (which seems to be used by a couple of other hudson jobs as well;
> it was probably recommended in some instructions I donât remember):

> -Dorg.eclipse.update.jarprocessor.pack200="/shared/common/sun-jdk1.6.0_21_x64/jre/bin"

> I assume we will have to adopt this to something like the following then, right:

> -Dorg.eclipse.update.jarprocessor.pack200="/shared/common/jdk1.8.0_x86-latest/jre/binâ

This was probably recommended because you were using Java 7 or 8 to "run your build" but it was determined to "minimize issues" to use the lower version of pack200 (Especially when many users or pre-reqs were using Java 6).  So, a) yes, eventually you'll "minimize issues" by using jdk 7 or 8 version, but b) that should be "automatic" if you are using Java 7 or 8 to run your builds, so I'd try leaving it out .. perhaps commented out, for when you need it again when you move to Java 9 :)

As far as checking yourself, Ed has already described one basic procedure. The b3 aggregator will also check, but only if you select the "build" option, which can take a long time, if you are trying to build the whole Sim. Release. It is possible, though, so set up your own b3 aggregator job (just for testing) that uses only your contribution, plus the minimum number of "prereq" contributions. It is not exactly easy to set this up -- the first time -- but, once set up, is relatively easy to keep up to date.

One more: There are some bash scripts in the sim release test project, that are not very efficient, but are an easy way to check a whole directory of jars and pack.gz jars. Those two scripts are verify.sh and verifydir.sh (both need to be in your on your path, and you execute verifydir.sh (and it calls verfiy.sh). You may need to make your own copy and adjust for your system, and your needs.

Another almost-off-topic tidbit: Many are surprised they "have errors" because they have tested "installing" or "updating" using p2, and it works fine. The reason is that p2 tries it's best to "recover from errors" so often if there are problems with *.jar.pack.gz, then it will simply try the *.jar file directly, which would work. But, for our Sim. Release repo (and should for all repos) we want the repo to be "perfect" and not allow it to contain "badly packed" jars. Doing so ends up causing "extra downloads", and wasting users time when they do an update.

I realize it is not easy to get a "perfect repo", it takes extra effort, but that's part of the whole purpose of having or joining the Sim. Release, so everyone's extra effort is acknowledged, and appreciated.

Thanks for the questions!





From:        Alexander NyÃen <nyssen@xxxxxxxxx>
To:        Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>,
Date:        04/11/2015 03:59 AM
Subject:        Re: [cross-project-issues-dev] Notice of change to batch "signing        service"
Sent by:        cross-project-issues-dev-bounces@xxxxxxxxxxx




David,

I admit I am not very acquainted with the internals of signing and pack200. However, we are using Tycho for our builds, but our jobs provide the following option when calling Maven (which seems to be used by a couple of other hudson jobs as well; it was probably recommended in some instructions I donât remember):

-Dorg.eclipse.update.jarprocessor.pack200="/shared/common/sun-jdk1.6.0_21_x64/jre/bin" -Xmx768m

I assume we will have to adopt this to something like the following then, right:

-Dorg.eclipse.update.jarprocessor.pack200="/shared/common/jdk1.8.0_x86-latest/jre/binâ

Is there a way to confirm that a build behaves as expected (do the sim-rel repo-reports for instance cover this)?

Cheers
Alexander

Am 11.04.2015 um 08:48 schrieb David M Williams <david_williams@xxxxxxxxxx>:

I wanted to let everyone know that "we" are changing the version of Java's pack200 at the infrastructure service I call "batch signer".  (Bug 463510)
It was using Java 6 level of pack200, and we have moved to the Java 8 level of pack200.

This command is described at

https://wiki.eclipse.org/IT_Infrastructure_Doc#ZIP_and_JAR_files_from_the_Commandline_.28Queued.29

This does not directly impact Tycho users or Buckminster users (which do their own re-pack and pack, based on the VM the build is running, I assume) but might effect PDE users, which traditionally have used this service to "re-pack" (condition) and sign jars -- and, then at some short later time are "packed".  At that "some later time", it is the release engineers responsibility to use the same version to 'pack' as was used to 'repack'.


While no direct impact to Tycho or Buckminster builders, the principles and consequences of moving from one level (Java 6) to another (Java 8) apply with any builder, so the following points may be useful information to all.


If you have "old byte codes", or, even new ones, compiled at the Java 4, Java 5, or Java 6 level, you *might* find some of those bytes codes no longer survive the "re-pack", sign, and "pack" process (where as maybe the would survived, when using Java 6 to run your build (and hence your repack/pack). That is, if user tries to download the pack.gz file, and unpack into a normal jar, it may be reported to have in "invalid signature" or in some other way "broken".


In my experience, with Orbit jars, the "error rate" is about 1.7%. Others, for some unknown reason, see higher rates (such as 20-30%?) . Some of these cases *might* be bugs in the pack/unpack programs, but it's just as likely that there are "bugs" (fringe cases) where the byte codes for lower levels of Java are not quite "right".  All I know for sure: it has never been perfect.


But, luckily, easy to solve.
All three builders (PDE, Tycho, and Buckminster) allow you to specify an 'eclipse.inf' file in the META-INF directory, with a directive (line) that is:
 
jarprocessor.exclude.pack=true

Thus, that one problematic jar is not packed, but is still signed.


So important point 1: be sure you "pack200" with the same version you used to do "repack" (if the builder doesn't do it for you automatically).
Important point 2: I wouldn't recommend studying the byte codes and trying to find a pattern of code that is the cause (unless you just happen to love byte codes), ... just slap in an eclipse.inf file and move on to more important things.
Important point 3: You should not, ever never, "re-sign" and certainly not re-pack or pack200 a jar that has already been signed or processed by pack200. That pretty much guarantees the jar will be broken. In more ways than one. (
Bug 461669)  

The reason for making this move, now, is that it is best to "keep up" with what our users are using, and, with versions of Java that are expected to stay in service for a while ... otherwise, we might have to monkey around with this during a service release, which is less than ideal.


* Bonus, for reading this far in my wordy note: Why jump two levels, from 6 to 8, why not move to Java 7 first? Well it turns out, at the moment, the versions of pack200 and unpack200 that ship with Java 7 and Java 8 are
exactly the same. But, the advantage is, in some service release, there might be a new version and we'd pick it up automatically, as long as we use /shared/common/jdk1.8.0_x86-latest consistently.  Double bonus: remember that pack200 and unpack200 have little to do with Java source code they are stand-alone executables, written in C-code, that simply manipulate byte codes. Which is how p2 can let you specify any version of pack200 you want, no mater which version of Java you are using.

Naturally, feel free to comment in
Bug 463510 if any questions or concrete problems known.
 



_______________________________________________
cross-project-issues-dev mailing list

cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

--
Dr. Alexander NyÃen
Dipl.-Inform.
Principal Engineer

Telefon: +49 (0) 231 / 98 60-202
Telefax: +49 (0) 231 / 98 60-211
Mobil: +49 (0) 151 /  17396743

http://www.itemis.de
alexander.nyssen@xxxxxxxxx

itemis AG
Am Brambusch 15-24
44536 LÃnen

Rechtlicher Hinweis:

Amtsgericht Dortmund, HRB 20621

Vorstand: Jens Wagener (Vors.), Wolfgang Neuhaus, Dr. Georg Pietrek, Jens Trompeter, Sebastian Neus

Aufsichtsrat: Prof. Dr. Burkhard Igel (Vors.), Michael Neuhaus, Jennifer Fiorentino


[attachment "signature.asc" deleted by David M Williams/Raleigh/IBM] _______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev