Community
Participate
Working Groups
Created attachment 170998 [details] listof unsigned bundles in Helios RC3 (currently in staging) Yes, even JDT! "batch compiler". I've not heard of a reason it can not be signed? Oh, and EclipseLink has two ... I know one of those is a known, required exception (commonj.sdo) ... but, javax.persistence? Where's that come from? Not Orbit? Overall, though, I was glad to see it so few. Seems improved over previous years (or, I'm just looking later :) I'm sure these teams must plan on signing their final RC4 contributions ... right? I've attached the list I see (I just use a little shell script to call jarsigner -verify ... takes forever :(
org.eclipse.jdt.core.compiler.batch.sources_3.3.0.jar org.eclipse.jdt.core.compiler.batch_3.3.0.jar As far as I know the Eclipse project doesn't deliver bundles by this name - they aren't in our integration build repository. I'm digging around, but is it possible some other project is contributing these?
We build a two ecj* bundles that include the org.eclipse.jdt.core.compiler.batch bundles. They aren't signed because they are built separately from our other build artifacts as standalone downloads and aren't included in our repository or any features. I wasn't aware that another team was consuming them and contributing them to Helios. If this is an issue, the contributing team could sign them.
Sorry, in comment #2, I was wrong. These are not the same as the org.eclipse.jdt.core.compiler.batch* bundles that are contributed to Helios by an unknown party. In any case, who ever is contributing them shouldn't be using our namespace :-)
I see this one: org.eclipse.emf.teneo.jpox.libraries_1.0.0.v200806111928.jar.pack.gz: org.eclipse.emf.teneo.jpox.libraries_1.0.0.v200806111928.jar: But these are not part of the teneo build since 1.5 years or so and not part of any of the features of the helios.build file. Is there a way to find out where they are coming from? gr. Martin
> > Is there a way to find out where they are coming from? > Sort of. The log can get you closer. You can search the aggregators log for the run, such as https://build.eclipse.org/hudson/view/Repository%20Aggregation/job/helios.runAggregator/151/consoleFull Search for org.eclipse.emf.teneo.jpox.libraries Where it will say something like [exec] - mirroring artifact osgi.bundle,org.eclipse.emf.teneo.jpox.libraries,1.0.0.v200806111928 and then look _back_ in log until you see which repo it was mirroring. In this case, it says [exec] Mirroring meta-data from from http://download.eclipse.org/modeling/emf/updates/milestones [exec] - mirroring artifacts reference http://www.eclipse.org/modeling/updates [exec] - mirroring meta-data reference http://www.eclipse.org/modeling/updates [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/emf/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/mdt/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/emf/updates/ [exec] - mirroring artifacts reference http://download.eclipse.org/modeling/emft/updates/ [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/mdt/updates/ [exec] - mirroring meta-data reference http://download.eclipse.org/modeling/emft/updates/ [exec] Mirroring artifacts from from http://download.eclipse.org/modeling/emf/updates/milestones So ... not sure how to narrow that down, but you could naturally look for .build files that contributed these repositories -- if they are not yours. HTH
(In reply to comment #3) > Sorry, in comment #2, I was wrong. These are not the same as the > org.eclipse.jdt.core.compiler.batch* bundles that are contributed to Helios by > an unknown party. In any case, who ever is contributing them shouldn't be > using our namespace :-) Using the lookback in log method .... looks like it is jetty. [exec] Mirroring meta-data from from http://download.eclipse.org/jetty/7.1.3.v20100526/repository [exec] Mirroring artifacts from from http://download.eclipse.org/jetty/7.1.3.v20100526/repository
Reacting on comment 5 from David, hmm, I don't understand, there is a helios.build file which describes the Teneo contribution to Helios. How can I prevent that old teneo stuff (this plugin is 2 years old) is downloaded from other update sites? Is someone else including a dependency on this old plugin? gr. Martin
> > How can I prevent that old teneo stuff (this plugin is 2 years old) is > downloaded from other update sites? > Is someone else including a dependency on this old plugin? > I've opened bug 315709 to see if the aggregator can do anything (now, easily) for cases like this. I suspect some bundle requires one of the packages that the bundle exports (which includes very many very, commonly imported ones, such as org.apache.commons.collections, log4j, etc.) and p2 encounters a need for those packages, and finds this bundle an ideal candidate to satisfy many of what it needs! :/ so p2 pulls it in. Just guessing. We'll see what the b3 team thinks. Thanks,
Ok thanks, I contacted Nick to see if I can remove this old plugin somehow without corrupting the update site. gr. Martin
CC'ing Hugues from the jetty team to see if he can shed some light on the org.eclipse.jdt.core.compiler.batch bundles included with the jetty Helios contribution.
(In reply to comment #10) > CC'ing Hugues from the jetty team to see if he can shed some light on the > org.eclipse.jdt.core.compiler.batch bundles included with the jetty Helios > contribution. Hi Thomas and everyone. Jetty uses the JDT batch compiler for the JSP support. It is actually not signed: http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.6RC3-201005271700/ecj-3.6RC3.jar I suspect (or I have been told) that it is meant not to be signed so that projects that use this jar outside of equinox do not suffer from the signing overhead. Let me know what I should do.
(In reply to comment #11) > (In reply to comment #10) > > CC'ing Hugues from the jetty team to see if he can shed some light on the > > org.eclipse.jdt.core.compiler.batch bundles included with the jetty Helios > > contribution. > > Hi Thomas and everyone. > > Jetty uses the JDT batch compiler for the JSP support. > It is actually not signed: > http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.6RC3-201005271700/ecj-3.6RC3.jar Could you directly get the classes out of the jdt.core bundle ?
(In reply to comment #12) > > Hi Thomas and everyone. > > > > Jetty uses the JDT batch compiler for the JSP support. > > It is actually not signed: > > http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/S-3.6RC3-201005271700/ecj-3.6RC3.jar > Could you directly get the classes out of the jdt.core bundle ? Well we are using those classes in OSGi and the dependencies required by jdt.core are not available starting with org.eclipse.runtime. So we can't use jdt.core as a replacement of the batch compiler. Did you have something else in mind?
(In reply to comment #13) > Well we are using those classes in OSGi and the dependencies required by > jdt.core are not available starting with org.eclipse.runtime. > So we can't use jdt.core as a replacement of the batch compiler. > Did you have something else in mind? Maybe this is not possible, but using import packages you might be able to retrieve the types from the batch compiler without duplicating the types inside Jetty.
> I suspect (or I have been told) that it is meant not to be signed so that > projects that use this jar outside of equinox do not suffer from the signing > overhead. > > Let me know what I should do. If they must be redistributed in our common repo, they must be signed, "... unless there is a technical reason not to, approved by the Planning Council". So ... is it known for sure there would be a performance problem? Maybe its obvious to you and others, but I ask because in the past I know I've heard many people/developers say they didn't want to sign for fear of performance problems ... but, in reality there was none. So want to be sure we are not operating on myths.
(In reply to comment #14) > (In reply to comment #13) > > Well we are using those classes in OSGi and the dependencies required by > > jdt.core are not available starting with org.eclipse.runtime. > > So we can't use jdt.core as a replacement of the batch compiler. > > Did you have something else in mind? > Maybe this is not possible, but using import packages you might be able to > retrieve the types from the batch compiler without duplicating the types inside > Jetty. In fact we are not duplicating them. We depend on them and we only use Import-Package for those. (In reply to comment #15) > If they must be redistributed in our common repo, they must be signed, > "... unless there is a technical reason not to, approved by the Planning > Council". Ideally the jdt compiler batch is provided by the platform build. Right now we are picking it up ourselves. > > So ... is it known for sure there would be a performance problem? Maybe its > obvious to you and others, but I ask because in the past I know I've heard many > people/developers say they didn't want to sign for fear of performance problems > ... but, in reality there was none. So want to be sure we are not operating on > myths. We are using them with Equinox so performance really is not a problem for us. Thinking aloud: Right now the compiler batch contains an eclipse.inf file that expcitly prevents the jarprocessor from signing it: jarprocessor.exclude.sign=true jarprocessor.exclude.children=true I can insert in the jetty build a step that removes the eclipse.inf's directive from that jar and it will get signed. What do you think?
> Thinking aloud: > Right now the compiler batch contains an eclipse.inf file that expcitly > prevents the jarprocessor from signing it: > jarprocessor.exclude.sign=true > jarprocessor.exclude.children=true > > I can insert in the jetty build a step that removes the eclipse.inf's directive > from that jar and it will get signed. > What do you think? Sounds good, and appropriate, to me, given the short runway. It'd be the safest change. Long term, the JDT team should considering removing those directives, and even providing a signed version in their repo. Then an unsigned version could be saved away in a special location, for special cases, for those that need it (if there are known cases where an unsigned version is required). Naturally, I'm saying all this with the least understanding what JDT needs and why they provide it the way they do ... so, could be good reasons they are doing the way they are?
(In reply to comment #17) > > Sounds good, and appropriate, to me, given the short runway. It'd be the safest > change. Long term, the JDT team should considering removing those directives, > and even providing a signed version in their repo. Then an unsigned version > could be saved away in a special location, for special cases, for those that > need it (if there are known cases where an unsigned version is required). > > Naturally, I'm saying all this with the least understanding what JDT needs and > why they provide it the way they do ... so, could be good reasons they are > doing the way they are? OK, I'm working that. In fact ... I'll be redoing the JDT package directly from the jdt.core as this is the only way to stay up to date. In have filed a request for improvement regarding providing the jdt batch compiler as part of the eclipse platform repository: bug 315844
(In reply to comment #18) > (In reply to comment #17) > > OK fixed for RC4 for the jdt batch compiler. I am currently using the rc3 version of ecj.jar and will redo a build for every new build of the eclipse-platform.
Created attachment 171688 [details] list of unsigned bundles in Helios RC4 Seems to have gotten worse? Jetty!? DLTK!? But appears a few projects were fixed? This list was based on initial RC4 repo (not RC4a), I would like the projects listed in this "unsigned report" to say explicitly if they plan to get their bundles signed and will request a final respin, or if they would prefer to be removed from the common repo aggregation. (According to the Sim. Release requirements, even if you do not sign your bundles, you can still technically "release" with Helios, and just not be in the common repo ... you'd provide your updates/distributions on your own project site). So, please be explicit. Thanks,
Hi David, Some old teneo jar files (from 2008) are listed in the attachment. These jars are not part of the Helios release at all. I have asked Nick what I can do about them, but have not heard back. But as these jar files are not part of Helios I assume that this does not affect Teneo's Helios contribution, or does it? gr. Martin
(In reply to comment #21) > Hi David, > Some old teneo jar files (from 2008) are listed in the attachment. These jars > are not part of the Helios release at all. I have asked Nick what I can do > about them, but have not heard back. > But as these jar files are not part of Helios I assume that this does not > affect Teneo's Helios contribution, or does it? > > gr. Martin Well ... should effect it for two reasons! ... they are unsigned, and they should not be in there at all if not part of helios release. But, agree, you have a special case. I'd appreciate it if you could continue to try and get them removed from the source repo unless there's some reason not to. Since they get "pulled in" to common repo by some odd quirks, they could also get installed into someone's ide by those same quirks. So, pretty bad to have them in there. Is there a bug open yet? And, thanks very much for the kind reminder of their history.
(In reply to comment #20) > Created an attachment (id=171688) [details] > list of unsigned bundles in Helios RC4 > > Seems to have gotten worse? Jetty!? DLTK!? Jetty: I don't understand what happened and I am investigating. The next build is signed.
(In reply to comment #23) > Jetty: I don't understand what happened and I am investigating. > The next build is signed. OK I am missing something: on my machine the jarsigner reports a signed jar. On the build machine it is unsigned. I am investigating on the old build while the new one spinning. When I run jarsigner -verify on my machine everything is fine: malphettes@hmalphettes-laptop:~/proj/osgi-experiments/jetty-cvs/built-not-packed/7.1.4/org.eclipse.jetty.product-7.1.4-SNAPSHOT/plugins$ ls -la org.eclipse.jetty.client_7.1.4.v20100609.jar -rw-r--r-- 1 hmalphettes hmalphettes 78133 2010-06-09 15:22 org.eclipse.jetty.client_7.1.4.v20100609.jar hmalphettes@hmalphettes-laptop:~/proj/osgi-experiments/jetty-cvs/built-not-packed/7.1.4/org.eclipse.jetty.product-7.1.4-SNAPSHOT/plugins$ jarsigner -verbose -verify org.eclipse.jetty.client_7.1.4.v20100609.jar 7064 Wed Jun 09 18:11:02 PDT 2010 META-INF/MANIFEST.MF 4786 Wed Jun 09 18:11:04 PDT 2010 META-INF/ECLIPSEF.SF 5638 Wed Jun 09 18:11:04 PDT 2010 META-INF/ECLIPSEF.RSA 0 Wed Jun 09 12:13:40 PDT 2010 META-INF/ 0 Wed Jun 09 12:13:40 PDT 2010 META-INF/maven/ 0 Wed Jun 09 12:13:40 PDT 2010 META-INF/maven/org.eclipse.jetty/ 0 Wed Jun 09 12:13:40 PDT 2010 META-INF/maven/org.eclipse.jetty/jetty-client/ sm 2961 Wed Jun 09 12:07:20 PDT 2010 META-INF/maven/org.eclipse.jetty/jetty-client/pom.xml sm 124 Wed Jun 09 12:13:38 PDT 2010 META-INF/maven/org.eclipse.jetty/jetty-client/pom.properties sm 76 Wed Jun 09 14:56:32 PDT 2010 META-INF/eclipse.inf 0 Wed Jun 09 12:12:22 PDT 2010 org/ ... sm 1539 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/CachedExchange.class sm 3745 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/RedirectListener.class sm 736 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/HttpEventListener.class ... sm 984 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/webdav/PropfindExchange.class sm 6017 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/webdav/WebdavListener.class sm 543 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/HttpExchange$ContentExchange.class sm 17493 Wed Jun 09 12:12:22 PDT 2010 org/eclipse/jetty/client/HttpClient.class sm 1614 Wed Jun 09 12:12:22 PDT 2010 about.html s = signature was verified m = entry is listed in manifest k = at least one certificate was found in keystore i = at least one certificate was found in identity scope jar verified. When I run it on the build machine: hmalphett@build:/home/www/jetty/7.1.4.v20100609/repository/plugins> ls -la org.eclipse.jetty.client_7.1.4.v20100609.jar -rwxrwxr-x 1 hmalphett common 78133 2010-06-11 10:59 org.eclipse.jetty.client_7.1.4.v20100609.jar hmalphett@build:/home/www/jetty/7.1.4.v20100609/repository/plugins> jarsigner -verbose -verify org.eclipse.jetty.client_7.1.4.v20100609.jar 7064 Jun 9, 2010 6:11:02 PM META-INF/MANIFEST.MF 4786 Jun 9, 2010 6:11:04 PM META-INF/ECLIPSEF.SF 5638 Jun 9, 2010 6:11:04 PM META-INF/ECLIPSEF.RSA 0 Jun 9, 2010 12:13:40 PM META-INF/ 0 Jun 9, 2010 12:13:40 PM META-INF/maven/ 0 Jun 9, 2010 12:13:40 PM META-INF/maven/org.eclipse.jetty/ 0 Jun 9, 2010 12:13:40 PM META-INF/maven/org.eclipse.jetty/jetty-client/ m 2961 Jun 9, 2010 12:07:20 PM META-INF/maven/org.eclipse.jetty/jetty-client/pom.xml m 124 Jun 9, 2010 12:13:38 PM META-INF/maven/org.eclipse.jetty/jetty-client/pom.properties m 76 Jun 9, 2010 2:56:32 PM META-INF/eclipse.inf ... m 1539 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/CachedExchange.class m 3745 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/RedirectListener.class m 736 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/HttpEventListener.class ... m 6017 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/webdav/WebdavListener.class m 543 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/HttpExchange$ContentExchange.class m 17493 Jun 9, 2010 12:12:22 PM org/eclipse/jetty/client/HttpClient.class m 1614 Jun 9, 2010 12:12:22 PM about.html s = signature was verified m = entry is listed in manifest k = at least one certificate was found in keystore i = at least one certificate was found in identity scope jar is unsigned. (signatures missing or not parsable) I'll continue to digg around but if someone has clue let me know. Thanks!
I will correct DLTK contribution.
Answering on the EclipseLink javax.persistence jar. - EclipseLink requires shipping of an unsigned version of this jar for support JPA 2.0 on nonOSGI containers. - As we are the only project using the JPA 2.0 version of javax.persistence this jar is not yet in Orbit.
I'm not sure anymore that h2 comes from my Sourceforge repository. There the bundle is named org.h2.jdbc not org.h2 I removed it nevertheless and I'm running a new build...
(In reply to comment #24) > (In reply to comment #23) > > > Jetty: I don't understand what happened and I am investigating. > > The next build is signed. > > OK I am missing something: on my machine the jarsigner reports a signed jar. On > the build machine it is unsigned. I think my testing using the jarsigner on build.eclipse.org is not conclusive: hmalphett@build:/home/www/eclipse/updates/3.6milestones/S-3.6RC4-I201006031500/plugins> jarsigner -verify org.eclipse.core.runtime_3.6.0.v20100505.jar jar is unsigned. (signatures missing or not parsable) David, can you let me know how you verify the signature please?
(In reply to comment #28) OK I found out what I am doing wrong. The packed files are not signed. I am fixing this. Sorry for the noise.
Created attachment 171776 [details] list of unsigned bundles in Helios Post RC4 Looking pretty clean. Still waiting for DLTK. There are those few odd cases that should not even be in the repo ... please do continue to investigate the old teneo bundles, and the redundant, 3 part version, unsigned org.h2 bundle. It would be good to understand them, even know what a future fix might look like ... even if we can not literally fix in the next few days. Thanks all for your attention to this. Just for your reading pleasure ... I collected some numbers for this near file aggregated repo (which does not include the Eclipse Platform or Equinox repos. 640 feature jars 2777 bundle jars 2265 packed.gz files 993 Megs
Looking at the log from the aggregation build, the odd 'org.h2' item comes from DLTK (found by searching for "org.h2" mirroring line, and then looking back, in log). So, once DLTK is signed, it at least would not show up as an unsigned bundle ... but, it is still incorrect to have a 3 part version in the repo (against the Sim. Rel. and Eclipse versioning rules) and it would not be used from the common repo. The 4 part version would be used if someone was trying to install from the common repo [as far as I know]. The "good" version of org.h2 comes from CDO. I wonder if putting CDO contribution before the DLTK contribution would be a "fix" (of sorts) and cause/allow DLTK/P2 mirror to not mirror the three part version because P2 would find a satisfactory (4 part) version already exists in the repo from CDO. I wonder if an ordering trick would help avoid the old teneo bundle too ... if P2 is giving it preference because it (also) has those exports for log4j, xml, etc. then perhaps it teneo comes last in the whole list, those other requirements (log4j, xml, etc.) would have already been "solved" by SAT so it would not be so motivated to lock in that particular contributed source? But, I will try a standard rebuild first to get newly signed DLTK bundles.
I'm still running the full verify script, but will report what I see so far: The 3-part version number org.h2 is no longer in the repo (much less unsigned) ... so, DLTK must have fixed that too ... THANKS! The org.eclipse.emf.teneo.jpox.libraries bundle is actually pulled in/mirrored BEFORE the teneo contribution (so moving that later, won't help!) ... from what I can see from the aggregator's log it appears to be being pulled in while the aggregator is "getting" org.eclipse.emf.compare. I looked, and I don't see it in emf.compare directly ... so, I think back to the theory that some bundle in emf.compare is indirectly causing it to be pulled in because org.eclipse.emf.teneo.jpox.libraries satisfies a number of 'requirements' one of those bundles have. An interesting test would be to try and install emf.compare feature by itself and see if that org.eclipse.emf.teneo.jpox.libraries gets installed?
I'm resolving as fixed. The latest verify test on staging shows only jars that are expected to not be signed: two expected from eclipselink commonj.sdo_2.1.1.v200905221342.jar: javax.persistence_2.0.1.v201006031150.jar: and this problematic case from emf org.eclipse.emf.teneo.jpox.libraries_1.0.1.v200902271808.jar.pack.gz: org.eclipse.emf.teneo.jpox.libraries_1.0.1.v200902271808.jar: The concern for this teneo one isn't so much it is unsigned (in fact, it's eclipse.inf file says not to sign it, so I'm sure that's intentional). The concern instead is that its old and not expected to be pulled in at all. The situation is discussed some in bug 315709 but this may deserve a bug of its own in emf, and emf may want to prune some of its old code to remove this bundle. (I am not sure how to do that, and still leave the repository "valid", but if there is not currently a way, someone needs to develop one :) The following is all theory (could be other issues or explanations): The problem with the bundle ... the reason it could be said to be a "bad, unruly bundle" is that it exports a number of commonly imported packages, such many from org.apache* so as far as p2 is concerned, this is a perfectly valid bundle to pick to provide those org.apache* packages when others say they require them (in fact, it could be said it is a good example of p2 optimizing what it installs ... picking minimum bundles to maximize number of requirements satisfied). But ... problem is, that was not really the intent/design of the bundle so 1) some extra things are dragged in (the old jpox code) and 2) who knows if the org.apache code is really the desired/correct versions of that code (that code should be coming from Orbit, at a minimum). And, the unruly teneo bundle just showed up here because it was unsigned. I wonder ... there could be other "old" bundles being pulled in inadvertently? I'd assume not, or surely people would have noticed in their extensive testing :) If inadvertent deliveries is found to be a large problem, we may need to change the way people contribute to this "common build", so instead of projects using URLs which point to some large repos that contain many old releases, they may need to prepare repos with exactly what they intend to contribute for any particular build or release ... right? Of course, in a perfect world (where each bundle was perfect) it would not matter ... after all, the teneo bundle does say it exports org.apache* and if someone imports org.apache* they are saying they don't care where it comes from. But ... history has shown not all our bundles are perfect and we need some way to narrow what is considered for our common repo. Pruning old repos is not really a very good solution in general, since even if incorrect for current release, presumably in some old release or distribution it was required, even if not perfect. But ... this signing bug is solved (there now, was that so hard :)
