Community
Participate
Working Groups
I am a committer of the Babel project, and we ship translations of Eclipse as fragments features. As such we face a serious problem as some projects tend to add more plugins to Babel than what they actually ship. In some cases fragments don't have a plugin host. Then people cannot update anymore. I see 2 complementary solutions to this problem: 1. Don't bother too much with fragments. The old update manager was just ignoring them during updates. This new P2 behavior is felt like a regression by users right now. 2. Install fragments carefully: instead of taking the whole feature, just take what you need.
Hi, the 2nd option isn't working. Eclipse FR bundle cover too many missing hosts. I'm using Eclipse 3.4 SR1 modeling version and I can't update my custom plugins. The update manager should ignore or warn about missing hosts. There's some examples: Cannot complete the request. See the details. Unsatisfied dependency: [org.eclipse.text.tests.nl_fr 3.4.0.v20081130043401] requiredCapability: osgi.bundle/org.eclipse.text.tests/0.0.0 Unsatisfied dependency: [org.eclipse.swt.tools.nl_fr 3.4.0.v20081130043401] requiredCapability: osgi.bundle/org.eclipse.swt.tools/0.0.0 osgi.bundle/org.eclipse.jface.tests.databinding.conformance/0.0.0 Unsatisfied dependency: [org.eclipse.jdt.compiler.apt.tests.nl_fr 3.4.0.v20081130043401] requiredCapability:
(In reply to comment #1) > Hi, the 2nd option isn't working. Eclipse FR bundle cover too many missing > hosts. I'm using Eclipse 3.4 SR1 modeling version and I can't update my custom > plugins. The update manager should ignore or warn about missing hosts. Of course the 2nd option is not working right now. At Intalio to work around the issue, we use a Ruby tool to take only the plugins that are bundled into our product. 1. We unzip a finished untranslated copy of the product 2. We take all the plugin names 3. We get the site.xml of the Babel update site 3. We download the feature for the languages we are interested in. 4. For each plugin present in the feature, we look if it is present in the product. 5. If yes, we download the plugin 6. We zip the plugins found as a new feature. Hopefully p2 could do that as well. But that would mean they would need to ignore the feature, not sure they are ready to ditch those. :) As a Babel committer, I can guarantee it is near to impossible to get all the features right. We will do the best we can but we will fail sooner or later.
> We will do the best we can but we will fail sooner or later. That is the funniest thing I've read in a while. It applies to so many things I do!
(In reply to comment #3) > > We will do the best we can but we will fail sooner or later. > > That is the funniest thing I've read in a while. It applies to so many things > I do! > Some context. Denis is a Babel committer too... if you had hope this was an isolated case.
(In reply to comment #4) > (In reply to comment #3) > > > We will do the best we can but we will fail sooner or later. > > > > That is the funniest thing I've read in a while. It applies to so many things > > I do! > > > Some context. Denis is a Babel committer too... if you had hope this was an > isolated case. > (In reply to comment #4) > (In reply to comment #3) > > > We will do the best we can but we will fail sooner or later. > > > > That is the funniest thing I've read in a while. It applies to so many things > > I do! > > > Some context. Denis is a Babel committer too... if you had hope this was an > isolated case. > Sorry, I completely miss the point "just take what you need" and the context... The core plateform fragment bundle tests and all the related stuff. The scripted method is a nice workaround. Thanks!
The problem comes from the fact that the "default" metadata generated by p2 behind the scene when it connects to the legacy update site of babel update does not represent the intent of what is made available. Let me detail here: When p2 generates metadata for the babel feature (or for any other feature), the relationship between the feature and the plug-in is "strict" (in that it can only be satisfied), which means that all the plug-ins have to be installed or none. And then the metadata generated for the fragment also result in a strict and greedy dependency on the host. This explains the behavior you are experiencing, since when you are trying to install the feature p2 will try to satisfy all the dependencies which will result in installing more than what you want (installing actual code instead of just the translation). The solution for the Babel project is to create p2 metadata to have the feature optionally requires the plug-ins it includes and for each fragment to have a non greedy dependency on its host. With this structure one will be able to install the "French" language and only get the translation for the plug-ins that are installed. Even better, with this it becomes possible to have only one feature for all the projects (so there would be no more modeling feature, eclipse feature, webtools feature). For the user this has the nice advantage that if he wants one language he does not have to worry about where it is coming from. Now as for generating this metadata, you have two ways: write your own metadata generator reusing the code from p2 (should be relatively straightforward) or write an XSLT transform that tweaks the metadata resulting from a typical p2 metadata generator invocation (http://help.eclipse.org/ganymede/index.jsp?topic=/org.eclipse.platform.doc.isv/guide/p2_metadata_generator.html)
Thanks Pascal for the info! Your suggestion may solve our problem. I followed the link and read the "Generating p2 metadata" doc. However, I do not see the option "to have the feature optionally requires the plug-ins it includes and for each fragment to have a non greedy dependency on its host". Maybe I just missed that. Could you provide more info or some examples? Thanks!
Pascal, as Kit said, and we agreed on that during the last Babel call, we'd like to pursue the work on this. However we would really need some more samples and documentation to get started. None of the Babel committers is committed full time on the project, and we'd have a hard time shooting in the dark. I hope to be able to dive into the link you provided myself later this week. Any additional help is very appreciated.
If you could make a zip available with some (or all) of the features then I can look into it.
Created attachment 119995 [details] .log Hi Pascal, here is a simplified testcase using Eclipse SDK and its language pack only. But, the problem of not finding the host test and example plug-ins applies to other Eclipse projects too. - download and unzip Eclipse SDK 3.4.1 http://download.eclipse.org/eclipse/downloads/drops/R-3.4.1-200809111700/download.php?dropFile=eclipse-SDK-3.4.1-win32.zip - download and unzip the Babel French Language Pack for Eclipse 3.4 http://www.eclipse.org/downloads/download.php?r=1&file=/technology/babel/babel_language_packs/BabelLanguagePack-eclipse-fr_3.4.0.v20081207043402.zip - launch Eclipse (okay to launch in English) - receive "Unable to satisfy dependency" messages in .log file
Created attachment 120124 [details] Java tool to update fragments using an existing application. There's a little tool in java to fetch NL fragments for an existing host (a la Intalio). BabelFragmentFetcher Usage: <eclipsePath> <updateSiteURL> <locale code> 1. Scan eclipse plugin directory. 2. Fetch features and extract plugins info 3. Match and download plugins in the user temp directory Enjoy!
Created attachment 120540 [details] xslt to make content.xml's dependencies non-greedy and optional (In reply to comment #8) > Pascal, as Kit said, and we agreed on that during the last Babel call, we'd > like to pursue the work on this. However we would really need some more samples > and documentation to get started. None of the Babel committers is committed > full time on the project, and we'd have a hard time shooting in the dark. > > I hope to be able to dive into the link you provided myself later this week. > Any additional help is very appreciated. I had a go at the XSLT transform suggested by Pascal (for JBoss Tools' langpacks - I'm having similar problems with P2 as the Babel langpacks). I generate the P2 metadata from the command line (without -compress option), and apply it with this Ant snippet: <move file="target/jars/content.xml" tofile="target/jars/content_orig.xml" /> <!-- Process content.xml with <xslt> --> <xslt style="content.xsl" in="target/jars/content_orig.xml" out="target/jars/content.xml" /> I couldn't find any documentation of the content.xml schema as such, but http://wiki.eclipse.org/Installable_Units describes the meaning of the greediness and optionality flags. I'm not completely sure if this is what Pascal meant, because I'm running into other dependency problems, but my errors now seem to come from the transitive dependencies of JBT, which is progress, of a sort!
(In reply to comment #12) > I couldn't find any documentation of the content.xml schema as such, but > http://wiki.eclipse.org/Installable_Units describes the meaning of the > greediness and optionality flags. Oh, this too: http://wiki.eclipse.org/Equinox_P2_Resolution#Requirements_greed
Created attachment 120986 [details] Tweaked p2 metadata repository for the french translation Find attached a p2 metadata repository containing the changes I was talking about. You will notice that the greedy flag is only put on the translation fragments. More precisely on the requirement that goes from the fragment to its host (e.g. <required namespace='osgi.bundle' name='org.eclipse.swt' range='0.0.0' greedy='false'/>). And the optional dependencies are only to be on the requirement that goes from the feature to the fragments (e.g. <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.ant.tests.core.nl_fr' range='[3.4.0.v20081214043401,3.4.0.v20081214043401]' optional='true'/>) I have given a try to the XSLT attached however it does not fix the metadata as required and also causes the loss of the feature. I'm sorry but I'm not familiar enough with XSLT to fix it. My other concern is that after running the transform with the XSL Tools from WTP, the resulting file had been altered a lot more than I was expecting. The single quotes had been replaced by double quotes and characters that were encoded got unencoded.
Created attachment 121036 [details] Revised xslt to make content.xml's dependencies non-greedy (fragments -> host) and optional (features -> fragments) I have changed my xslt so that nl features have optional dependencies on the fragment plugins, and the nl fragments have non-greedy dependencies on their hosts. (The "contains '.nl'" expressions are to exclude the units with id "config.a.jre" and "a.jre", which I hadn't noticed before.) I'm testing against JBoss Tools, not Babel, but it's definitely working better for me than the default p2 metadata conversion did. Pascal, if the p2 metadata converter produces bad metadata for Babel, is that a bug we should report? Or is it just the old metadata/dependencies for Babel were technically invalid, and the p2 metadata is finally expressive enough to do what was needed all along? As for the changes introduced by XSLT processing, I don't think changing single to double quotes should be a big problem. Not sure what the other encoding changes were for Babel's XML, but with JBT I noticed that encoded linefeeds were replaced by actual linefeeds. Not a huge problem in my case, but YMMV.
Created attachment 121038 [details] Re-Revised xslt to make content.xml's dependencies non-greedy (fragments -> host) and optional (features -> fragments) This version of the xslt should preserve processing instructions (ie <?metadataRepository>), and no longer makes dependencies on *.feature.jar optional, so it should be closer to Pascal's example content.xml. (I couldn't find the source content.xml used by Pascal, so I stripped the @optional and @greedy attributes from the example, ran my xslt [cmd: xmlstarlet tr content.xsl content.xml] and compared the result with the example. As far as I can tell, the only differences are the single/double quotes and the encoded/decoded linefeeds. Anyway, I shall try not to bother the CC list with further updates to my humble little xslt!
These are all nice works! Thanks Pascal, Sean, and Benoit! The current Babel server design has a cron job to kick off a PHP to generate the Babel language pack update site and downloadable language packs. We need an automated solution. How could we integrate the "p2 metadata generator invocation" step and the "XSLT transformation" step into the PHP? Are there a PHP interface to each of these steps?
(In reply to comment #17) > These are all nice works! Thanks Pascal, Sean, and Benoit! > > The current Babel server design has a cron job to kick off a PHP to generate > the Babel language pack update site and downloadable language packs. We need an > automated solution. How could we integrate the "p2 metadata generator > invocation" step and the "XSLT transformation" step into the PHP? Are there a > PHP interface to each of these steps? I would love to help, but I don't know anything about PHP. Perhaps you ought to consider integrating *some* Java modules into the Babel infrastructure, since most of the programmers interested in Babel are likely to be Java programmers, but perhaps not PHP. For instance, if you call out from PHP to an Ant script, I could give exactly you what I'm using for this. Anyway, for the initial p2 metadata generation, unless you want to embed a JVM into PHP, I think you'll have to call out to the command line, and run something like the command line example in <http://wiki.eclipse.org/Equinox_p2_Metadata_Generator>. (Additionally, I would recommend removing the options -compress and -append, and adding "--launcher.suppressErrors -nosplash -consoleLog".) For the xslt step, any modern environment (including Ant, Linux command-line, PHP) should offer the ability to run a transform given an xml file and a stylesheet. From a quick google, Example #1 at <http://us.php.net/manual/en/function.xslt-process.php> should cover what we need here. Here are my Ant targets, for reference: <target name="deletep2" description="Remove p2 metadata (revert to site.xml)"> <delete dir="${jardir}" includes="content.jar,content.xml,content_orig.xml,artifacts.jar,artifacts.xml"/> </target> <target name="p2" depends="deletep2" description="Generate metadata for Eclipse 3.4's update manager (P2)" > <!-- Generate P2 metadata so that update manager won't take forever. http://wiki.eclipse.org/Equinox_p2_Metadata_Generator --> <exec executable="eclipse"> <arg value="-application" /> <arg value="org.eclipse.equinox.p2.metadata.generator.EclipseGenerator" /> <arg value="-updateSite" /> <arg value="${jardir}" /> <arg value="-site" /> <arg value="file://${jardir}/site.xml" /> <arg value="-metadataRepository" /> <arg value="file://${jardir}" /> <arg value="-metadataRepositoryName" /> <arg value="Translation Update Site" /> <arg value="-artifactRepository" /> <arg value="file://${jardir}" /> <arg value="-artifactRepositoryName" /> <arg value="Translation Artifacts" /> <!-- <arg value="-compress" /> <arg value="-append" /> --> <arg value="-reusePack200Files" /> <arg value="-noDefaultIUs" /> <arg value="--launcher.suppressErrors" /> <arg value="-nosplash" /> <arg value="-consoleLog" /> <arg value="-vmargs" /> <arg value="-Xmx256m" /> </exec> <move file="${jardir}/content.xml" tofile="${jardir}/content_orig.xml" /> <!-- Process content.xml with <xslt> --> <xslt style="content.xsl" in="${jardir}/content_orig.xml" out="${jardir}/content.xml" /> </target> Alternatively, if you just want a shell script for cron, the previous xmlstarlet example [cmd: xmlstarlet tr content.xsl content.xml] should work for the xsl transform, assuming you can install xmlstarlet. But you probably want Ant or PHP for portability.
I'll take this bug and will do the Ant integration. I will install an Eclipse instance on the Babel machine, and call it through ant with php. I just did for STP BPMN here: http://dev.eclipse.org/svnroot/stp/org.eclipse.stp.bpmn-modeler/org.eclipse.stp.bpmn/trunk/build/Buildfile with Ruby calling the p2 metadata generator.
Note that you don't want to use the -compress at generation time because it will produce a jar that you will then have to unzip before transforming with XSLT. However once the transform has been done you will want to jar up the new content.xml.
*** Bug 256252 has been marked as a duplicate of this bug. ***
*** Bug 254581 has been marked as a duplicate of this bug. ***
(In reply to comment #22) > *** Bug 254581 has been marked as a duplicate of this bug. *** > OK, I uploaded an Eclipse platform to the babel machine and ran the metadata generator, it works as expected. I am doing the XSL part.
Changing the component from Plugins to Server.
Created attachment 122980 [details] Changes in generate1.php to add an extra step of generating metadata This patch makes use of the XSL stylesheet Sean provided. It also makes use of the Eclipse metadata generator. However it doesn't jar the content.xml and artifacts.xml files - just yet. It fixes this bug, but we might want to add an extra step to jar up the p2 files.
Committed the patch to staging. Let's try this on!
(In reply to comment #26) > Committed the patch to staging. Let's try this on! Could someone please remind me the URL for staging? http://build.eclipse.org/technology/babel/staging-updates/ganymede/ is from last October. Also, is there a staging equivalent for downloadable packs like: http://download.eclipse.org/technology/babel/babel_language_packs/ ?
The Nightly Build update site is at: http://build.eclipse.org/technology/babel/test-updates/ganymede I don't think we publish the language pack zips nightly. Denis or Antoine maybe able to answer the question.
Ho hi! Sorry, the staging update site is disabled if I understood correctly, and I have no rights to change the crontab to do something about it. So here is what I do currently, I login to babel, go to the staging area and run: nohup php5 generate1.php & It takes about an hour to execute. I did that twice yesterday, to see that I had two problems: -runMetadata.sh was not directly executable. Had to add "sh " before it. -the eclipse folder is removed from babel-working, as it is cleaned every night. I moved eclipse to /home/genie/eclipse, should be safer. I am running it again today. I should have results in an hour. I will attach the content.xml here.
(In reply to comment #29) > I am running it again today. I should have results in an hour. I will attach > the content.xml here. I don't suppose that means content.xml (and artifacts.xml?) will appear at http://build.eclipse.org/technology/babel/test-updates/ganymede/ in about an hour? If not, are you able to put them there, please?
(In reply to comment #26) > Committed the patch to staging. Let's try this on! > Do you want me to build an update site based on the staging code before we put this live?
That's what I'm doing it now... and it's still very much in progress. I target a fix for the end of the week. Stay tuned, don't worry.
(In reply to comment #32) > That's what I'm doing it now... and it's still very much in progress. > > I target a fix for the end of the week. Stay tuned, don't worry. > OK, the upload to the build.eclipse.org machine works finally. The content.xml and the artifacts.xml files are present, and content.xml contains the greedy tags. However, I don't know how to make that build visible on build.eclipse.org. Right now it is located in /shared/technology/babel/staging-updates. Denis, can you please copy that to the right place ? Pascal, please tell me if it's a very bad idea if the content.xml and artifacts.xml files are not jarred, and if you would have a utility to do the compression, or if I need to jar this stuff manually.
Thanks to Denis help, the staging updates are available here: http://build.eclipse.org/technology/babel/staging-updates/ganymede/ It contains the content.xml and the artifacts.xml. Now let's test this stuff.
Created attachment 123480 [details] Errors when uninstalling, after installing langpack Well, the good news is that it doesn't seem to be any worse. I tried it with a Ganymede 3.4.0 (I20080617-2000) instance I had lying around, and it was able to install the eclipse-en_AA feature. It took forever because of the uncompressed XML, but it worked (ie it installed the pseudo langpack, and it worked after a restart). However, when I tried to uninstall an unrelated plugin (gted), I got some familiar error messages (attached). Sorry, but it took so long to install that I don't want to try it again (eg with a fresh Eclipse) until the XML files get jarred. 25MB of metadata is too much!
(In reply to comment #35) > Created an attachment (id=123480) [details] > Errors when uninstalling, after installing langpack > > Well, the good news is that it doesn't seem to be any worse. Bummer. OK, I'll try adding a jar command. Note that it is slow because build.eclipse.org is not supposed to be serving files so much. It's a busy beast.
I should point out that it's probably a pretty messy installation. I think I may have installed the langpacks into that Eclipse before, so it might already have been broken. It might be best to ignore the above results entirely.
Sweet glimmer of hope. I am enjoying it for yet a few seconds and I will try to install the language packs on top of a pristine Platform. Thanks!
OK, so I downloaded the site and installed a language pack as a local language pack. Then I tried to install something else, and it worked. Now, it seems that doesn't work when using the site as remote. So I think for remote sites it is necessary to have a content.jar.
I still reproduce the problem when installing, even with the content.xml and artifacts.xml jarred. Pascal, can you please look into http://build.eclipse.org/technology/babel/test-updates/ganymede/content.jar and tell us what is wrong with it ? The greedy and the optional attributes are in there, applied through the XSL stylesheet Sean provided. I jarred the content.xml and artifacts.xml directly using the jar command. Should I have used something else ?
I'm marking this bug as a blocker. There is no point at distributing language packs if they break the Eclipse instances they are installed in. Pascal, please do come back to us on this.
On top of 3.5 M5, I have successfully installed the french pack and everything worked properly. After the install, I have been able to update some software. Could you please be more specific as to what the problem you are experiencing is? My suspicion is that the obtention of the content.jar fails (for a reason or another) and p2 then reads the site.xml and generate p2 metadata on the fly that exhibit the same problem than the original one. What I recommend you to to is to not make the site.xml available on the update site (to avoid p2 falling back to it) and only make the jar'ed version of the files available. In short, to me this bug is fixed.
I tested the installation with 3.4.1. That might be why things don't go so well. I'll try with 3.5M5.
From a dependency resolution standpoint, things should also work with 3.4 as well. Once you have removed the site.xml, try again on 3.4.
(In reply to comment #39) > OK, so I downloaded the site and installed a language pack as a local language > pack. > > Then I tried to install something else, and it worked. > > Now, it seems that doesn't work when using the site as remote. > So I think for remote sites it is necessary to have a content.jar. I finally got around to trying it again with a pristine copy of 3.4.1 on Linux, although I used the wrong update site, and had to download the 22M xml file again. D'oh! Anyway, I am seeing exactly the same thing as Antoine. Using the remote update site <http://build.eclipse.org/technology/babel/staging-updates/ganymede/ > (which contains site.xml, content.xml and artifacts.xml) I get the "Unsatisfied dependency" errors when I try to install other plugins after Babel. I don't have a local mirror of that site, but whenever I install the JBoss Tools langpacks (processed via the same xsl) they work fine. Before the xsl, I used to get lots of "Unsatisfied dependency" errors, just like the Babel langpacks under P2. The xsl fixed them for me. But I haven't tested my JBT langpacks via http. (I'm trying to now, as soon as I can get a suitable pristine installation.) It does sound as if P2 in 3.4.1 uses the correct P2 metadata for *local* update sites, but uses site.xml (if available) for *remote* sites. Sounds like a bug in P2 to me. But as Pascal suggested, we can work around it by removing site.xml from the update site. Bit of a shame if Babel can't support both P2 and non-P2 installations of Ganymede though.
(In reply to comment #45) > But I haven't tested my JBT > langpacks via http. (I'm trying to now, as soon as I can get a suitable > pristine installation.) Okay, I've tested this now, with a local httpd. I'm using JBDS 2.0CR1, which is apparently based on Eclipse 3.4.2.something. And the langpacks work fine, whether site.xml is available or not. And whether I'm using the file: or http: URLs. > It does sound as if P2 in 3.4.1 uses the correct P2 metadata for *local* update > sites, but uses site.xml (if available) for *remote* sites. Sounds like a bug > in P2 to me. No, what I said above is all wrong. Looking at the log for my local httpd instance, Eclipse 3.4.x hasn't once requested the site.xml. It only does so if I force it to, by including site.xml as part of the update site URL. If I do force the use of site.xml, I can install my langpacks, but I can't install an unrelated plugin (FileSync) afterwards. So it breaks the installation, same as before. In light of the above, I've just done my "pristine" 3.4.1 test again, with the Babel langpacks. Last time (comment 45), I used a directory which was unzipped a while ago, but supposedly never modified, and I was using the update URL http://build.eclipse.org/technology/babel/staging-updates/ganymede/ (with the 22MB content.xml). This time, I extracted a brand new copy from eclipse-SDK-3.4.1-linux-gtk.tar.gz, and I used the update site at http://build.eclipse.org/technology/babel/test-updates/ganymede/ (which has content.jar). And it seems to be working fine. The "eclipse" langpack installed easily enough, it works, and I am still able to add and remove plugins. I obviously wasn't careful enough to keep my "eclipse-ganymede-pristine" directory genuinely pristine. I apologise for my misleading test results. In short, there seem to be two remaining cases which lead to the "Unsatisfied dependency" errors (when installing/updating plugins after langpacks): 1. The Eclipse installation (P2 configuration) has already been messed up, probably by earlier versions of Babel. 2. Using a "site.xml" URL, which prevents use of the new, more relaxed, metadata in content.xml/jar. Other than that, it's working absolutely fine! Antoine, is it possible that your "remote" url included site.xml, eg http://build.eclipse.org/technology/babel/test-updates/ganymede/site.xml ? If removing site.xml does fix the problem for you, on reflection I think we should remove it from the update site. It means we can't support non-P2 installations (saves testing that case!) but the presence of site.xml can lead to broken P2 installations. As you say, that's a blocker.
Thanks for the testing work Sean. I added an extra instruction to remove the site.xml.
I'm going to mark this fixed as it now works allright from what we could look at.
(In reply to comment #48) > I'm going to mark this fixed as it now works allright from what we could look > at. I see it's already made it to the main site; great! Couple of things though: - I don't think content.jar should contain artifacts.xml; I think they're meant to be in two separate jars. - Weren't you going to remove site.xml?
The fix to remove site.xml is pending on staging. When we release I will close this bug. Dunno for artifacts.xml.
Artifacts.xml should not be included in the content.jar, however it's presence will not hurt.
OK, I removed it from the jar. Thanks for the heads up.
(In reply to comment #52) > OK, I removed it from the jar. Thanks for the heads up. Thanks! Could you please jar up the artifacts.xml into artifacts.jar? 4MB -> 200KB is fairly significant.
OK, done that as well. Sean, where did you find instructions regarding what to jar and when ? Sounds like I obviously missed a tutorial or something.
