Community
Participate
Working Groups
feature.xml of I20071029-1800 looks like this: <feature ... version="3.4.0.v20070824-7M7Q_7BIkmNVnZXl-MDVHmvfwcE0" ... <includes id="org.eclipse.platform" version="3.4.0.v20071029-9y9jFD8Ezbubi1ctDPXiKX_TEz01"/> ... </feature> feature.xml of I20071030-0010 looks like this: <feature ... version="3.4.0.v20070824-7M7Q_7BIkmNVnZXl-MDVHmvfwcE0" ... <includes id="org.eclipse.platform" version="3.4.0.v20071029-9y9jFD8Ezbubi1ctDPXiKXA5F209"/> ... </feature> Note the last six digits of the platform qualifier have changed, but the SDK qualifier has not changed.
Created attachment 81655 [details] Zip of the features directories from these two builds
Full length qualifiers before truncation would have been: I20071029-1800 7M7Q_7E7QYGHEMfV681Xz0p8yaHTjJZ9RW7UwHZt4PLHQ-Pi ^ I20071030-0010 7M7Q_7E7QYGHEMfV681Xz0p8yaHTjJZ9RW7UwHZt4Pz-HV-Xi ^ The qualifier did increase, however, the default length (because of path length issues) is 28 characters and the change is at character 43. (z > L)
The qualifiers in comment #2 don't match the qualifiers in comment #0. These are for the SDK feature?
hmm, I don't know why I didn't notice that. Yes these were for the SDK feature. My test was cheating a little, I will have to check again that it isn't doing something slightly different. However, I expect the result will be the same wrt the length
Andrew and I were just chatting about this and currently when the versions are truncated, they are truncated in both the manifests and on the file-system. It might be tricky to get right, but there isn't any reason why we can't leave the versions in the manifests alone and just change the file-system paths.
So the idea is that the actual feature version is something like 3.4.0.v20070824-7M7Q_7E7QYGHEMfV681Xz0p8yaHTjJZ9RW7UwHZt4PLHQ-Pi but that the jar/folder would be named org.eclipse.sdk_3.4.0.v20070824 One thing to check to make sure it works with this is source lookup since the source plugins get the same version as the feature so a similar mismatch would exist there.
This doesn't really fit with the approach in p2 where all bundles are thrown into a single pool. If I were to update from I20071029-1800 to I20071030-0010, I would have both of those features in my pool, so colliding directory names would still be a problem. Having said that, there may be more fundamental changes to how features work in p2, so this problem may go away...
Just to note, this happened again in the M5 test candidate builds - see bug 217726.
And again between the I20080206-1800 and I20080207-0010
It looks like we'll need to increase the qualifier length - this will be too painful when the larger team is self-hosting on p2. Now that source bundles are not nested within a source feature, our path length problems are not as bad.
We talked about this last week, and decided we need to increase the feature suffix length so we avoid this problem in the future. Andrew, can you recommend a length that would be sufficient? It seems from this bug that it needs to be at least 43 characters.
The property to set is "generatedVersionLength" and I think a value of 45 should be enough.
I changed the sdk and platform features to have a generatedVersionLength=45 yesterday
Changing title and moving to future. We have no particular ideas/plans for 3.4 in this area.
This hits us nearly every milestone. I suggest a fix where a build takes as input a list of versions produced by the previous build. The suffix generation can then ensure generated versions are greater than the previous ones. In the absence of this "advice", the build generates suffixes in the same way it does now.
Would it be sufficient for the pde build to have access to the previous's featureVersions.properties. Of course, this wouldn't be needed for nightly builds since they are all unique and represent the buildId. -bash-3.00$ cat featureVersions.properties #Thu Sep 18 03:56:05 EDT 2008 org.eclipse.equinox.p2.director.feature,0.0.0=v20080827-0900 org.eclipse.platform,0.0.0=v20080911 master-equinox,0.0.0=v20080506 org.eclipse.rcp,0.0.0=v20080911 com.ibm.icu.base,0.0.0=v20060418 master,0.0.0=v20080610 org.eclipse.cvs,0.0.0=v20080603 master-root,0.0.0=v20080911 org.eclipse.releng.tools,0.0.0=v20080603 org.eclipse.equinox.p2.user.ui,0.0.0=v20080827-0900 org.eclipse.sdk,0.0.0=v20080911 master-jetty,0.0.0=v20080506 org.eclipse.equinox,0.0.0=v20080611 org.eclipse.equinox.p2.agent.feature,0.0.0=v20080827-0900 org.eclipse.jdt,0.0.0=v20080915-0800 org.eclipse.test,0.0.0=v20080507 org.eclipse.equinox.jmx.server.feature,0.0.0=v20070510 org.eclipse.equinox.p2.generator.feature,0.0.0=v20080827-0900 org.eclipse.equinox.jmx.client.feature,0.0.0=v20070608 org.eclipse.sdk.examples,0.0.0=v20080717 org.eclipse.equinox.jmx.common.feature,0.0.0=v20070507 org.eclipse.pde,0.0.0=v20080908 org.eclipse.equinox.executable,0.0.0=v20080915-1045 org.eclipse.help,0.0.0=v20080917 org.eclipse.sdk.tests,0.0.0=v20080717 org.eclipse.pde.p2,0.0.0=v20080522 master-equinox-p2,0.0.0=v20080506
(In reply to comment #15) > This hits us nearly every milestone. > > I suggest a fix where a build takes as input a list of versions produced by the > previous build. The suffix generation can then ensure generated versions are > greater than the previous ones. > In the absence of this "advice", the build generates suffixes in the same way > it does now. > This doesn't seem like much of a fix. It'd require all build teams to "hack" their scripts to get this behavior. I'm also not sure how you'd know when to apply the hint (that is, when is it correct to "ensure generated versions are greater than the previous ones". But, I'm guessing you actually know the answer to that or else you wouldn't have suggested it. Not that I have any other suggestions ... I'd have to read closer and understand it to do that :)
Kim, we would likely need the finalFeaturesVersions.properties file. David, there are already properties to control the generation of the suffixes (generatedVersionLength, significantVersionDigits). These can be manipulated to partially address issues, but the platform team has had path length issues in the past if we use too many digits. So yes, there is no way around this other than teams updating their builds. The hint would be used as: if the normally generated suffix is not greater than the hint, then do something. Where something could be one of: 1) fail with a build error 2) generate a higher suffix by some method (perhaps increment the hint and use that)
An implementation note: We must consider the contents of the feature to decide whether or not it needs to be incremented over the previous version. (ie if children increased, then the feature must increase). If script & version suffix generation remain depth first, then we can assume we properly incremented nested features, and only need to consider the immediate children. In addition to finalFeaturesVersions.properties, we will also need finalPluginsVersions.properties.
If you dont' mind if I make an implementation suggestion ... It seems as though the discussion has implied the solution will be for the higher level build script to provide the files, that have the previous values. Hence all build scripts would need to change to take advantage of this. I wonder if it would be feasible for there to be a property to name a directory, and for the suffix Generator itself to remember it's own "history" there? (that is, for it to leave a copy there). Seems it'd be easier to add a property to an existing build script, rather than a some "copy" tasks at several points in the build. Thanks for considering. P.S. I think we've seen this once or twice in WTP too.
This has happened to us again, in WTP, in maintenance builds a few weeks apart. To quote the report I received: The feature org.eclipse.wst.xml_ui.feature (3.0.4.v200811211541-7F2ENnCwum8W79A1UYNgSjOcFVJg) has the same version there but the content is different. In the feature xml, 20090212194735 version has: <includes id="org.eclipse.wst.common_ui.feature" version="3.0.4.v200811200248-7C78ELhE8VrRVqtHp4iT8PuSwZ5W" /> In 20090205224313, it has <includes id="org.eclipse.wst.common_ui.feature" version="3.0.4.v200811200248-7C78ELhE8VrRVqtHp4iT8PuSvaAW" /> This is problematic for one adopter, whose build process assumes perfection in these qualifiers (that is, if qualifier doesn't change, they assume content hasn't changed). This maintenance stream is built with the "old" base builder of RC2_34.
I think this has happened again ... in a way that cause me the better part of 8 hours of time before it dawned on me. This time, it shows up in a composite repo, where org.eclipse.equinox.core.sdk has same id, but different content. http://download.eclipse.org/eclipse/updates/3.5.x/ M20100120-0800 M20100205-0800 namely, org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar they have different content. M20100120-0800 includes org.eclipse.osgi [3.5.2.R35x_v20100114] but M20100205-0800 includes org.eclipse.osgi [3.5.2.R35x_v20100126]
(In reply to comment #22) > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar > > they have different content. > Not sure I understand the issue. The qualifiers are different which means that the features have different versions. ----------------------------------------------------------------------------------------------| v org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar So two different versions of the core.sdk feature contain two different versions of the org.eclipse.osgi bundle. That looks all OK to me. What am I missing?
(In reply to comment #23) > (In reply to comment #22) > > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar > > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar > > > > they have different content. > > > Not sure I understand the issue. The qualifiers are different which means that > the features have different versions. > > ----------------------------------------------------------------------------------------------| > > v > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar > org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar > > So two different versions of the core.sdk feature contain two different > versions of the org.eclipse.osgi bundle. That looks all OK to me. What am I > missing? Ah, maybe I was reading it wrong (or, picked the wrong one to copy/paste). But, pretty sure there's some in there with same id, different contents.
(In reply to comment #24) > Ah, maybe I was reading it wrong (or, picked the wrong one to copy/paste). > But, pretty sure there's some in there with same id, different contents. Just to clarify, are you saying that you see more then one feature with exactly the same id and version but different content, or just the same id but perhaps with different versions?
(In reply to comment #25) > Just to clarify, are you saying that you see more then one feature with exactly > the same id and version but different content, or just the same id but perhaps > with different versions? Maybe I was seeing it wrong, but here's whole story. When using the composite repo, at http://download.eclipse.org/eclipse/updates/3.5.x/ the galileo build was failing with hard to explain errors. https://build.eclipse.org/hudson/view/Repository%20Aggregation/job/galileo.runBuckyBuild/823/console messages such as Cannot satisfy dependency: org.eclipse.equinox.core.sdk.feature.group 3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7 depends on: org.eclipse.osgi [3.5.2.R35x_v20100114] didn't make much sense, since where not in the "current" (most recent build). If I use only the most recent repo, it works fine. http://download.eclipse.org/eclipse/updates/3.5.x/M20100205-0800/ So, looking at it, I thought I say features with same IDs and versions, but different contents ... but, maybe I was reading the versions wrong, and missing where they differed by a letter or two: ./M20100113-0800/features/org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh1j3BB7.jar ./M20100120-0800/features/org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7.jar ./M20090917-0800/features/org.eclipse.equinox.core.sdk_3.5.1.R35x_v20090811-7sF9-FRFFR1En_CEJqJkQv5n3BB7.jar ./M20100203-0800/features/org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar ./M20100205-0800/features/org.eclipse.equinox.core.sdk_3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7.jar
You're right. I tracked this down. This is the culprit feature: org.eclipse.equinox.sdk.feature.group 3.5.2.r352_v20100118-7G7MAD7lQiEeFVMr1y1GvAQowEhS In http://download.eclipse.org/eclipse/updates/3.5.x/M20100120-0800/content.jar this feature has a requirement to: org.eclipse.equinox.core.sdk.feature.group 3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh3h3BB7 but in http://download.eclipse.org/eclipse/updates/3.5.x/M20100205-0800/content.jar the exact same version of that feature has the requirement: org.eclipse.equinox.core.sdk.feature.group 3.5.2.R35x_v20100105-7sF9-FRHFR1En_CEJqLTQh5l3BB7 which is slightly different (more recent) which explains why the build works OK when it sees just the last repository. I consider this a major bug since it breaks the notion that all IU's are immutable and uniquely identified by id + version. Here we have two variants.
Currently there are two options when this occurs: 1) Discard the build, retag and do another. This needs us to notice the problem right away. The mirror against baseline using a comparator should detect this problem, I believe Kim is looking at this. 2) Modify your builder to specify a longer suffix by setting generatedVersionLength to a big number. Otherwise, some input regarding the history of previous builds is required.
(In reply to comment #28) > Otherwise, some input regarding the history of previous builds is required. I've been dreaming about that for some time, see bug 265900. The input to the service outlined there could be very long qualifiers indeed. It wouldn't matter much.
*** Bug 310338 has been marked as a duplicate of this bug. ***
If this bug can not be fixed would it make sense for each component team to add an additional verification step to make sure 'indential features' really are identical ... perhaps simply 'diffing the feature.xml file would suffice. If features with exact same version and qualifier have different contents in feature.xml file, then the feature should be re-tagged to force the non suffix portion higher". It is ashame that this system almost works. It makes teams believe that they do not need to deal with history to deal with the feature version problem when in fact they must.
Can I ask for clarification on "what the platform uses"? In comment #13 its said generatedVersionLength=45 But in recent Helios SR1, I see the qualifier or org.eclipse.platform is r361_v20100909-9gF78GrkFqw7GrsZnvz0JWNTeb6fue6896L which is 50 I'm asking because I'm thinking of increasing ours in WTP. There a recently generated qualifier was v201008170029-7F77FJSC25Tkdy1nqglPjqLAoDgh which is 42 characters long. I'm assuming that's the default? So, I'm just wondering if there's much to (easily) gain by increasing to 45 or 50. The reason I'm asking is that now that I've added a "verification" step to check the features built in WTP, it seems like it fails the verification several times every milestone, and I'm just wondering if increasing the length will help reduce that ... or, if we are stuck with the frequent "manual" tagging to get around the problem. Any advice?
Yes, we still use generatedVersionLength=45 for the SDK, but you are correct, it actually is 56 characters. I'll have to investigate why this is happening.
(In reply to comment #32) > But in recent Helios SR1, I see the qualifier or org.eclipse.platform is > r361_v20100909-9gF78GrkFqw7GrsZnvz0JWNTeb6fue6896L > which is 50 The first part of that qualifier is the CVS tag. The generated qualifier in this case is only 36 characters long.
(In reply to comment #34) > (In reply to comment #32) > > But in recent Helios SR1, I see the qualifier or org.eclipse.platform is > > r361_v20100909-9gF78GrkFqw7GrsZnvz0JWNTeb6fue6896L > > which is 50 > > The first part of that qualifier is the CVS tag. The generated qualifier in > this case is only 36 characters long. ok, so difference between 36 and the property set at 45? Does it "grow to maximum" only if needed, or something? at any rate, this means the wtp feature's suffix is 29 ... so maybe some advantage to setting to 45 ... I just want to make sure we in WTP don't end up providing the longest directory names :/ Thanks,
(In reply to comment #35) > ok, so difference between 36 and the property set at 45? Does it "grow to > maximum" only if needed, or something? > > at any rate, this means the wtp feature's suffix is 29 ... so maybe some > advantage to setting to 45 ... I just want to make sure we in WTP don't end up > providing the longest directory names :/ > > Thanks, PDE/Build truncates the generated suffix at generatedVersionLength (45) and appends it to the existing tag/qualifier. You are correct that it only "grows" to the maximum if needed, if the generated suffix is less (36 here) then we didn't need to truncate and no information is lost and the version should properly increase. While writing this, one idea occurred to me: given a suffix that must be truncated, currently we just chop off the end. If instead we truncate and then increment the last character, then we should have an increased version. We then only lose version increments on the third build if it was truncated in the same way. Perhaps that help this occur a little less often.
> > While writing this, one idea occurred to me: given a suffix that must be > truncated, currently we just chop off the end. If instead we truncate and then > increment the last character, then we should have an increased version. > > We then only lose version increments on the third build if it was truncated in > the same way. Perhaps that help this occur a little less often. Sounds worth it to me. While you in that code, you might add a log message for "Warning: feature qualifier suffix was truncated at <nn> characters for <feature id>." Might help us all better know when it was happening so could better investigate solutions (especially if happened frequently). And to suggest a small tweak on the proposed heuristic, instead on "incrementing last letter", you might just append one. That'd seem simpler if/when to provide future enhancements where the builder could cache on disk (its own private location?) of "last letter used" and then (if found) just always use the next one in the sequence ... but, I don't mean to spoil it by asking for too much :)
For those of you that love statistics as much as I do ... in bug 328900 I've documented some "stats" about whether or not we can (afford to) increase our feature qualifier suffix, without "going over" the already longest directory names (to sum up, appears we can a little) ... but, thought you might be interested in what the qualifier size distribution looks like over a whole release. I wrote a little program to take the feature jar names from Helios SR1, used a heuristic to guess the suffix portion of the version qualifier, and the results are below in the form of a frequency table. It appears there's about 50 that are at or near the default truncation length of 28 (at least, I'm assuming its 28). That's about 5% of total. Of course, not all of those would necessarily "conflict" on a frequent basis ... but, 5% sounds like enough to warrant some effort to avoid clashes (or, detect them in post-build tests). (It also surprises me so many do not use any suffixes ... guess not everyone has such complicated features as WTP? :) qualifier length count 0 355 5 7 13 7 14 7 15 5 16 1 17 50 18 44 19 43 20 25 21 12 22 18 23 18 24 10 25 16 26 7 27 7 28 48 35 2 45 1 And, the winner with a 45 character qualifier is ... org.eclipse.sdk_r361_v20100714-0800-7Q7m6DDaKf5o2z-L9LxPSe6ygafz-KKIqk1rr_3j4dn7J good thing it has a short name :) enjoy :)
In general the length of the suffix depends on the things it includes and their qualifiers (including suffixes for nested features). Once you start nesting features, the suffixes start accumulating, this can be seen in bug 175714 when we looked into this before.
Just to cross reference, it appears to have happened (to someone besides me :) See bug 329890.
*** Bug 337053 has been marked as a duplicate of this bug. ***
It happened to me too, see bug 337053 :) This is about feature qualifiers only, right? I'm updating the summary since I had not found this bug searching bugzilla for "feature qualifier".
I believe [plan] should be removed from title line, since (as far as I know) there are no plans to fix this. I fact, I think it should be marked as "won't fix" since there seems little reason to fix it if PDE is not "primary builder". If anyone would prefer, it could be left open with "help wanted" tag, or something, since it is not a bad idea to fix it ... just, no plans to, as far as I know? I noticed this since I was trying to do a query on all '[plan]' items. Feel free to correct my edits if I have misunderstood.
