Bug 325181 - We should use "comparator" to detect changed bundles
Summary: We should use "comparator" to detect changed bundles
Status: RESOLVED FIXED
Alias: None
Product: WTP Releng
Classification: WebTools
Component: releng (show other bugs)
Version: 3.10   Edit
Hardware: PC Windows 7
: P3 enhancement (vote)
Target Milestone: 3.10.0   Edit
Assignee: David Williams CLA
QA Contact: David Williams CLA
URL:
Whiteboard:
Keywords: plan
Depends on:
Blocks: 322601
  Show dependency tree
 
Reported: 2010-09-13 19:15 EDT by David Williams CLA
Modified: 2018-06-29 15:30 EDT (History)
0 users

See Also:


Attachments
example comparator output (122.91 KB, text/plain)
2010-09-13 20:12 EDT, David Williams CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2010-09-13 19:15:17 EDT
There are some tools/techniques that can be used during a build (or mirror operation) to detect if a bundle has changed, but its version (even qualifier) has not changed).  We should add this functionality to our builds. 

For more information, see, for example, 
http://aniefer.blogspot.com/2009/08/versioning-p2-slides-from-eclipsecon.html

I'll attach a sample log I've obtained in some early testing. 

We'll have to add some "releng tests" to "fail" if unexpected errors show up in a log (and/or, improve "comparator" not to fail/log certain types of errors, such differences in "about.mappings". 

One complication of using this process, is that there can be cases where what is "in the latest build" ends up not being what is delivered. For example, if someone uses a final static constant from some other pre-req package, this will be "inlined" into the bundle. If that prereq bundles happens to change the value of that constant, then it will not be required for the consumer to increment their version (or, at least, retag with current date/time) so "new constant" version will appear newer, and be picked up, instead of having exact same version as old constant version, so p2 and installers will not know it was supposed to get a new version.
Comment 1 David Williams CLA 2010-09-13 19:20:44 EDT
>> then it will not be required for the consumer to increment their version

should have been it "will be required".
Comment 2 David Williams CLA 2010-09-13 19:23:35 EDT
In short, the thing to remember is, we version our source (and that sets bundle version) ... but other things can effect the finally built byte codes. 

The advantage, of course, is that we need to detect these situations anyway, at some point, or else things will mysteriously go wrong with some updates or installs. 

so, this new process should help us find it earlier, before release.
Comment 3 David Williams CLA 2010-09-13 20:12:34 EDT
Created attachment 178782 [details]
example comparator output

Most of these "errors" are about the "about.mappings" file. It is normal for that to change from build to build, and infact would be good to not "update" a bundle just based on the date it was built alone. At first, I was thinking it might be good to get ride of these about mappings files ... we don't especially use them ... but conversely, it might actually be good to this this record or "build date" inside the jar, especially if we don't update/ship new ones, if build date was only thing that changed. (for bugs related to these "poor" messages, see bug 325067 and bug 325134).

So, minus the about.mappings changes, there's only 4 "issues": 

Difference found for packed: osgi.bundle,org.eclipse.wst.xsl.sdk.documentation,1.0.100.v201006012005 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	Binary file build.xml: sizes differ by 15 bytes.

Difference found for canonical: osgi.bundle,org.eclipse.wst.xsl.sdk.documentation,1.0.100.v201006012005 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	Binary file build.xml: sizes differ by 15 bytes.

Difference found for canonical: osgi.bundle,org.eclipse.wst.jsdt.core.tests.model,1.0.301.v201007080225 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	IOException comparing /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar6259798808892428264.jar and /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar3276306043517706990.jar 
	error in opening zip file

Difference found for packed: osgi.bundle,org.eclipse.wst.jsdt.core.tests.model,1.0.301.v201007080225 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	IOException comparing /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar7150382796895120839.jar and /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar5255069612613771745.jar 
	error in opening zip file

two of those, are obviously test cases, where jars/zips can not be opened. 

The other two, might deserve some attention. I don't think a "build.xml" file should normally be "in" a delivered bundle anyway, but doubt its actually hurting anything.
Comment 4 David Williams CLA 2010-10-13 14:18:47 EDT
I've implemented this new process in our scripts. Probably needs a little fine tuning so will leave this bug open a bit longer. 

Note I have written up a general description in on of our WTP Build wiki pages: 

http://wiki.eclipse.org/WTP/Build/Accounting_for_history_during_builds

(which also needs a little more fine tuning :)
Comment 5 David Williams CLA 2011-05-17 14:32:59 EDT
fixed long ago