Community
Participate
Working Groups
This bug is just to address any specific issues related to doing 3.8 I builds on build.eclipse.org. See related bug 376216, bug 376182, as well as the big one of moving eclipsebuilder to git in the first place, bug 375807.
I've looked at the differences between building 4.2 and 3.8 and will document some here ... the end result is I could use some "git advice". So, 4.2 is working with the 'master' version of eclipsebuilder. The previous builder for 3.8 is now R3_development, in git. I think there is a small enough set of differences, that we could just use one version of eclipsebuilder, to build both 4.2 and 3.8. There would be changes needed to make that happen, but long term, I think easier to maintain one version of eclipsebuilder, to the extent possible. Some of the changes/differences could be handled by improvements to the ant code, and others would require a few "critically different" files to the maps repo/project, and then "get" the correct one to overlay or put into correct place, once we have checked out eclipsebuilder and the releng.maps project. We already do that for some files, such as "clickthroughs", so it'd be the same idea. The critical files (or file related) areas I've seen are: p2.inf for platform p2.inf for sdk repos.properties (used for misc. ant properties to codify names and locations for various repos.) ... there might be easier/better ways, since mostly differ by "eclipseStream" name. emf pre-req/repo. (Currently this is sort of "hard coded" in the master version of eclipsebuilder, but ... with a few tweaks, we could get a property from releng.maps file (which would not exist in 3.8 maps directories) so in 3.8 build we'd skip that step, but in 4.2 ue the emf.repo codified in that file (which, I'd like to make that change anyway, as mentioned in other bugs). The other class of differences/changes is that there are a number of places we have hard coded come names similar to 4.2.0.@qualifier" vs. 3.8.0.@qualifer@ where the @qualifier@ is already substituted by ant scripts, so we'd just change that to something like @eclipseStream@.@qualifier@ and replace with the value of "${eclipseStream}" (which we already have as a value in ant. So, with those differences, we could build 3.8 or 4.2 with simple changes in the command line: $buildRoot/masterBuild.sh -buildType I -eclipseStream 3.8 -buildRoot $buildRoot -mapVersionTag master vs $buildRoot/masterBuild.sh -buildType I -eclipseStream 4.2 -buildRoot $buildRoot -mapVersionTag R4_HEAD vs $buildRoot/masterBuild.sh -buildType N -eclipseStream 4.2 -buildRoot $buildRoot -mapVersionTag R4_HEAD But, admittedly those changes could take a couple of days, and likely be bugs here and there ... differences that are not obvious to me? So, the git approach ... I wouldn't mind trying to "merge"? master into R3_development and leave the "significant" differences, such and 3.8 name, p2.inf files ... and give R3_development a whirl ... but ... I'd like some hand-holding while "merging?" master and R3_mainentance. Even with help, I'm a little concerned that'd be error prone and still spend a lot of times tracking down bugs/differences ... and know for sure without help/hand-holding I'd definitely make errors :) So, if we did want to try "merging" ... I think it'd be backward from usual, right? I'd have R3_maintenance loaded as my workspace, then in git repo view select master and "merge"? I've tried that and a number of changes "come in" fine, which I'd assume I could fairly safely keep, then there's about a dozen "conflicts" ... for those, I'd just pick what's correct for R3_maintenance, and "add" that conflicting file to R3_maintenance, right? commit, then push. then build. :) Advice welcome.
Instead of merging, another approach could be to parametrize the 4.2 build so that it builds 3.8 and then verify that we covered all the little differences.
(In reply to comment #2) > Instead of merging, another approach could be to parametrize the 4.2 build so > that it builds 3.8 and then verify that we covered all the little differences. Yes, that would be my preference. It didn't seem the direction "the team" was headed when I took on this releng task, so wanted to be sure I wasn't cutting against to grain, or otherwise missing some less than obvious reason. Since we're using master for make 4.2 I and N builds now, I'll still make my own "feature branch" to do the 3.8 work in and then merge those back into master once its working fairly well. I think that's a fairly standard work flow I can figure out. Thanks for the sanity check confirmation. I'll start that 3.8 I-build work later today.
To report status ... My "experiments" are working out well, and as predicted, except that I had to spend more time wrestling with Git (or, maybe its Ant?) (bug 377132) than with the changes to the Eclipse Builder itself. The only "surprising" thing I hadn't anticipated, was that some tasks ... such as getting the map files/map project had to be "moved up" earlier in the process than it was previously which had a little ripple effect (meaning a few other things had to move around, be defined earlier, etc.). [This probably has a positive side effect of making it easier to fully integrate with "auto tagging" at a later date. Correct me if I'm wrong, but I think all the "map related" tasks in customTargets.xml will no longer be needed? I've been doing this work in a feature branch of eclipse builder, and maps, named david_williams/38Ibuilds. If anyone wants/has time to review. I'm doing a final (unsigned) test build now, then then may try a fresh signed build (to make test build easier, more valid to compare) with last known good 3.8 I build. Then (or, in parallel) I'll be looking at "merging back" the changes from my feature branch back into master. The merge back into eclipse builder master and maps project should be straight forward (I've read http://nvie.com/posts/a-successful-git-branching-model/ that when integrating/merging back a "feature branch" its often useful to NOT use fast forward, to leave a more traceable version of "what had to change to implement this feature" but ... not sure I can easily pull that off ... not to mention there were a lot of changes, that were later un-changed, so the end result is not as much changed as it might appear). BUT, after the "merging back", the tricky part, in terms of timing, will be to make the same corresponding changes to the R4_HEAD branch of maps, so as to not interfere or risk N builds or I builds. I think it'll be easy enough, but there is a little risk of disruption there. (e.g. could not reliably do an I-build respin today, or might have to skip tonight's N build if the timing is bad). I'll use next comment for substance of changes.
The most visible change is that in the maps project, there will now be a new folder, /org.eclipse.releng/configuration that has "build related" things that vary between streams. We will always want to keep things in there to a minimum (the whole purpose of this is to avoid dual maintenance!) but there are some valid things where its easier just to have a unique copy. There are two main entities in this configuration folder /org.eclipse.releng/configuration/streamSpecific-build.properties /org.eclipse.releng/configuration/eclipseBuilderOverlays The properties file is where we can have "control variables" such as the emfRepo we need for 4.2, but not for 3.8. So the property will be defined in the 4.2 (R4_HEAD) version of the file, but not in the 3.8 (currently master) version of the file. The eclispe builder will always have the "get emf pre-req" task in it, but if the emfRepo variable is not provided, it won't execute. Ant 101. The eclipseBuilderOverlays folder is where we "mimic" the exact directory hierarchy of the eclipse builder itself, and literally overlay, at beginning of the build, any files found in eclipseBuilderOverlays into the build's area for org.eclipse.releng.eclipsebuilder. So far there are 3 such files, 3 p2.inf files. (But, it is conceivable, in future, if "large changes" made in product definitions, etc., that more files might deviate. My current thinking is that we will have "eclpise builder" accurate for 4.2 build (what ever is "most forward") and hence nothing in "eclipseBuilderOverlays" in that branch of maps project. The other substantive change that makes a lot of this possible is that the build now requires our already existing "eclipseStream" variable to be specified for all three parts, major, minor, and service, we programatically break that into eclipseStreamMajor, eclipseStreamMinor, and eclipseStreamService and once eclipsebuilder is "in place" and we've done the overlay, then we use ant's <replace task to "fix up" the files where they specify eclipseStream (in some places, we need all three segments, and just use ${eclipseStream}, in other cases, we need only major and minor, so use only those. Most of the "new" programming changes occurred in buildAll.xml, where I added a getMapsProject task, and a streamSpecificInit task. The proof will be in the pudding. :)
Doing a "full" test build of 3.2 I build now (signed and all), and by chance noticed this message in the log ... maybe its always been there ... but ... seemed odd. Maybe someone else will know off the top if their head if any of this sounds like a problem for "delta packs". (I think if those features didn't exist at all, the build wouldn't have gotten this far! so maybe a quirk of how delta packs created.) I'd assume (little that I know) the "delta packs" for 4.2 and 3.2 would be the same, right? build: [mkdir] Created dir: /shared/eclipse/eclipse3I/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/rcp.deltapack/featureTemp [mkdir] Created dir: /shared/eclipse/eclipse3I/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/rcp.deltapack/featureTemp2 [p2.mirror] Messages while mirroring artifact descriptors. Unable to find: Installable Unit [ id=org.eclipse.jdt.feature ] Unable to find: Installable Unit [ id=org.eclipse.jdt.source.feature ] Unable to find: Installable Unit [ id=org.eclipse.platform.feature ] Unable to find: Installable Unit [ id=org.eclipse.platform.source.feature ] Unable to find: Installable Unit [ id=org.eclipse.rcp.feature ] Unable to find: Installable Unit [ id=org.eclipse.rcp.source.feature ]
(In reply to comment #6) > Doing a "full" test build of 3.2 I build now 3.8 that is :)
(In reply to comment #6) > > build: > [mkdir] Created dir: > /shared/eclipse/eclipse3I/build/supportDir/org.eclipse.releng.eclipsebuilder/eclipse/buildConfigs/rcp.deltapack/featureTemp I see no evidence in the last 4.2 I build _log_ that we were doing anything with "delta packs". Wonder if they are just missing from 4.2? Haven't check repo or anything. But could use some expert eyes here just to see if we are missing it in 4.2? Or shouldn't be doing what we are dong 3.8? [I'll keep looking of course, but this is an area I know little of, so for me its mostly "file compares" :)
One problem that slipped through the refactoring of "eclipseStream" to mean all three version parts is there were all sorts of places we used some form of .../updates/${eclipseStream}-${buildTtype}-builds but what we really want is (both, conceptually, and to match what we had) .../updates/${eclipseStreamMajor}.${eclipseStreamMinor}-${buildTtype}-builds This was a problem with tonights "N build", you can see the note says Software site repository: http://download.eclipse.org/eclipse/updates/4.2.0-N-builds instead of what it used to say Software site repository: http://download.eclipse.org/eclipse/updates/4.2-N-builds Same think happened to the 3.8-I build, which is more important, so we can be sure we end up with an composite site at .../updates/3.8-I-build instead of a brand new one at .../updates/3.8.0-I-build I can fix that "by hand" ... since I noticed before "promoting" it. For the N build, I think I'll just leave as it, since from the retention policy those are replaced twice a week or so anyway.
Just to document it, Some other "little things" I'd forgotten about till I went to promote. The "drop location" was hard coded in masterBuilder to "drops4", but for 3.8 builds should be just "drops". While I was at it, I fixed the equinox drop location on build machine to be end with "drops3", instead of "drops", for 3.x builds. We already had logic not to promote 3.x build of equinox, but having them in a separate directory will help avoid confusion, and make it easier to compare, on the build machine, a 3.8 equinox build with a 4.2 build (they should be identical, if nothing in equinox changed). I also had to update the "promote script" for eclipse to deal with the download file summary page. For 4.x a file named create createIndex4x.php is used to create index.html For 3.x a file name eclipse3x.php is used to create a file named eclipse3x.html. In case not obvious, my goal is to have one promote script, which "does the right thing based on build stream (3.8.0, or 4.2.0, currently) the build type (I, or N, currently) and name of the build (e.g. I20120417-1000).
(In reply to comment #6) > Doing a "full" test build of 3.2 I build now (signed and all), and by chance > noticed this message in the log ... maybe its always been there ... but ... > seemed odd. Maybe someone else will know off the top if their head if any of > this sounds like a problem for "delta packs". (I think if those features didn't > exist at all, the build wouldn't have gotten this far! so maybe a quirk of how > delta packs created.) I'd assume (little that I know) the "delta packs" for 4.2 > and 3.2 would be the same, right? I don't recognize the error.. I don't know if that error was there before or not. I can say from looking at the latest 4.2 I-build that the delta pack appears to be broken. In particular, it doesn't contain SWT. I don't know if that maps to these errors you are seeing but there's definitely a delta pack problem. Maybe we could use bug 339430 to track that.
I think this is fixed enough to count as fixed. Future issues/problems should be new bugs.
