Bug 376033 - automate "uploading" from build machine to download server
Summary: automate "uploading" from build machine to download server
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 4.2   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: David Williams CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 355430
  Show dependency tree
 
Reported: 2012-04-03 22:45 EDT by David Williams CLA
Modified: 2012-04-26 23:43 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2012-04-03 22:45:21 EDT
With doing builds on build.eclipse.org, we (or, I) need to see how to "mirror" the build-time repos, so they are compared to the right repos. I guess the very first one could be "seeded" by pointing to the last good 3.8 I build .... and then go from there. 

After that, there are more complicated problems, to discuss in bug 375654 but _this_ bug is just to make sure the 4.2 primary builds are setup to do it and that they do in fact do it. 

This could already be working, as far as I know, but ... best to check before letting plugins "out into the wild" (even in a public I build).
Comment 1 David Williams CLA 2012-04-07 00:43:38 EDT
Just to note it, I would to "seed" the reference repo with 3.8 repo ... I guess the last good I build? (or milestone?) ... because the 4.2 repos have always been "messed up" (having both 3.8 and 4.2 content .... I don't trust them :) 

Anyone objections? I would think any 4.2 specific content would be rapidly changing anyway and no risk of having "same qualifier but different content"? 

Guess I could do something fancier if needed, but thought I'd make a note of overall approach.
Comment 2 David Williams CLA 2012-04-09 00:30:50 EDT
I'm not going to do what I said in comment 1 after all. 

It appears this is all set up and working again latest (current) I builds on download.eclipse.org, should be close enough (i.e. extra work not worth it). 

I did open a future enhancement, bug 376307. 

The hardest part of this will be that once an I-build is "published" up to downloads, its currently unclear how/where that works but in theory could do "by hand" for a few days if worse came to worst. So, that will be now be the focus of this bug, how to "publish" and "update" these repos on downloads.eclipse.org. Must be in there somewhere (still labeled "e4" and in some cases "3.8" :) 

As I recall, from WTP, the hard part is not updating the reference composite repo after a new build, but keeping it "accurate" as I-builds are removed after a milestone.
Comment 3 David Williams CLA 2012-04-09 01:42:09 EDT
Just so I don't forget when I come back to this, it appears there is (already) an update site being created/produced on the build machine, under 
.../eclipse4/build/supportDir/updates/4.2-I-builds
that looks "just like" it should when on downloads. 

I'm not sure if the design is this is supposed to be on "downloads" directly, when we are ready, but in the worst case, we could probably use rsync -a --delete and keep the build machine "in synch" with the downloads machine (without recopying everything, since we specify "archive" paramter it'll maintain dates, etc.). 

Will investigate. Just making notes.
Comment 4 David Williams CLA 2012-04-11 03:35:32 EDT
I'm going to change the meaning of this bug. I think the comparator is working fine, as it, and the key part (as mentioned earlier) is updating the "reference repo. The reference being the builds on downloads.eclipse.org. 

FYI, as noted in bug 375782 I've changed the location of the build machine "update site" from 
.../eclipse4/build/supportDir/updates/4.2-I-builds 
to 
.../eclipse4/siteDir/upates/4.2-I-builds

Essentially what I need to automate is moving an I build under 4.2-I-builds to its corresponding location under 4.2-I-builds on download server, and then updating the composite* files there. 

From some old scripts left around, I think the way this used to be done was to rsync the whole 4.2-I-builds directory from build machine to downloads machine (which could be made efficient) and that might be part of the answer, but I'd like to add other processing like p2.process.artifacts, etc. at this point, also. 

In the mean time, I can "manually" move stuff from one location to another until it gets automated.
Comment 5 David Williams CLA 2012-04-26 23:43:26 EDT
This is essentially fixed, now, along with bug 377033. 

with the e4Build ID that is in charge of the build, I create a "promotion script" that I put in a special place, and then from my committer ID, I can execute that promotion script to upload the build. I do have a cron job that "peeks" every 10 minutes to see if there is a promotion script and if not does nothing. If it finds one, it executes it. (I think I've documented this elsewhere .. but ... don't see it right off).