Community
Participate
Working Groups
This bug is a "Feature Design Specification" and "Request For Comment" all rolled into one. If you have suggestions/concerns/feedback, please post it here. -- As per bug 142565, cleaning up our old update sites has become a bit of a nightmare. Why do we need to clean up? Well, on download.eclipse.org, our update sites (EMF, EMFT, MDT, M2T) occupy a combined 5G of space, and the site*.xml files are increasingly huge and slow to parse. In talking with Christian today, we've concocted a possible new plan of attack that will require some retrofitting (regenerating old content) but will both save disc space and automatically cycle in new builds while trashing the old ones. Here's the proposal: a) each component will generate a project-component-branch-type-0, project-component-branch-type-1, project-component-branch-type-2 folder, eg: mdt-ocl-1.0-R-0, mdt-ocl-1.0-R-1, mdt-ocl-1.0-R-2, mdt-ocl-1.0-M-0, mdt-ocl-1.0-M-1, mdt-ocl-1.0-M-2, mdt-ocl-1.1-R-0, mdt-ocl-1.1-R-1, mdt-ocl-1.1-R-2, mdt-ocl-1.1-M-0, mdt-ocl-1.1-M-1, mdt-ocl-1.1-M-2, mdt-ocl-1.2-R-0, mdt-ocl-1.2-R-1, mdt-ocl-1.2-R-2, mdt-ocl-1.2-S-0, mdt-ocl-1.2-S-1, mdt-ocl-1.2-S-2, mdt-ocl-1.2-I-0, mdt-ocl-1.2-I-1, mdt-ocl-1.2-I-2, b) a 'copy and merge' process which will read from list of folders to collate, and generate a single update site for that whole project. For the MDT case, this would involve collecting all the latest builds for each component & branch into a pair of merged site.xml + site-interim.xml files, and copying all the jars into a central repo. So, instead of just getting bigger every week, the update sites will be self-replacing (only one I and S build per component per branch, for example), and only grow when a new release is available for a new branch (maintenance releases would replace previous releases on that branch). Thus for EMF's site.xml, we'd have 4 R EMF builds and 3x2 QTV builds = 10 entries; site-interim.xml would have 2x2 (I + S for 2.3 and 2.4 branches) + 2x2x3 (I + S for 1.1 and 1.2 branches) = 16 entries. c) The reason for the -0, -1, -2 folders is simply to have rolling backups. Part of the promote process is thus to delete the -2, rename -1 -> -2, and create a new -0 every week. We might not need all three, or in fact more than one. Thoughts? d) The added side effect here is that we'll be able to also do a cross-project "all of modeling" update site, if we want to, by simply extending the collection of dirs to merge. (Yes, this sounds a lot like what the Ganymatic does. However, the Ganymatic may not support multiple branches in a site, so I may need to deviate from there. Frankly, I'm not sure yet.)
Escalating priority and severity.
First draft is ready. Remaining issues include: + run prototype composite update site build for M2T (2 projects, 3 sites) + copy generated category.xml into Master zip (along w/ header), rename, and link on download site ? + TEST installation from archived update site + add site digest (bug 187339) + verify ganymatic .sc generation in promoteToEclipse.sh works w/ new paths + verify old URLs work even though /releases/ have moved + update feature.xmls to point to revised URLs
Found another issue -- wasn't accounting for automatic jar cleanup. Now, every time a site is updated from local to remote, any no-longer-needed jars will be deleted. Prototype M2T site gen works great, though the name of the site*.xml files differs on local and remote; local should be changed to match.
* site.xml is now used consistently on local and remote; site-ganymede.xml is used for the ganymede contrib (because Ganymede can't handle <feature url="http://...">). * added support for a digest (bug 187339).
* download stat tracking disabled (bug 228145) * list of valid jars & category.xmls are now merged from multiple server sources * site.xml and digest.zip are generated on the actual contents of the server rather than the local build server contents (previously assumed to be the same on remote). This assumption was only valid for the case where all project builds are published from a single build server, which breaks for M2T & MDT.
EMF, MDT, M2T new update sites rolled out. Disk savings (old -> new [saved]): * m2t: 49M -> 29M [ 20M] * mdt: 1.3G -> 288M [1.0G] * emf: 4.0G -> 451M [3.5G] One bug arose in Transaction's case, in that I can't publish Transaction 1.1.3's Maintenance build to the Interim site until there's a Query and Validation build there too, since Transaction's .qtv features bundle Q and V, and thus depend on those feature jars being present in order to generate the site digest. (There's no "ignore missing features" option in the digestBuilder tool, sadly.) This wasn't a problem in the past when everything just got dumped into one folder, but now that there are three sites per project, it's a little more complicated. Some options: a) publish M builds into the Releases site instead of the Interim site b) copy R bits into Interim site c) don't publish M builds to update site (provide standalone update zips only) Christian -- your thoughts?
(In reply to comment #6) > Some options: > > a) publish M builds into the Releases site instead of the Interim site > b) copy R bits into Interim site > c) don't publish M builds to update site (provide standalone update zips only) > > Christian -- your thoughts? Thanks, Nick. I like option (b), as the likelihood of Query and Validation producing 1.1.3 builds is small. I don't think we should publish releases that aren't consumable via the update site, and the releases site probably should be exclusive.
(In reply to comment #7) > > b) copy R bits into Interim site > The likelihood of Query and Validation producing 1.1.3 builds is small. > I don't think we should publish releases that aren't consumable via > the update site, and the releases site probably should be exclusive. So, ideally I'd need something like this: result = digestBuilder(); // first try if (!result) { copyMatchingRbuilds("/releases/", "/interim/"); digestBuilder(); // second try } For the Transaction 1.1.x M build case, this only duplicates 3.6M of jars, and as this is probably a one-off case, I'll just build it (for now) as a single exception clause, rather than trying to worry about making this fallback fully generic (eg., for other projects that "bundle" their siblings' features). if (isTransactionBuild) { copyMatchingQueryAndValidationBuilds("/releases/", "/interim/"); } digestBuilder();
Lucas has unearthed a new limitation: file permissions. Need to add an extra step after creating the sites: chmod -R g+w * chgrp -R www *
Permissions fixed. Also added p2 metadata as per bug 229913.
It's been suggested that we reduce the # of builds per site to 1 per type per branch per component (bug 271486). As such, I've changed the buildUpdateSite.sh script to do so. It goes into effect tonight. Closing.
