Generating a p2 repo can be done a number of ways, from the trivial case for a single feature build (add p2.gathering=true into your build.properties file, as discussed with slides and working sample here, to the more elaborate, with signed & packed jars (eg., using the buildUpdate task here).

Just bear in mind one caveat: while packed features work great on a regular p2 repo (eg., http://download.eclipse.org/modeling/emf/updates/milestones/, you cannot yet include packed features in the archive, only packed plugins.

If you want to produce an archived p2 repo but don’t care how it’s done… try an Athena build, and you’ll get one generated for you for free (as long as you enable the buildUpdate task to be run via your build.properties. If you’ve never tried Athena, I suggest downloading the slides & exercises and trying it for yourself (bottom of the page).

Should you want an unzipped version of your p2 repo published on download.eclipse.org (eg., so it can be consumed by EPP or Galileo), watch bug 266374. I’m working on improving this currently manual process - copy zip then unzip zip - so it’ll be less effort and more automated.

As always, feedback & contributions welcome.

A recent post in the p2-dev@eclipse.org mailing list got me thinking about the use cases for different ways to install software. Considering that the Linux world has this solved (and then some!) let’s look at the different ways I can update my recently repurposed xubuntu 8.04 laptop. Bear in mind this is without installing OTHER tools, just the ones that come OOTB with an xubuntu installation. (Yes, there are other choices too. Fedora has its tools, gentoo has its solution, etc.)

1. apt-cache: a commandline tool for querying the repositories for available packages, versions, and details.

2. apt-get: a commandline tool for installing/removing packages.

3. Adept Manager: a GUI tool to query the repos for available packages & to install/remove them.

4. Synaptic Package Manager: a more refined GUI tool to query the repos for available packages & to install/remove them.

5. Add/Remove Applications…: an application to ease Windows users into package management, for coarse-grained installation and/or groups of packages

6. Update Manager: a task tray resident application that monitors the repositories for updates and alerts users about available updates, also to ease Windows users into the Linux experience

So, do we need all of these? Perhaps not all 6, but linux distros are still trying to sort out their target audience, so they often include more tools that you need.

Sure, you can manage updates with apt-get, Synaptic, or Adept, but the Update Manager is smaller and more end-user focused.

Sure, you can install everything in Add/Remove Applications… with the tools above it, but Add/Remove Applications… is friendlier.

Personally, I use apt-get/apt-cache (if I more or less know what I want to install), Synaptic (if I want to browse for something new or install something with many dependencies), and Update (if I just want patches/security updates), because different tools are suited to different needs.

You can remove a screw with a coin, or hammer in a nail with a shoe, but there are better-suited tools for those tasks.

Similarly, Eclipse 3.4’s Equinox p2 includes 4 tools for provisioning:

1. an end-user Update UI, for easily updating your Eclipse folder,
2. an Admin UI (or Agent), for creating shared or reusable install folders,
3. an Installer, for doing shared product installs, and
4. a Director, for headless (no GUI) installs/updates.

Do we need all 4? Try them and decide for yourself. Like with Debian/Ubuntu installers, there’s bound to be some overlap. But each serves a purpose by itself, and does so with as little installation overhead as possible.

Could things be merged? Perhaps, if p2 wanted to follow the hierarchical model of Synaptic building on apt-get, and Add/Remove Applications… & Update Manager building on Synaptic. There are certainly places to simplify the UI experience. What would you do?

Voice your opinion here, or in the mailing list. Better yet, write a patch and submit it!

A number of people have remarked to me that there’s a very important feature missing in Eclipse 3.4 Ganymede’s p2 Update UI, which they consider a regression from Eclipse 3.3 Europa’s Update Manager UI.

Specifically, the feature — introduced in Eclipse 3.1? — that allows users to choose where to install a new feature, rather than the default eclipse/features/ and eclipse/plugins/ folders.

The simplest hack available to get this option back is to use the Update UI to install what you want into Eclipse, then go move all the newly created files in eclipse/features/ and eclipse/plugins/ into a new folder. You’ll know which ones are new (and therefore safe to move) because they’ll all share the current timestamp, whereas the base Eclipse install will have its earlier timestamp. Kludgy, perhaps, but it works.

But, you say, surely there’s a better way? Yes, there is. It’s the p2 director.

Here are two ways to install a feature using the p2 director: one installing directly into the dropins folder, and one installing into any folder on your drive, then LINKED from dropins using a .link file. Yes, .link files still work.

• Install into eclipse/dropins/
• Install into eclipse/dropins/ with .link file

Should you want to script the installation of a feature into the eclipse/ root folder (ie., for building a product, perhaps?) you can use this third script to reuse the existing SDKProfile. The added benefit is that only the single feature you chose to install shows up in the Help > Software Updates... Installed Software dialog, rather than ALL the subfeatures required or contained by that feature. It’s a little cleaner, but you’re back to the “everything in eclipse/plugins/ and eclipse/features/” scenario — though for a product, or “all-in-one” bundle, like the Standalone BPMN Modeler that’s probably better than spread around the disc.

• Install into eclipse/ using SDKProfile
  (the p2 Update UI’s default behaviour)

Got a few minutes to try this? Then why not contribute your own example (eg., for Windows or Mac), for your project?

Like Evil Dead 2, this “remake” of 2006’s Build Workshop was far more groovy than the first, in terms of special effects producing concrete results. I look forward to the next one… perhaps in the fall, when the leaves are turning and I need to get out of Toronto again? I could see these workshops becoming a quarterly event, if nothing else to keep people talking about and actively working on this issue. Facetime is important, especially when we’re all otherwise swamped with Real Work ^tm.

While the last one produced ideas, plans, and documentation about best practices, it failed to materialize its one big requirement, which was a commmon build infrastructure, hosted at Eclipse. I’ve since created documentation ([1], [2]) for doing a DIY build server, which has been successfully implemented by at least two projects. But it’s still fairly labour intensive, and it’s tough to share.

This time around, we focused on something that’s been on my TODO list for about a year: running my build system on build.eclipse.org. We’d originally planned to produce a vserver image or vserver config script, but since there’s still ample work to be done to genericize my existing image to work outside Modeling, this seemed a shorter initial path to prototype. And the fact that we can’t distribute such an image (because of all the GPL code in a Linux distro) was also a bit of a blow to the idea.

Bjorn and Denis, in trying to understand a little of the madness that is my system, have made me revisit all my original assumptions and requirements, to ensure they’re still valid, and that there’s not a better approach. I love having my assumptions challenged — it’s the only way to prove a system matches its demand, and that I’m not simply stagnating under a mantra of ‘because that’s how we’ve always done it’. (It’s sort of like my attempt to challenge the assumption that next year’s coordinated release should be called “Io”… but I digress.)

One thing we are still clinging to, for this first iteration at least, is that we’ll be building all-java, single-platform builds, for small projects & components who want a website with downloads, an update site, p2 metadata, jar signing, pack200 optimization… and little or no overhead in terms of infra setup. So, this solution will NOT address complex builds like the Platform, WTP, TPTP, or product builds. This is strictly (for now, anyway) designed to ease the burden on developers who don’t want to have to care about web/build infra. Of course none of this addresses the releng code that defines WHAT and HOW to build — it only enables a faster route to market for running and publishing builds. If you’re a project of the size of VE, PDT, or STP — or something smaller — this system’s for you.

Building anything more complex remains out of scope for now, and I admit freely that some of the reason for that is that Denis doesn’t do builds, Bjorn does small Technology Project builds, and I do Modeling builds — none of which motivates us to spend effort solving problems we don’t (yet) have. For 2 years my system didn’t do UI testing, because until UML2 Tools & GEF joined the party, there was no need. Now there are several projects w/ UI testing, so the system allows for that.

What is in scope is to explore the use of the Cruise Control interface to improve build scheduling and queuing, so that we can better manage disc and cpu usage. In time, the hope is that if a build queue gets too long, we’ll have statistics to back up the claim that we need more memory, cpu, or disc space, in order to better meet demand.

Clearly, I have a lot of work ahead of me, but today showed that both Bjorn and Denis are willing help out here. (That’s not meant perjoratively — only that we all have other time constraints pressing on us, but that we’ve collectively agreed to set aside cycles to focus on this.)

Here are the five pieces that must come together to make a build system work:

1. properly defined features and plugins — the responsibility of the component lead
2. a .releng project (or perhaps a Buckminster project?) to define what to build, what order to build, when to test, and HOW to package — shared responsibility between component lead and the release engineer, if your project is large enough to have one. Note that for these first two, my Summer of Code student is exploring the use of JET to produce wizards to guide this process and make it a little more friendly and less RTFM’ish.
3. UI to run builds on demand
4. UI to validate builds (JUnit results summary, links to build metadata like logs and config files). This could be views in Eclipse, but because publication involves putting bits on a website, it’s currently handled predominantly as PHP (with some Ant and bash scripts)
5. UI to publish acceptable builds to Eclipse.org & generate other build output (eg., an update site, Ganymede site contribution file, etc.); this could be merged into the build itself, but I split it because IMHO not all builds need to be published, and generating all the extra meta isn’t required when all you want to do is test the user’s install experience with your project. But of course this assumption can be challenged…

Then, in terms of automation (and places we can improve), there’s:

1. feeding the latest dependencies to the system so that when a new Platform (or EMF, or GEF, etc.) is available, the ad hoc and automated builds can simply use that new dependency. RSS feeds of course come to mind here, which though I was a big proponent of, haven’t really done much with (insert age-old “time constraints” excuse here)
2. scheduled builds are great, and can be set to run only if CVS has changed, but continuous building might be handy too. However, it’s important to consider how often to check CVS for changes [frequency], what’s considered a complete change (vs. part of a series of commits) [threshhold], and where to check [all the sources? or just the mapfiles]. Build too often & you’re wasting others’ cpu time. Not a big deal when there’s only 3 builds on build.eclipse.org, but if all 20+ Modeling builds move there… sharing and coordination suddenly becomes very important. And if your project consists of less than, say, 5 committers… do you really need continuous builds?
3. automated cleanup of old dependencies so the UI stays clean and disc usage is kept reasonable

And then, of course, there’s room to improve integration.

1. supporting Subversion sources
2. supporting Maven-based dependencies
3. converting bash scripts that “do work” to Ant scripts w/ custom tasks; submitting these back to PDE build or releng.basebuilder for reuse
4. converting bash scripts that “do calculations” to PHP-based web apis, so they can be called by web, shell, ant, or java
5. porting configuration parameters to the Portal

At the end of the day, we had:

1. evaluated Maven, Buckminster, PDE Build, basebuilder.releng, and the stuff I’ve done to simplify the PDE/basebuilder experience
2. successfully run the GEF build on build.eclipse.org (with some UI problems to be fixed, and at least one failing JUnit test)
3. implemented code to extract build parameters from the Portal instead of from static php config files; testing and iteration TBD (probably more things to add/remove/simplify)
4. created a way for the genie daemon to run builds launched from the web or crontab (but jar signing fails as we longer need to push bits to build.eclipse.org)
5. dumped a lot of the “common modeling build” code into a new Dash project, org.eclipse.dash.commonbuilder, which will house common web UI, server config scripts/properties (eg., paths for JDKs & build folders), and build/promote scripts
6. begun scripting the process of bootstrapping (or updating from CVS) this common builder, so it’s reproducable and hands-off; verification TBD

If that doesn’t sound like an exhausting enough day, have a look at this.

Note that this is not a project plan, and until one is drafted, nothing is set in stone. More people willing to help will of course allow more things to get implemented. So, does this project interest you? Are you willing to contribute time and effort kicking its tires by porting your build to this system, in order to make it better for all?

Calling all Eclipse Release Engineers!

If you plan to be in Ottawa next week for the Ganymede festivities or for the Demo Camp on June 26, why not drop by the Eclipse Foundation offices for Build Workshop 2: Build Harder, and help us build a “buildserver in a box” prototype?

Sign up now!

One simple scenario for all you “I like to just unzip stuff” people out there, who don’t want/need p2’s ability to resolve dependencies a la Synaptic / apt-get / yum. If you find the new p2 UI is overkill for your needs, you can still kick it old-school:

1. Download the latest Eclipse SDK or EPP bundle
2. Download your add-on SDKs, eg., EMF, Mylyn
3. Unpack Eclipse into your folder of choice, eg., ~/eclipse/eclipse34/
4. Unpack your add-ons into the dropins folder, eg. ~/eclipse/eclipse34/dropins/emf/, ~/eclipse/eclipse34/dropins/mylyn/
5. Start up Eclipse. p2 will discover anything you’ve added in the dropins folder, and load them up. To remove extensions, just remove them from the dropins folder. That’s it!

You can also combine zips with update sites for a more complex install, like for installing PDT 1.1 from a Nightly zip + the Ganymede update site.

Of course, if you have issues w/ the new p2 UI, report it! Over 600 squashed bugs this year alone…

Over the last couple of weeks, I’ve been rebuilding the way the Modeling Project’s update sites work. There are a number of reasons for this:

1. To provide a Milestone (S) update site, separate from Interim builds (I & M), in accordance with what GMF & GEF used to provide before they adopted my build system.
2. To reduce the disk footprint that has over the years gone from reasonable to insane, and make it self-cleaning.
3. To provide site digests for improved performance.

The upshot is that I now have a system that incrementally adds new builds’ jars to the site, cleans out any obsolete jars (ie., if replaced by newer version) and regenerates both the site.xml and digest.zip for the site. Instead of 7.9G, the 7×3 sites (EMF, EMFT, MDT, M2T, M2M, GEF, GMF) now occupy 1.3G.

How does it work? Read on. Don’t care? Skip here.

It all starts with either:

• an SDK zip (or a few zips, if you produce more than one like GMF),
• a “Master” zip (the result of collecting all your build’s features and plugins into a single zip, then sending it to build.eclipse.org for signing), or
• an Update zip (much like the SDK, but including a site.xml file and with the jars in plugins/ and features/ instead of in eclipse/plugins/ and eclipse/features/).

From there, an individual “sitelet” (category.xml) is created for those jars. Unjarred features/plugins are jarred. The sitelet - a single update site category - is combined with other local sitelets & their associated jars into one composite update site. A digest is produced.

Rather than just pushing that whole site from the build server to the remote public server, I use a few tricks to avoid having to upload the same stuff over and over.

1. If the new site, which I’ll call the n^th release, or rN, is identical to the zeroth (current) release, r0, then the process stops right there, as there’s no point publishing the same bits again.
2. if rN != r0, we cycle the releases so we only ever have three of the same build’s branch (2.1.0) and type (S). r2 is deleted, r1 becomes r2, r0 becomes r1, and rN becomes r0.
3. Depending on the type of site, we will then aggregate 1, 2, or 3 of these individual sites. For R, we only need the latest; for S we want the last two milestones; for the others, we keep 3.
4. This aggregate site is used to produce a “clean” list of all the jars that SHOULD be present for the current crop of builds.
5. A zip is produced containing the new plugin & feature jars, all the current individual category.xml files, a couple of shell scripts (buildUpdateSiteXML.sh, buildUpdateSiteDigest.sh) & the jar list. After pushing this zip to build.eclipse.org, the zip is unpacked and its contents combined with the current update site of the same type.

The new site’s contents are collected to produce a “dirty” list of jars. This list is compared to the “clean” list, and anything no longer needed is purged from the site.

The category.xml files are combined into a single site.xml, and from that the digest.zip is created.

If you’ve been installing any of the Modeling projects from update sites in the past, please try out the new version and compare its performance with your old experience. Feedback to bug 212203. See also the following p2-related issues. There’s not a lot of time left in this cycle, so vote for your faves!

• 200380: [prov] [solver] Improve detection of incompatible solution
• 223655: Cannot install from an update site that does not use site.xml
• 228145: p2 is inventing invalid jarname suffixes [SOLVED]
• 228310: Installing a newer version of a pre-installed feature should proceed as an update
• 229679: p2 does not install dependencies properly

Test early, test often!

If you run more than one automated JUnit test suite during your build, it can at times be difficult to ascertain why a suite failed. You can of course run the tests locally, but if your local environment is not identical to the build/test environment (eg., your build server is virtual and its X server is actually just a framebuffer), you may not see the same problem, especially with UI tests.

To capture the console logs for each independent test suite (.metadata/.log from the test’s workspace), you can edit the cleanup target of each test plugin’s test.xml. Make sure that ${classname} is defined, too.

<target name="cleanup">
        <mkdir dir="${results}/consolelogs"/>
        <copy failonerror="false"
             file="${eclipse-home}/results/${classname}.txt"
             tofile="${results}/consolelogs/${classname}_${platform}.metadata.log.txt" />
</target>

Now, after your tests run, you’ll get one more log file (for each test.xml you change) in your build folder’s testresults/consolelogs/ folder.

For other PDE Build testing tips, see Modeling Project Testing Problem FAQ.

In EMFland, we have some crazy-old internal FVT, BVT, and SVT tests that we run as a followup to our public JUnits. These are rather complex, involving codegen, hooking into PDE in less-than-standard ways, and doing all sorts of wierd modeling test work.

Unfortunately, they break a few times a year, and this time it took the better part of a week to sort out why. Ultimately, it comes down to the fact that p2 wants to do things smarter than its predecessor, and so some of the old hacks don’t work anymore.

The lessons learned here are:

1. If you still haven’t migrated from plugin.xml to MANIFEST.MF, you should. PDE UI provides a super-simple way to do this — just open the plugin.xml in the manifest editor, and on the Overview tab, select ‘create an OSGi bundle manifest’. (bug 224234)
2. If you must hook into <eclipse.buildScript> by hand, use the pluginPath attribute to help resolve your plugins. For example, use: ${eclipseDir}:${eclipseDir}/plugins:${dropinsDir}/eclipse:${dropinsDir}/eclipse/plugins. Might be overkill, but it works. (bug 224234#c3)
3. Ensure that your headless eclipse install, which is compiling and running your tests, includes both plugins AND any related features. The features might not actually be required, but why not include them so that you’re more closely synthesizing the user’s runtime? (bug 224195#c11)
4. If you encounter a situation where it seems your bundles aren’t loading and therefore code’s not compiling & tests are failing, you can always play with the OSGi console to verify you’re not going crazy. (Console)
5. Use eclipse/dropins/ instead of just eclipse/ for all your runtime and test plugins. (Migration)
6. Never attempt to “clean” eclipse by hand by deleting most of the stuff in eclipse/configuration/. Instead, back up your dropins/ folder, use a minty-fresh eclipse w/ a complete configuration/ folder, and copy anything else you need back into eclipse/dropins/. (bug 224195#c11)

Last year, I blogged about how to manage your plugins and features with link files and extension locations. Suggested by Euxx, I thought I’d do a quick side-by-side comparison of what p2 provides vs. the old ways. As you’ll see, p2 supports all the old ways, plus a number of new ones. Kudos to the p2 team!

• Equinox p2 Migration Guide