[equinox-dev] Bundle directories

[Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>]

As part of the Equinox transition we
want to put together a directory of bundles that people can get and use.
 It turns out that many of the Eclipse bundles, upon closer inspection
and after a bit of reorganization in the lower reaches of the Eclipse runtime,
are really quite independent of Eclipse and should be able to run in many
different situations.  Things like the extension registry, jobs, preferences,
help, swt, ... are all generally useful.  To help promote the use
of OSGi (and Eclipse of course :-) we want to ensure that it is easy for
people to discover and use these bundles.

The obvious approach is to enhance the
update technology to make it easier to manage individual plugins and to
separate the payload from the metadata.  Add to that some additional
PDE function for adding plugins from the update site to the target and
it starts getting pretty attractive.  

As part of the due diligence for this
we looked around and found a couple of related technologies that may be
interesting to explore instead or in addition.  Maven 2 repos (and
Wagon) and the Oscar Bundle Repository (OBR).  Below is a capsule
summary of the technlogy and some of the issues.

Coincidentally it turns out that OSGi
itself is looking to setup a directory on osgi.org.  This makes perfect
sense.  Persumably we should try to coordinate and use similar/compatible
technology, use the same directory, ....

There are two main use scenarios.  

Populating a running instance
=============
The user is running an OSGi framework
and invokes some operation (e.g., browses a repo) that triggers the download
and installation of a bundle.  Ideally all of the prerequisite bundles
are also downloaded and installed.  This is very interesting.  However,
forcing people to be connected to install is problematic.  Connectivity,
firewalls, etc all get in the way.  There needs to be a way of simply
acquiring the bundles and installing them later.  Perhaps multiple
times.

Aquisition of function
=============
Here the user is looking to simply discover
and acquire desired function.  They want to browse available bundles,
check licenses, and then aquire the bundles (and optionally the related
source) and perhaps the prerequiste bundles as well.  The downloaded
bundles can then be installed, added to targets, traded with friends, ...

In both cases the user should be able
to talk about groups of bundles (this scales better and allows people to
capture logical groupings of otherwise unrelated bundles) and MUST be able
to preview and filter by license.  In this day and age this is critical.
 The IBM Eclipse team members, for example, are not allowed to download
anything without knowing the license first.  All it takes is a few
bytes of GPL to ruin your day.

In any event, it seems that the ideal
is for some combination of OBR and Update capabilities with the option
to back onto a Maven repo.  Why not do it all :-)

Jeff



Maven 2 repos:
=============
- The repo itself is just a particular
disk layout where paths are group-id/artifact/version/<stuff>. so
for example, 
        eclipse/org.eclipse.osgi/3.1.0/org.eclipse.osgi_3.1.0.jar
- No surprises.  Quite simple.
- It is not listable and there is no
index
- It follows a strict containment model
so the same content in multiple groups would be duplicated
- "Wagon" appears to be some
technlogy/API for accessing the repo locally and remotely but I'll be darned
if I can find any doc or other info on it.  There isn't even a download
or indication of what to check out of the repo.  I'm not sure of the
status of this work and thus how you would access the repo from Java code
is unclear.  Perhaps it is only intended to be accessed from Maven
during a build?
- There appear to be some number of
people setting up Maven repos and if one had a maven build going on, you
could point it at the bundle directory (assuming it was Maven format).
 Unclear if you can run a build with multiple remote repos and search
them...
- There is no notion of searching, grouping
or categorization.  Additional organizational structure would be needed
to enable discovery
- There does not appear to be a standard
way for repo entries to expose a license either to humans or as search
criteria


OBR
====
- I have not had a chance to try this
myself.  I've just read the doc, talked to a couple people and guessed
the rest :-)
- Repo is just disk, remote or local.
- Disk structure unknown but the general
architecture is one where metadata is added to the repo but content is
hosted elsewhere.  so in effect you don't care
- Eclipes Plugins exist or populating
an OBR's metadata. I have not tried this but heard about it from Mikael
Desertot.  He is talking about it at OSGi World Congress in Oct.  I
hope to find out more before then.
- OBR client talks to the repo (protocol
unknown) and fetches all available metadata.  User selects some bundles
to install.  OBR figures out what other bundles are needed and downloads/installs
them into the current instance
- Unclear if one can just download the
bundles and stash them for later/alternate use (e.g., as part of a target)
- Unclear if the resolution/picking
accomodates R4 things like Require-Bundle
- So far I have not seen a standard
way of exposing a license to users or filtering download/install candidates
based on license type
- There is a rumor that ORB supports
a characteristic matching facility that sounds promising in terms of selection
of bundles
- the client already appears to support
some level of search over the metadata
- downside is that it downloads all
the metadata.  This may not scale as the repo grows to hold many versions
of many bundles.
- no facilities for managing groups
of bundles.  Sometimes individual bundle dependencies are not enough.
 For example some bundles connect together via loose couplings like
extensions.
- client (command line) appears to be
reasonably functional

Update (for completeness)
=======================
- Disk layout based repo.
- http access
- site list/index in site.xml.  Metadata
embedded in artifacts (features, plugins).
- standard practice todate has not filtered
up the dependency metadata into features so it can be hard to get the complete
picture without actually downloading the plugins. This is not a technological
problem but rather a convention/practice that would have be overcome
- supports license exposure
- supports groupings of bundles (e.g.,
features)
- would need utility helpers to ease
the addition of individula (or groups) of bundles by generating a feature
for them.  The update site builder technology can be leveraged here.
- No "smarts" on the client
in that it only downloads what you told it.  There is some very modest
search capability that helps find required features.
- rudimentary command line client capabilities
but non-trivial API in the background.
- Categorization on the site.xml  Only
one level?
- Update site is expected to physically
contain all bundles and features.  Curently no facilities for serving
content separate from metadata. One tractible change would be to have the
plugin jars on a separate site but the feature.jars contain the interesting
metadata so either we have to extract the feature.xml or the whole feature
has to stay with the site/directory.