Re: Re: [platform-update-dev] Investigating some alternative update mana

["Alex Blewitt" <alex.blewitt@xxxxxxxxx>]

On 14/09/06, Dejan Glozic <dejan@xxxxxxxxxx> wrote:
> As somebody who was there through all the iterations of the Update Manager,
> let me assure you that we didn't like features as an idea ourselves.
> However, no matter how we tried, we could not shoehorn the distinct
> functionality that Mark describes below into a simple plug-in extension
> point. Of course, you can cram most of the feature.xml content into an
> extension, but all you are doing is overloading the plug-in concept with
> addition semantics not initially anticipated.

Not so. A bundle defines all the dependencies it needs; in fact,
that's the whole *point* of having OSGi bundles. There's even a
Requires-Bundle tag that the OSGi spec defines.

Consider:

org.eclipse.jdt.feature.bundle
META-INF\MANIFEST.MF
Requires-Bundle: org.eclipse.jdt.ui, org.eclipse.jdt,
org.eclipse.jdt.runtime ...

That's exactly in the scope of what an OSGi bundle is, and fits in
with the idea of what a feature contains (as a container of other
bundles). It may not contribute code, but it could contribute
resources (such as the license, copyright information etc.) which
could then be utilised by UpdateManager2 to give the appearance of a
feature.

The bootstrapping argument isn't valid; you need a VM installed
already and the base platform before you can do anything. The same is
also true of the current update manager; whilst you can download
features (and indirectly, plugins) you don't have the ability to
launch an update process without having some kind of update manager
already installed. In fact, the update manager doesn't need to worry
about what features are present; it just depends on bundles already
installed in the system.

(There is a separate discussion regarding the updating of the core
components, such as startup.jar, which -- whilst it needs addressing
-- I don't believe belongs in the features-are-bundles discussion).

> Essentially you will end up doing
> exactly what you are doing today (and the tooling would need to have
> specialized knowledge of the 'feature' extension point, which just moves
> the problem around but does not makes the code smaller).

By the time you download feature.xml, you are already in an OSGi
environment. You would have to deal with specialised knowledge of
either the 'feature' extension point or the feature.xml file. One is
just a file stored in a JAR, one is stored as a file on a directory.

> No matter how you slice it, you need to be able to provide a 'bill of sale'
> for a bunch of plug-ins/bundles you want to move over the wire and install
> somewhere.

Agreed, but two observations need to be made:

1) The bill-of-materials needs to have not only 'me' but 'my
dependencies' listed, or computable. Currently, features have this on
a I-depend-on-feature point of view. This is entirely within scope of
the proposal I suggested; a plugin has dependencies on other plugins,
which need to be satisfied internally or by downloading from other
places. For example, org.apache.xalan needs to depend on
org.apache.xerces. When you update xalan, you don't want to just
update xalan; you want to update xalan and its dependencies, too.
There's really no difference between that and updating JDT, which
depends on jdt.ui, jdt, jdt.runtime...

(You may have mistakenly assumed that by updating bundles, I was
referring to just a single bundle-at-a-time. That would not work, or
be sensible. You would have to compute the transitive list of
dependencies for any bundle(s) that were downloaded; the point raised
by Mark echoes that sentiment. I propose, that by updating the JDT
bundle, you are also in fact updating any of its dependents as well.
Ergo, updating a bundle can either be just a single bundle (in the
case of org.foobar.bundle) or a set of bundles
(org.eclipse.jdt.feature-bundle))

2) There is no reason this bill-of-materials cannot either be computed
from the bundle itself (i.e. from the Manifest) or supplied
externally. In fact, you'll note that I proposed having an Atom feed
to represent these bundles; you would have an Atom feed for
org.eclipse.jdt in just the same way that you would have an atom feed
for org.apache.xalan. The Atom feed is merely a wrapper around
whatever XML format you choose for representing it. Heck, you could
even include the body of the current feature.xml format into the Atom
feed. The thing is, what you want to get out at the end of the day is
the runtime collection of bundles. A feature is just a way of getting
you to update what's there, in much the same way that an update
site.xml is polled to find out what version(s) are available.

> This 'bill of sale' listing the bundles and possibly providing
> additional information needs to be somewhere outside of these bundles. If
> you put it in one of the bundles, you will need to somehow designate this
> bundle as 'special' so that the transporting software can peek into its
> plugin.xml and extract the shipping info.

Again, two points:

1) The bill-of-materials doesn't have to live outside the bundle.
However, it would make sense (for efficiency) if on the server, it was
able to query the bill-of-materials without having to download the
bundle itself. Thus, the idea of an Atom entry for that bundle. It can
be stored in the bundle, and then when extracted onto an update site,
all the atom entries can be expanded to be separate downloadable files
on the remote server for querying with compatibility etc. I would
expect that this information would contain a similar set of
information for the Manifest, especially with regards to dependencies
and versioning of those dependencies; but could contain other
information too. This has the effect of nominating each of these
bundles as their own 'featurette' with the atom entry being the
feature dependencies.

2) Those Atom entries can then be weaved into a Atom feed, to produce
a document similar to site.xml at present. In fact, because you now
have the information in infinitely more configurable ways, you can
have many such feeds; one might contain entries for all the JDT
bundles; another might just contain an XML parser; others could be for
source or help files (which, after all, would be listed as 'optional'
in the atom feature). Unlike now, when you download (say) the SDK, or
update it with the feature, when you get all the source and help
whether you like it or not (and without both source and help, you can
trim the install down from 120Mb to 50Mb).

> What you just did was
> re-introduce features by asking this bundle to 'play' the feature role.
> Once your product grows, you will realize you need to provide groups of
> bundles that provide a distinct function and can be grouped an reused in
> larger groups in many interesting ways. Pretty soon, you will check all the
> feature spec checkboxes.

Yes, but I'm saying that the implementation can be changed. Don't make
it so hard to have a 'feature'. Give each bundle its own -- separate
-- bill of materials (and dependencies). Let people update/download
aggregations of those bundles. But don't just let them download what
*you* think should be grouped together; let people download whatever
pick-and-mix combination of bundles that they want.

Atom feeds provide a timestamped, well understood payload mechanism
for transporting this data. They don't define what the data contains
inside. Heck, if you really wanted to take the easy way, you could
just dump the Manifest.MF into the atom entry, and let the client
figure it out (but there's better ways of doing it). You could in a
matter of days exchange the feature.xml for an atom feed doing exactly
the same job. An update site would then be an aggregation of atom
feeds; and the update manger would connect to many atom feeds.

What I am arguing for is getting rid of the features/ directory out of
an Eclipse install. It doesn't have any place in OSGi runtimes. It's
also the reason why Eclipse extensions are so poor; you *have* to have
a directory called 'eclipse', and it *has* to have a subdirectory
called 'features' and 'plugins'. Why? Because you store features as
top-level elements. Move whatever content you want from features (e.g.
copyright notices, overall bill-of-materials -- and remember, it can
be an atom feed stored inside it for trivial extraction onto a server)
into a bundle for containment/downloading/reporting process (e.g. Have
I got the JDT feature installed? What version is it? etc.)

(It should be noted, in passing, that the OBR -- Oscar Bundle
Repository -- does just this (oscar-osgi.sourceforge.net) and that the
OSGi alliance has a bundle repository as well (www.osgi.org) that uses
just bundles and metadata associated with them to calculate
dependencies. Saying it can't be done without features flies in the
face of others already doing this.)

Get rid of features, and soon you just need eclipse/plugins. The
update URLs for the atom feeds are stored as plugin state data of the
update manager bundle; and, let's not forget that each Atom entry and
Atom feed has a URL that uniquely identifies where they came from, so
when you need to update bundles you just query all the atom feeds to
find out if anything is new.

If you just need eclipse/plugins, you can be more flexible about its
storage location. So, it could be called just plugins. Good if you
want to build a RCP that doesn't want to explicitly advertise its
eclipse heritage.

If you can just call it plugins, why not have several of them? Now, an
extension location doesn't need any of that .eclipseextension crap in
a folder called eclipse/plugins before it will work. You could have a
parameter -- say, eclipse.bundle.path=${eclipse.home}/plugins -- that
says where to find them from. Or, why not have multiple? Why not have
eclipse.bundle.path=${eclipse.home}/plugins;/usr/local/jdt;/usr/local/pde;/usr/local/gef
? Download and install a suite of bundles into each place, then start
eclipse with the combination of directories (of bundles) that you
want. Heck, I'll be updating Maclipse so that you can drop whatever
bundle you want into ~/Library/Application Support/Maclipse/Plugins --
because that's where any self-respecting Mac user would expect to be
able to drop in bundles. (I can bodge this with symbolic links in
hidden directories; but the fact is that an extension location
mandates a file structure precisely because of how features are
stored.)

> You (Alex) do raise an important issue regarding small contributors who
> want to publish single plug-ins. For them, having to provide a feature
> seems like an overkill. For people who are shipping more complex products,
> features make a lot of sense and are very useful. If Update Manager was
> able to handle a plug-in by generating a transient feature on the fly, it
> would certainly help. However, I don't think that ditching features (or any
> grouping concept that the community may come up with) is going to help more
> ambitious Eclipse projects.

You are partitioning the two into separate areas. However, they're
just ends of the spectrum. On the one hand, you might have a single
bundle (only one bundle; no dependencies). My atom suggestion works
for that. In the middle, you might have a bundle that depends on five
others. My atom suggestion also works for that; the metadata from the
'parent' bundle would encode that dependency, and it should be
derivable programmatically without the user having to do anything. At
the far end of the scale, a JDT bundle that depends on all the
constituent bundles inside JDT also works for the atom suggestion.

The only difference in these is that the feed is bigger, or contains
more dependencies, than others. The fact that a bundle's atom entry
lists zero or one or many pre-requisites is irrelevant.

So, let's turn it on its head; given an atom feed (and constituent
atom entries); why *couldn't* you replicate
what-you-get-from-features-now? In other words, given your blanket
assertion:

> For people who are shipping more complex products,
> features make a lot of sense and are very useful

What is it that features do that an atom feed couldn't?

Alex.