[buckminster-dev] Introduction and Buckminster walkthrough

[Kenneth Ölwing <kenneth@xxxxxxxxx>]

Everyone,

Presumably this is the first post to the buckminster-dev mailing list. I 
welcome
you all who have signed up so far, and hope that you will have patience 
while we
work towards a first publically available prototype.

In response to a separate discussion originating on the eclipse.tools.jdt
newsgroup ('Public versus internal interfaces'), we thought it prudent to 
start
off with some description of what Buckminster intends/does. So, please find 
a
longish tale below!

Regards,

ken1

(Kenneth Olwing)

==========================

A walkthrough of Buckminster (somewhat simplistically):

Buckminster is basically about recognizing that software are commonly
constructed out of many separate 'components'. We feel a perennial problem 
is to
manage the common confusion of 'how to know what versions works together'.
Ideally, we envision that a project lead only need to say to a new hire to 
the
dev dept: 'you should work on the com.somedomain.product project', and all 
the
dev has to do is enter 'com.somedomain.product' in their Eclipse and a 
suitable,
working workspace will result, even if the product is built up of
dozens/hundreds/lots of components. A 'no confusion, quickly get up to 
speed'
scenario.

The primary focus of Buckminster for the moment is to help the user to 
populate
a workspace with a suitable 'configuration' of such components (i.e. which
components, what version of said components - but also selecting different
representations of the component based on user intentions/needs/capabilities
etc). A somewhat crude analogy might be to support development of 'things' 
(e.g.
typically Eclipse plugins but also other component types) as conveniently as
Eclipse manages plugins/features in very dynamic manner. Note that 
Buckminster
not only aims at Java development, but other types as well.

There is generally no problem if you work with a monolithic codebase; just 
check
out a single toplevel directory (which is typically trivial in most CM 
systems),
and off you go. This essentially applies even if the codebase is well
modularized/componentized - you still check out 'everything' every time.
This can have detrimental effects though: one problem can be that the 
codebase
gets so large that doing a clean build takes 'forever' (and where even an
incremental build can take a long time - probably most visible in classic
makefile scenarios). Another problem can be in the simple usecase that the
software (and development team) is roughly divided in 'framework' vs
'application' levels. Framework developers (producers) sometimes tend to 
have a
wish to 'move on'. This can lead to changing interfaces which directly 
impacts
the app developers (consumers) since their stuff no longer works. In a 
worst-
case scenario the app devs have to play catch-up all the time (we call this 
the
consumer/producer problem).

The above problems definitively have their solutions. For example, a common
approach is to build the lower levels into libraries/jars, and check them in 
for
use by the higher levels. But this also carries with it management issues of
their own. There can also be difficulties with succinctly describing the
configuration. And what happens when some component owners want to do
exploratory work that shouldn't impact app devs at all at this stage (this 
is
particularly prevalent at the end of a release cycle when the framework is
basically frozen)? The consumer/producer issue is about letting the consumer
choose the timing of seeing change rather than having it foisted on them at
inconvenient times.

In any case, expand the monolithic case into a situation where components 
are
truly stored in 'different' locations - this can be as 'simple' as having a
single CVS repository with hundreds of top-level modules where valid
configurations each contains only subsets of them all. Thus, it's 
impractical to
check out *all* modules just to work with a certain subset. But also 
consider
that a number of components going into a certain product may also come from
various open source projects. For best CM tracking you might want to get 
those
from their real repositories (which may include other CM systems such as
Subversion, Perforce...or simply just FTP sites that allows downloading of 
fresh
releases).

The plot thickens. In larger efforts it quickly becomes impossible for each
individual to know where all these things come from, especially given 
transitive
dependencies - after all, an app dev really only cares about the 'framework
component', (s)he's not really interested in the dozens of components that 
make
up the 'frameworx component'. To top it all off, all those components come 
in
many versions...which one is correct?

Buckminster wants to help with this type of scenario - each component 
declares
its dependencies and Buckminster helps extract and/or bind them together in 
a
workspace. It will follow dependencies and try to make sure it garners a
workspace that is 'correct' (i.e. it's the same for all involved developers,
avoiding the 'it works on my machine' syndrome).
To do this dependencies can be specified with not just a component name. It 
may
also involve a version selector. In the simplest case there is none, and a
typical mapping would then be to get the HEAD version. This obviously 
devolves
to something that will not be consistent over time, but may be appropriate 
in
certain stages of development. To be more consistent, the dependency could 
state
'1.2.3' which (assuming the CM system is trusted) should always produce the 
same
thing. This is not unlike how Eclipse plugins can be matched at
deployment/runtime.

A concept of a 'resolver' is inserted 'between' abstract component names and
version selectors. It makes use of a 'resolver map' (xml) to map component 
names
onto repository providers/locations (typically several possibilities, 
including
already present in the workspace, present as a plugin in the running Eclipse
instance, or something else).
A simple reason for this is where a given repository, say a CVS repo, should 
be
accessed differently by different users - for user A, the CVSROOT should be 
just
a 'local' reference, while user B needs to access it using a 'pserver'
reference. In resolver parlance, they belong to different 'sites', which 
helps
abstract out differences between them.

There are other more advanced reasons; early on it was stated that 
Buckminster
would aid in selecting different component representations based on user
intentions. This is where match/source/mutability rules come in. Imagine 
that
the developers of 'the framework component' have specified that component 
FOO,
version 1.2.3 is a dependency. Normally, the app dev 'framework' doesn't
care/know about this. By default, they should get a representation of 
FOO-1.2.3
that best matches their needs. In their case, the best match is (typically) 
a
prebuilt library, no need to have source, and no need for it to be mutable. 
The
resolver will start looking and try to select the representation with the 
best
fit. If it indeed finds such a representation on a convenient FTP server, it
will be selected as the representation to be downloaded and bound to the
workspace/component. But it can also happen that no such lib is prepared
yet...so they will get the only possible representation - the tagged source 
from
the repository.

OTOH, the framework dev knows FOO well and may even need to have the source
available for changing - so (s)he needs the flexibility to essentially say 
'give
me the work branch 1.2.x of FOO, with source and mutable'.
In such cases, the resolver will probably find that, yes, a representation 
of
FOO is available on the FTP ...but while it has source, it is immutable and 
thus
not a good fit for the query. So it goes on through the resolution and
eventually comes back to the last possibility: the actual repository source
representation.

It is here that the resolver needs to be able to peek into possible
representations and actually understand the underlying repository in order 
to
map version selectors properly. It's also to figure out what the component
contains so it can use the match rules correctly. Finally, it's important to
extract the dependency information so that a user can make an informed 
choice
(interactively through a wizard) - e.g. 'if you choose v1 of component A, 
you
will need B and C but if you choose v2 you will also need D'.

This has been a whirlwind tour about Buckminster - no doubt I've forgotten 
some
important points, and likely raised more questions than answered any...:-). 
Please
don't hesitate to ask about the murky areas.

There are other possible focus areas for Buckminster, such as having a 
framework
that can provide help with process issues such as work cycle support; e.g. 
based
on a given set a of known good configuration with fixed component baselines, 
set
up work branches for a few of them, help sandbox it all, prompt to 
selectively
rebase and then subsequently help deliver changes back to involved CM 
systems in
an as atomic operation as possible. But this is for
later...