[virgo-dev] Re: Running integration tests with maven

[Steve Powell <spowell@xxxxxxxxxx>]

(My apologies for a long post.)

In discussing the build mechanisms we currently use, and how they might be improved, I would like to be explicit
about the advantages and disadvantages of the present dependency control. At least IMO.

These might be viewed as potential requirements of inter-Git Repository dependency management.

[I shall use the terms GR for a Git Repository and GRuB for a Git Repository Build (a potential bug, perhaps?).]

The present build system has fixed version dependencies between GRuBs until they are explicitly updated (either
manually with update_dependency.rb, or by means of Ripplor, etc.).

The primary advantage of this is that a GR can be worked on (and pushed) and not affect the rest of the build tree.
Even though a GRuB is successful, it doesn't follow that the GRuBs that depend on it will succeed.

When we want to use the dependency on a successful GRuB in another GR, we can either make a manual update_dependency.rb
change (and push the GR), or, more commonly, perform a "Ripple" of builds.

If we make every GRuB automatically use the most recent (successful) GRuBs it depends on, we gain the advantage that
we do not have to perform the dependency updates (no more Ripplor).  However, we make the set of GRs more brittle,
subject to subtle breakage. This sort of breakage is infuriating when it happens in the same room. If the cause is
potentially half-way round the world in another time-zone then solving it will be harder.

When running a "ripple" of builds between GRuBs the complete (sub) tree is built locally before imposing the
changes on every one else (before they are pushed). This means that the impact of a GRuB can be detected
before the rest of the tree is broken.

The drawbacks seem to be that ripples are time-consuming, and can be invalidated due to parallel changes (this is
essentially the optimistic locking syndrome), which make them even longer. Furthermore, if a ripple detects a problem
caused by a GRuB dependency, the cure may involve other people and further delay.

Incidentally, the full power of a Ripple (or update_dependency) is rarely needed: often the Ripple actually just brings
everything 'up-to-date'. If that is all it did (essentially a manual step to perform the automatic dependency management
under discussion), then it would suffice.

Under the automatic system, it is possible for users to perform full tree builds locally before pushing their single changes
to a GR.  This would make a Ripple unnecessary. However, the overhead of this (about an hour-and-a-half minimum
elapsed time) is high enough to mean that people (including me) just won't do it.  Also, I will not do it when it is most
necessary: either when I think there is no impact, so "it can't hurt", or when I want to use the dependency in a later 'GR'
urgently and I 'cannot afford' to wait.  These are the times that chaos can ensue.

In Summary:
A system that retains the protection of a private, manual, dependency update--or at least meant that
updates to GRs that broke the tree were not adopted automatically--would be preferable to one that
took the latest of everything all the time.  Latest isn't necessarily greatest.

Regards,
Steve Powell
[W:+44-2380-111-528; M: +44-7815-838-558; H:+44-1962-775-598]

Cannelloni n. A narrow boat for one. [SPD]
Verisimilitude adj. Extremely evenly masticated. [SPD]
Decrepit v. To muck-out Royal stables. [SPD]