Re: [cross-project-issues-dev] Builds and repository specificity

[David M Williams <david_williams@xxxxxxxxxx>]

By chance, I was lucky enough to discuss
this issue with some smart people on the Planning Council call this morning
and helped clarify to me how it is a bad idea to provide only "one
big repository"  (which below I mentioned was "ok")
or even use a generic general purpose one in your own build. Guess we should
try and formally document this better, I know I've seen pieces here and
there ... maybe we'll capture the result of this discussion. 

We, as providers, need to provide repositories
for several audiences or types of users. I'll give some mostly hypothetical
examples below to  illustrate, and perhaps this list can fill in details
or issues I omit. 

End Users: 

The "update site" URL in a
feature.xml file should be specific to one "release stream" or
release train. For example, in webtools we use 
http://download.eclipse.org/webtools/repository/helios
This is because it would be bad (confusing
at least, if not outright wrong) for our helios users to  be presented
with indigo content. Either now, or, even when indigo formally releases
next June. 
But, its goodness if they can automatically
get updates to things they have previously installed from our Helios site.


As a practical matter, it is often easiest
to provide these release URLs as a composite repository, where the "real"
repositories, are more specific, but those more specific repositories are
"an implementation detail" and could change, should not be "published",
counted on for builds, etc. For example, in webtools, our release repository
is a composite that conceptually combines several other repositories: 
http://download.eclipse.org/webtools/repository/helios/sr0
http://download.eclipse.org/webtools/repository/helios/sr1
but normally no one should have to use
those specific repositories directly. 

Release engineering.

Here, repositories must be provided
to be specific to one build. This is so others builds can be reproducible.
Some teams provide these as zipped-up p2 repositories, sometimes as a URL
specific to exactly one location. For example, 
a build repository from Orbit is  
http://download.eclipse.org/tools/orbit/downloads/drops/R20100519200754/repository
Sometimes teams might be able to count
on a "release repository" having only one version, so they can
count on having reproducible builds from it, such as 
http://download.eclipse.org/webtools/repository/helios/sr1
but that is pretty variable project
to project, and depends on a project's policies, agreements, etc. 
So, while you are building your own
stuff ... you should be using one exact repository provided by others so
your builds will be reproducible. 

I know its tempting to use some "general
purpose" URL to "get the latest" of whatever someone is
providing ... p2 makes that so easy! :) ... but ... especially for "release
builds" (which you need to have reproducible) you need to use a specific
repository or know how to always get exactly the same bundle versions you
got before. 


So, my view is, we should "constrain"
end users some, to help them get good installs, and keep good installs.
 (this is partially since feature versions are complex, wildly variable,
and there's no way for them to tell which one is intended for which stream).


And, more relevant to this discussion,
 builds should be very constrained, especially release builds, so
you get exactly the same thing when you rebuild. Hopefully everyone provides
these build specific repositories, but not sure we've ever spelled this
out before. 

This might be get more complicated by
using various tools or APIs, such as for b3 aggregator, or p2 mirror, etc.,
you _can_ specify an exact versions to copy/mirror ... but that only makes
sense if the provider provides the exact version, as we do tor the helios
aggregation builds. There's no way a consumer would know exactly which
version to get ... right? And bundle manifest constraints are not enough
(not nearly enough) since normally you want those "wide" to _run_
with many versions, but for a build, to be reproducible you need to get
a specific build (or set of bundles with exact versions) as your prereq.


Does this make sense? Other views? Have
I overlooked anything? Stated anything incorrectly? Am I stating the obvious?
Or something that's controversial? 

Thanks, 








From:      
 David M Williams/Raleigh/IBM@IBMUS
To:      
 cross-project-issues-dev@xxxxxxxxxxx
Cc:      
 Frédéric Madiot <fmadiot@xxxxxxxxxxxxxxxx>
Date:      
 09/15/2010 10:45 AM
Subject:    
   Re: [cross-project-issues-dev]
Build helios failed
Sent by:    
   cross-project-issues-dev-bounces@xxxxxxxxxxx

---

You call that simple? :)  (It probably
is, and I'm just being dense in "picturing" it). 

What's the next step?  How can we correct this in the quickest way?
(today is RC4 +2, after all)? 

While there's no "standard" way to do it (so far) some projects
do keep a "repository for everything" at one URL, but under the
covers that is just a composite that "points to" more specific
repositories, for specific releases, and clients could use the more specific
URLs if they needed to. Is that the case here? It does sound like some
separation is required for the different levels, especially for yearly
releases. 

I'm moving discussion to "cross project" since you imply there's
some indigo/helios bleeding-trough for other teams too ... and we need
to avoid that! 

Also, I'm confused about the comments about "update site" vs.
"p2 repository". It is required to provide a p2 repository. Maybe
I'm misunderstanding (lots of speed reading this morning). 
But I am aware of a bug in one of the common builders that was failing
to produce the p2 repository, but am assuming that _has_ to be fixed before
final releases/builds. 
See bug 324870. 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=324870
Not sure if that's related or not. 

Thanks, 





From:        Laurent
Goubet <laurent.goubet@xxxxxxx> 
To:        David
M Williams/Raleigh/IBM@IBMUS 
Cc:        Frédéric
Madiot <fmadiot@xxxxxxxxxxxxxxxx>, Hugo Brunelière <hugo.bruneliere@xxxxxxxx>,
Nicolas Bros <nbros.mia@xxxxxxxxx> 
Date:        09/15/2010
10:14 AM 
Subject:        Re:
Build helios failed

---

David,

The problem is quite simple actually :

- Acceleo milestones (RC1, RC2, RC3, RC4) are located on the aggregate
"m2t milestones" update site.
- Said aggregate "m2t milestones" update site contain all versions
of 
Acceleo : one for Ganymede, another targeted at Helios, and a last one
targeted at Indigo (M1)
- I contribute to the helios aggregator my RC4 milestone through this 
update site
- MoDisco depends on Acceleo without specifying a maximum version : it
then does not retrieve my contributed build (Helios RC4) but the very 
latest it can find on the update site : Indigo M1

And here is the issue : we get a MoDisco build for Helios that is built
against a dependency targetted at Indigo. What happens next :

- I build my Acceleo "final" release, it is no longer located
on the 
aggregate "m2t milestones" update site, but rather on the aggregate
"m2t 
releases" update site
- Said aggregate "m2t releases" site doesn't contain any version
targetted at indigo (M1 is not a release, and it's the only "Indigo"
build for Acceleo)
- I contribute to the helios aggregator my "final" build through
this 
new update site
- MoDisco was hoping to find an Indigo version of Acceleo, does not, and
the build fails.

I have no idea whether MoDisco is the only project that implicitely 
fetches a dependency that is _not_ aimed at the Helios release, but I am
pretty sure it isn't : if anyone else contribute an aggregate update 
site to the Helios aggregator, chances are one or more of the built 
plugins are built against a dependency that is in fact too recent (and
which API might have been broken since).

What's even worse is that Nicolas seem to have been able to install 
MoDisco from the Helios update site even though the version range for 
its Acceleo dependency was wrong, which would mean a version of Acceleo
that was meant for Indigo ended up on the Helios update site.

Laurent Goubet
Obeo

David M Williams wrote:
> I'll admit I don't quite understand who is doing what to whom ...
but, 
> hope you get it worked out soon!  But I'll make a few comments.
>
> a) It is normal for a repository to contain many levels of a bundles
> or features (that's one of design goals ... a "repository"
contains a 
> lot ... a big huge data base of code). But, this can get complicated,
> and requires good communication and perhaps some maintenance to make
> sure what is there is what is intended and "valid" (that
is all "fits 
> together").
>
> b) It is normal (nearly required) for bundle's to specify their 
> pre-reqs with a minimum and maximum range ... and, yes, we should
all 
> be following certain rules about what those versions mean.
>
> I doubt these brief comments will help that much (I'm probably missing
> this issue) .... but, hope if can focus discussion on how to get the
> problem solved.
>
> If there is a larger issue here that deserves broader discussion or
> agreements, perhaps a not to cross-project list would be best.
>
> Let me know if I can help,
>
> Thanks,
>
>
>
>
>
> From:        Laurent Goubet <laurent.goubet@xxxxxxx>
> To:        Nicolas Bros <nbros.mia@xxxxxxxxx>
> Cc:        David M Williams/Raleigh/IBM@IBMUS,
Frédéric Madiot 
> <fmadiot@xxxxxxxxxxxxxxxx>, Hugo Brunelière <hugo.bruneliere@xxxxxxxx>
> Date:        09/15/2010 09:08 AM
> Subject:        Re: Build helios failed
> ------------------------------------------------------------------------
>
>
>
> Nicolas,
>
> Unfortunately, the m2t update sites (we still use the old build system
> for now, at least until the "promote" from hudson  becomes
possible
> without too much hassle) aggregates the versions from europa, ganymede,
> helios, indigo ... indiscriminately. "milestones" include
our 0.8.1RC4
> (no longer the 0.8.0, and we had no 0.8.2), 3.0.1RC4 (Helios SR1),
and
> 3.1.0M1. The "releases" update site is the same : 0.8.1
final, 3.0.1
> final, and in June 3.1.0 final. I know Acceleo isn't the only one
to use
> one of the old aggregate update sites, and I am pretty sure the old
> "project" update sites (m2t, emf, m2m...) aren't the only
ones that
> aggregate multiple versions of the same plugin.
>
> If I understand correctly the requirement, we would need at least
2
> update sites for each simultaneous release :
>
> Acceleo - Ganymede - milestones
> Acceleo - Ganymede - releases
> Acceleo - Helios - milestones
> Acceleo - Helios - releases
> Acceleo - Indigo - milestones
> Acceleo - Indigo - releases
> ....
>
> and if we need "integration" update sites, that's one more
for each
> simultaneous release ... seems like a lot of work. We'll probably
cope
> with it when we finally switch to hudson builds, but I am pretty sure
> there are others in the same case as us for the SR1 release
> (emf/transaction, emf/validation, m2m/atl, gmp/gmf...), thus most
likely
> getting a lot of versions in Helios SR1 that weren't intended for
SR1
> ... or any Helios release for that matters.
>
> Specifying closed dependency intervals in the plugins doesn't seem
to be
> the right thing to do.
>
> Just my two cents.
>
> Laurent Goubet
> Obeo
>
> Nicolas Bros wrote:
> > I think I now understand what happened:
> > MoDisco plug-ins don't specify a dependency range for Acceleo.
So, p2
> > takes the higher version it finds when building MoDisco.
> > The version of MoDisco targeting Helios SR1 is built with the
same
> > update sites as those specified in the Helios build model.
> > So, since you specified the "milestones" update site,
MoDisco was
> > built using version 3.1.0 of Acceleo from this update site.
> > This dependency then becomes explicit in the MoDisco update site
built
> > with p2.
> > But when you switched from "milestones" to "releases"
in the Helios
> > build model, this version wasn't available anymore, causing the
> > aggregation failure.
> >
> > I believe the problem could have been avoided by either specifying
> > closed version dependency intervals in MoDisco plug-ins (which
> > requires a knowledge of Acceleo versioning rules), or making
sure that
> > the Acceleo update site specified in the Helios build model only
> > contains versions of Acceleo targeting Helios, at all times.
> > For example, for MoDisco we have 2 different update sites, one
> > specified in the Helios build model, and the other in the Indigo
build
> > model.
> >
> > On Wed, Sep 15, 2010 at 2:30 PM, Laurent Goubet
> > <laurent.goubet@xxxxxxx <mailto:laurent.goubet@xxxxxxx>>
wrote:
> >
> >     Nicolas,
> >
> >     As far as the Acceleo.b3aggrcon file is concerned,
here is the
> >     only change :
> >
> >     -----
> >      <repositories
> >     
> location="http://download.eclipse.org/modeling/m2t/updates/milestones/"
> >     description="M2T ACCELEO milestones">
> >       <features name="org.eclipse.acceleo.sdk.feature.group"
> >     versionRange="3.0.1.v201009150356">
> >     -----
> >     changed to
> >     -----
> >     <repositories
> >     
> location="http://download.eclipse.org/modeling/m2t/updates/releases/"
> >     description="M2T ACCELEO releases">
> >       <features name="org.eclipse.acceleo.sdk.feature.group"
> >     versionRange="3.0.1.v201009150438">
> >     -----
> >
> >     The two builds (3.0.1RC4 -on milestones- and 3.0.1
-on releases-)
> >     are exactly identical except for their qualifiers
... yet I
> >     believe the change that caused your build to fail
is the switching
> >     from the milestones update site to the releases
update site :
> >     3.1.0M1 (Acceleo version for Indigo M1) is present
on the m2t
> >     milestones update site, but no 3.1.0* release is
present on the
> >     releases update site (and there won't be any before
3.1.0 final,
> >     for Indigo, in June). David, Is it possible that
there was a
> >     "leak" that allowed MoDisco to fetch
a build from the m2t
> >     milestones (3.1.0M1) that wasn't the one I wanted
to contribute to
> >     Helios (3.0.1RC4)? That would mean we haven't got
any way to
> >     ensure that the Helios build is really ready for
consumers, as
> >     other could have the same problem as MoDisco without
being able to
> >     realize it : the only thing that allowed for us
to know that
> >     MoDisco was depending on the wrong version of Acceleo
was me
> >     switching to the m2t releases update site.
> >
> >     It would be a good idea to tell everyone to double
check that
> >     their "final" build really installs without
failure on an Helios
> >     platform (and that it installs without wanting
to fetch anything
> >     from the Indigo update site).
> >
> >     Laurent Goubet
> >     Obeo
> >
> >     Nicolas Bros wrote:
> >
> >         Thank you for the explanation.
> >         But there is something I don't understand
: we haven't changed
> >         the dependencies for SR1, and these
dependencies were resolved
> >         in Helios until now.
> >         So, since MoDisco was installable
with only the Helios release
> >         update site, does it mean that Acceleo
3.1.0 was available
> >         from the Helios update site?
> >         What exactly changed between the
aggregation builds #371 (Sep
> >         15, 2010 4:58:33 AM) and #372 (Sep
15, 2010 7:01:02 AM)?
> >
> >         On Wed, Sep 15, 2010 at 1:16 PM,
Laurent Goubet
> >         <laurent.goubet@xxxxxxx <mailto:laurent.goubet@xxxxxxx>
> >         <mailto:laurent.goubet@xxxxxxx
> >         <mailto:laurent.goubet@xxxxxxx>>>
wrote:
> >
> >            Hi,
> >
> >            Just saw that the Helios
aggregation build failed because of
> >            MoDisco with message
> >
> >            [exec] Missing requirement:
> >            org.eclipse.gmt.modisco.workflow.driver.acceleo
> >            0.8.1.v201009090831
requires 'bundle 
> org.eclipse.acceleo.ide.ui
> >            [3.1.0,4.0.0)' but it
could not be found
> >
> >            The change that triggered
the build was me contributing the
> >         final
> >            Acceleo SR1 b3aggrcon
file for the aggregation, but I
> >         believe this
> >            to be a failure from
MoDisco. I just wanted to point out that
> >            version 3.1.0 of Acceleo
(minimal requirement from the 
> failure
> >            above) is the version
that will be contributed to Indigo, the
> >            Acceleo version for
Helios being 3.0.1.
> >
> >            Regards,
> >
> >            Laurent Goubet
> >            Obeo
> >
> >
> >
> >
> >         --
> >         Nicolas Bros
> >         R&D
> >         tel: 06 75 09 19 88
> >         nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
> >         <mailto:nbros@xxxxxxxxxxxxxxxx
<mailto:nbros@xxxxxxxxxxxxxxxx>>
> >         nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
> >         <mailto:nbros.mia@xxxxxxxxx
<mailto:nbros.mia@xxxxxxxxx>>
> >
> >         Mia-Software, 410 clos de la Courtine
> >         93160 Noisy-le-Grand
> >         http://www.mia-software.com
<http://www.mia-software.com/>
> >         .: model driven agility :.
> >
> >
> >
> >
> >
> > --
> > Nicolas Bros
> > R&D
> > tel: 06 75 09 19 88
> > nbros@xxxxxxxxxxxxxxxx <mailto:nbros@xxxxxxxxxxxxxxxx>
> > nbros.mia@xxxxxxxxx <mailto:nbros.mia@xxxxxxxxx>
> > Mia-Software, 410 clos de la Courtine
> > 93160 Noisy-le-Grand
> > http://www.mia-software.com
<http://www.mia-software.com/>
> > .: model driven agility :.
>
> [attachment "laurent_goubet.vcf" deleted by David M Williams/Raleigh/IBM]

[attachment "laurent_goubet.vcf" deleted by David M Williams/Raleigh/IBM]
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev