[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [tycho-user] User Story - Component-oriented build system
|
staging.previous/ contains previous versions of the CI build in
staging/. Sometimes things disappear in transit (eg., when trying to
build an aggregate from a composite while one of the contained component
sites is being rebuilt/republished) so to work around that the hack
we're using is a reference to build #123 AND its previous version, #122.
This appears to help Tycho resolve things based on cached metadata even
if the contents have moved from staging to staging.previous... but it's
not a 100% foolproof workaround.
As to inter-job dependencies, we currently have jobs cascading in a set
order (as defined in the jobs) so that if Z depends on Y depends on X
and W, and there's a change in git for W's sources, it will kick a build
of Y and then Z when it's complete. In the past I didn't bother with
staggering the times of the source checks, but now they're all set up to
check sequentially so that in theory, changes will cascade down in the
correct order.
Again, it's not 100% foolproof because I can't predict when commits will
happen, so sometimes jobs DO fail simply because they build against old
upstream API which hasn't had a chance to respin (or conversely, because
they've spun against new upstream API which hasn't been accounted for in
the downstream project). In the first case, a respin usually fixes the
build failure. In the second case, the failure is a good thing because
it exposes the need for the downstream project to adapt to the upstream
API change.
There are some new Jenkins plugins available to help with watching job
interdependencies:
https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin
http://wiki.hudson-ci.org/display/HUDSON/Upstream+Downstream+Column+Plugin
http://blog.cloudbees.com/2012/01/dependency-graph-viewer-for-jenkins.html
And more complex ways to trigger downstream jobs:
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
Hope that helps!
Nick
On 01/15/2013 05:15 AM, Timo Rohrberg wrote:
Well, thank you very much Nick for this hint - it definitely looks like
a good approach which might help us establish our own build system.
However, after browsing your repositories, I do have two questions:
1) What are those staging.previous repositories for that are referred to
in all the composite repositories, such as
staging/jbosstools_4.1_trunk.component--birt/all? Are those some kind of
previous build results? But for what are they used?
2) How do you handle dependencies between the individual components when
building them separately from each other? Example: Component A depends
on component B. Both, component A and component B are changed, but now
component A depends on the new version of B. Those changes trigger a
separate build of component A and component B respectively. If the build
of component A runs first (depends on commit order, right?), it will
fail as the new version of component B has not yet been built. Also, it
will not be triggered again after the new version of component B becomes
available. Unfortunately, the hudson buildserver seems to be unable to
trigger build jobs from OSGi dependencies as it can do it for pure Maven
dependencies (pom-style).
Thanks again for your hints and suggestions.
Regards
Timo
2013/1/14 iulian dragos <jaguarul@xxxxxxxxx <mailto:jaguarul@xxxxxxxxx>>
Thanks Nick and Timo! This is one of the biggest pain points for our
build as well: the lack of a simple, incremental, publishing
mechanism, similar to remote maven repositories. P2 composite
repositories seem like a good alternative.
cheers,
iulian
On Mon, Jan 14, 2013 at 5:22 PM, Nick Boldt <nickboldt@xxxxxxxxx
<mailto:nickboldt@xxxxxxxxx>> wrote:
You can build the individual components in order, publish their
p2 repos (update sites) onto a webserver, and composite those
sites into a single site from which you can then build your
aggregate site, product, or appplication.
That's what we do for JBoss Tools / JBoss Developer Studio, and
it allows us to build the various components (~25 of them) in
order or out of order (ie., in the maintenance branch when only
a few of the 25 need to be rebuilt).
Jobs:
http://hudson.jboss.org/__hudson/view/JBossTools/view/__JBossTools_4.0.juno/
<http://hudson.jboss.org/hudson/view/JBossTools/view/JBossTools_4.0.juno/>
Published sites for staging:
http://download.jboss.org/__jbosstools/builds/staging/
<http://download.jboss.org/jbosstools/builds/staging/>
Composite site:
http://download.jboss.org/__jbosstools/builds/staging/___composite_/core/4.0.juno/
<http://download.jboss.org/jbosstools/builds/staging/_composite_/core/4.0.juno/>
Aggregate site built from composite:
http://download.jboss.org/__jbosstools/updates/nightly/__core/4.0.juno/
<http://download.jboss.org/jbosstools/updates/nightly/core/4.0.juno/>
And then, JBoss Developer Studio (the product) is built from the
aggregated JBoss Tools site, and we publish the results here:
https://devstudio.jboss.com/__earlyaccess/
<https://devstudio.jboss.com/earlyaccess/>
https://devstudio.jboss.com/__updates/6.0/
<https://devstudio.jboss.com/updates/6.0/>
Hope that helps!
Nick
On 01/14/2013 10:11 AM, Timo Rohrberg wrote:
Hello everybody,
I would like to share the following user story / use case
description
with you, hoping that some of you might have some hints for
how put an
appropriate system into practice.
We are developing a small component-oriented client/server
application
based on Eclipse Equinox, RCP, and RAP which is constantly
growing
(components are changed and new ones are added). The individual
components' API is also far from stable and thus version
numbering of
components itself and their exported packages is also
constantly changing.
Currently, we build the entire application on our Jenkins
buildserver
with Eclipse Tycho using a reactor project referring the
individual
components. There's a continuous integration build job which
runs on
every commit on one of the components, and a nightly build
job which
builds the entire application on a nightly base.
We would now like to move to a more mature and
component-oriented way of
building the system and its individual releases. The ideas
for our
release management system are as follows:
Development on the individual components is done on the
TRUNK stream.
About four times a year, a release of the entire system
(including all
its components at the particular point of time) is created.
The TRUNK
stream is branched and a complete build is triggered. The
result of this
build becomes the base repository of the release (i.e. the
set of
components forming the release). Main development continues
on the TRUNK
stream, but the new release stream is also maintained. This
means, new
versions of individual components (with changes in minor and
micro
segment) are created and need to be built for individual
shipment to the
customer using the corresponding release.
Unfortunately, we did not yet find a way to put all this
into practice
with a Tycho- and possibly Nexus-based build system. We
would need to
build components individually at least for the release
maintenance
streams, but probably also for the TRUNK stream. While this
is possible
to a certain degree with Tycho (assuming that all
dependencies can be
resolved via the local Maven repository), we need to make
the build
components available for consumption by the system deployer.
One idea is
to deploy them into a Nexus repository, but this seem a
little awkward
as Tycho cannot consume those repositories and also access
to it is not
that straightforward for the system deployer for downloading
individual
components. Another idea is to host p2 repositories on a web
server, but
those cannot be built incrementally, i.e. every single build
of a
component needs to build the entire application keeping all
unaffected
components in the same version which introduces a lot of
redundancy...
So far we did not find an appropriate approach that fits all our
requirements. Maybe you guys do have any idea or hint?
Thank you for any comment.
Timo
_________________________________________________
tycho-user mailing list
tycho-user@xxxxxxxxxxx <mailto:tycho-user@xxxxxxxxxxx>
https://dev.eclipse.org/__mailman/listinfo/tycho-user
<https://dev.eclipse.org/mailman/listinfo/tycho-user>
--
Nick Boldt :: http://nick.divbyzero.com
_________________________________________________
tycho-user mailing list
tycho-user@xxxxxxxxxxx <mailto:tycho-user@xxxxxxxxxxx>
https://dev.eclipse.org/__mailman/listinfo/tycho-user
<https://dev.eclipse.org/mailman/listinfo/tycho-user>
--
� Je d�teste la montagne, �a cache le paysage �
Alphonse Allais
_______________________________________________
tycho-user mailing list
tycho-user@xxxxxxxxxxx <mailto:tycho-user@xxxxxxxxxxx>
https://dev.eclipse.org/mailman/listinfo/tycho-user
_______________________________________________
tycho-user mailing list
tycho-user@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/tycho-user
--
Nick Boldt :: http://nick.divbyzero.com
