[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
RE: [wtp-dev] Stability of I-builds during release shutdown
|
Drawing from what others have said, here are my thoughts on improving the build process.
Some assumptions, please set me straight if I'm off base:
1) In principal, changes should not be released to the map files if they cause compilation errors in any WTP components or cause any JUnit failures. The "nightly" build from head should be used to check for compilation and test failures if you are not building and running tests yourself.
2) Declaration of I-builds currently involves a fair amount of manual smoke testing and subjective approval.
3) Builds from head and from the map files can be scheduled/automated. (Naci, David, how much manual work are you doing when you "re-spin" an I-build from map files?)
4) Update of the website with the latest build from head or map files can be automated.
Given that, I'd propose we have four types of build in the short term, with the goal of going down to three as our automated test coverage improves.
1) Head build: Peformed as frequently as possible, somewhere in the every 4 to 12 hour range (thus "nightly" is kind of misnomer). The website only maintains the last 1+ head builds (perhaps you can only download the last head build zip, but view stats for the last N head builds?). It is acceptable to check in changes that cause compilation and test errors in the head build, it's assumed that you are in the process of integrating with other components, fixing up tests, etc.
2) Map build: Performed as frequently as possible, also in the every 4 to 12 hour range. As with head builds, the website maintains the last 1+ Map builds as space allows. Causing compilation or test errors in a Map build is unacceptable - you should have waited for a head build pick up your changes and verify no compilation or test errors before releasing the change. I'd recommend that a breakage mail goes to wtp-dev to "encourage" compliance. Compilation breakages of Map builds are expected to be fixed immediately, test breakages might get a few hours grace period.
3) Integration build: This is a map build that has zero compile or automated test errors that has "blessed" via some amount of manual testing. Because producing this build requires manual certification, it can only be achieved every couple weeks. But note that the only reason we have an Integration build at all is because the automated test suite isn't rich enough yet. In theory, as the test suite matures, this type of build would be obsolete. Teams wishing to adopt a pre-Milestone build would simply take the most recent Map build, which we continuously strive to keep error free.
4) Milestone/RC build: A blessed map build that has achieved some level of functionality. Once every 6 weeks or so.
So I agree with both Thomas' point that we need a map-based build containing released changes much more frequently. And with Naci's point that we should be decreasing the frequency of I builds because they are labor intensive.
I think the only thing we are lacking are continous Map builds, and stricter enforcement of breakage policy for these builds.
-Ted
-----Original Message-----
From: wtp-dev-bounces@xxxxxxxxxxx [mailto:wtp-dev-bounces@xxxxxxxxxxx]On
Behalf Of Naci Dai
Sent: Wednesday, June 29, 2005 8:40 AM
To: General discussion of project-wide or architectural issues.
Subject: Re: [wtp-dev] Stability of I-builds during release shutdown
I like Tims's suggestion of the variable frequency of I-Builds, we will
remain with the weekly, but increase the frequency nearing the
end-of-cycle periods. I will also suggest for these frequent I-Build
period, we adopt the name Release Candidate (RC) instead of the usual
I-DateTime labels.
We should not commit to a certain number but if we produced one every
2-3 days leading to R0.7, the last two weeks of July, we will have 3-4
of them. Also, calling them RCs will make us produce them to near
release quality. Also, see Tim's (Wagner) R0.7 end game plan. It is
clearly stated that the last two weeks is about wrapping up wtp so this
will be appropriate.
>
> I think the issue here is that adopters of WTP need a two things:
>
> 1) they need stability so they can develop new function instead of
> catching up with interface changes
> 2) they need fast turnaround time for critical fixes so they can
> proceed with testing
>
> These are valid requirements and we need to start satisfying them at
> this point in the cycle. We need to stop making interface changes and
> focus exclusively on quality. If we do that then our I-builds will
> have high quality and adopters will be able to move to them quickly.
>
> I suggest we continue with weekly I-builds. I believe no one is
> planning any interface changes at this time. Let's monitor the
> situation and take a checkpoint a week after M5 is released. I'd like
> to ask Tim and Thomas to immediately flag any regressions during this
> time so we can understand the cause.
>
> Arthur Ryman,
> Rational Desktop Tools Development
>
> phone: +1-905-413-3077, TL 969-3077
> assistant: +1-905-413-2411, TL 969-2411
> fax: +1-905-413-4920, TL 969-4920
> mobile: +1-416-939-5063, text: 4169395063@xxxxxxx
> intranet: http://labweb.torolab.ibm.com/DRY6/
>
>
> *Timothy Deboer/Toronto/IBM@IBMCA*
> Sent by: wtp-dev-bounces@xxxxxxxxxxx
>
> 06/29/2005 08:50 AM
> Please respond to
> "General discussion of project-wide or architectural issues."
>
>
>
> To
> naci.dai@xxxxxxxxxxxxx, "General discussion of project-wide or
> architectural issues." <wtp-dev@xxxxxxxxxxx>
> cc
>
> Subject
> Re: [wtp-dev] Stability of I-builds during release shutdown
>
>
>
>
>
>
>
>
>
>
> Hi Naci,
>
> I agree about N-builds - they are only useful for getting the very
> latest changes, with absolutely no stability guarantee. I do not think
> we should change our N-builds at all.
>
> What I think both Thomas and I are getting at is that for external
> teams, weekly I-builds are not enough. If somebody is waiting for a
> change or a bug fix and it is done incorrectly or misses the next
> I-build, you can easily wait two weeks to get a build that you can
> use. During early milestones we might be able to slow down the
> I-builds; during intermediate milestones once a week is ok; but during
> release shutdown this lag in the cycle is too much - our turnaround
> time for builds will keep people from verifying bugs or getting
> through a complete scenario with WTP.
>
> The Eclipse platform does an RC-build, stops producing I-builds for a
> couple days so that people can test, and then does an I-build every
> few days (spinning until clean) until the next RC. All I'm suggesting
> is that we do something similar to ensure that we have a stable RC- or
> I-build at least once a week, and preferrably more like twice. Since
> we'll be in shutdown, it should be easier (and safer) for component
> leads to release their map files more often.
>
> Thanks,
> Tim deBoer
> WebSphere Tools - IBM Canada Ltd.
> (905) 413-3503 (tieline 969)
> deboer@xxxxxxxxxx
>
> *Naci Dai <naci.dai@xxxxxxxxxxxxx>*
> Sent by: wtp-dev-bounces@xxxxxxxxxxx
>
> 06/29/2005 08:02 AM
> Please respond to
> naci.dai and "General discussion of project-wide or architectural issues."
>
>
> To
> "General discussion of project-wide or architectural issues."
> <wtp-dev@xxxxxxxxxxx>
> cc
>
> Subject
> Re: [wtp-dev] Stability of I-builds during release shutdown
>
>
>
>
>
>
>
>
>
>
>
> I will have to add to the discussion as some of the suggestions are
> not workable.
>
> N-Build, or a nightly build, is a build constructed from the cvs head
> directly. This build at the best of the times indicate the status of
> the latest changes. It is not meant for adoption and/or testing, but
> it will give an indication of the current status of the code. The
> fundamental property is that this builds configuration is *not*
> managed. An n-build is not reproducible. Current N-Build frequency
> is once every 12 hours if there are any changes. However, N-Builds
> have not been used very successfully in the last 10 months. N-Builds
> do not offer good feedback to produce I-Builds.
>
> I-Build, the integration build is configuration managed build. The
> component owners release their cvs tagged code into this build. These
> are maintained in a set, called the build map. Each integration build
> has a unique cvs tagged mapped. It is reproducible. An explicit
> cross-component effort and testing takes place to declare an I-Build.
> Only way to get feedback on the quality and status of an I-Build is
> to try to produce that I-Build repeatedly until satisfied.
>
> Back to the main point, we should and do have only one Integration
> build per week. The fact that there are many trial I-Builds produced
> in the open confuses the issue, we will start restricting access to
> these warm up/trial builds, as they are only meant as a feedback to
> the committers to fix the problems before the I-Build is declared.
>
> One I-Build per week maybe too frequent in itself, it gives little
> breathing room. However, this was necessary because of the loose
> coordination between the teams, and number of changes accumulated in
> two weeks made the integration harder and often impossible. We can
> switch to a longer frequency after we are convinced that the code base
> has stabilized to afford a longer integration period. Currently the
> number of changes each week are massive to keep the time short.
> Remember the M1 & M2 horror, they were the integration builds due to
> the lack of integration builds.
>
> Post R0.7, wtp adopters can use M-builds, or communicate the I-Build
> they are based on so that we continue maintaining it to assist less
> painful adoption.
>
> Finally, the quality of an I-Build can only be good (all tests pass,
> smoke testing approved, component owners approve, etc), and, it
> should not regress wrt to a prior build. We had hard time achieving
> this goal due to test quality issues and api changes, and relaxed our
> policy during the M5 cycle. Relaxed means we declared some builds
> with test failures. Post M5, we should activate the strict policy again.
>
> My experience trying to improve I-Build quality by suggesting good
> practices over email has been pretty much useless, we will have to
> enforce the practice locally within each team. Peer-pressure and good
> quality feedback is the mechanisms for achieving the level of quality
> we seek. WTP is the second largest eclipse project (platform being
> the first), and probably the largest geographically in terms of number
> of teams on how they are distributed so we will have to find the
> different ways to manage it ourselves.
>
> In terms of build frequency, can I have people declare opinions on the
> following survey:
>
> a) Produce an I-Build once a week (Thursday - as usual)
>
> b) Produce an I-Build once every two weeks and more only when needed.
>
>
> I also support frequent *trial* I-Builds (4 hours etc. more if
> needed), prior to declaring each build. Doing this the day before is
> too short as the team distribution crosses the day-night barrier so we
> will have to extend it to at least 48 hours.
>
> Yes. We observed similar pattern.
>
> Adopt an I-Build is a very costly process for us. It is measured in
> man-days.
>
> Every time, we adopt I-Build that doesn’t go as far as the previous
> one, we take big hit on productive and we would not able to proceed
> with some implementation. Without proceed with implementation, we
> won’t discover critical bug that need to discover before 0.7.
>
> Because we only have 2 weeks, I would think that we want to go one
> level further. We should respin nightly build every 4 hours, and make
> it the first priority to have all compile error and JUnit test fixed
> once it is discovered in nightly build. And, all JUnit test should be
> deemed critical.
>
> This suggestion might appear to be costly. But, I believe the saving
> from the unstable build’s cascade effect is beyond the cost. After
> all, compile error and JUnit tests need to be fixed anyway. Fixing it
> early enable people/team in the downstream to start working on that
> earlier.
>
> In short, I counter propose:
>
> 1/ We move the expectation of a nighlty build up, as what we would had
> expected from a I-Build
>
> 2/ We respin the nightly build every 4 hours.
>
> 3/ We fix all compile error and all JUnit error within a I-Build.
>
> 4/ We produce a warm-up I-Build on Thursday.
>
>
> Thomas
>
>
>
>
> ------------------------------------------------------------------------
>
> *
> From:* _wtp-dev-bounces@eclipse.org_
> <mailto:wtp-dev-bounces@xxxxxxxxxxx>
> [_mailto:wtp-dev-bounces@eclipse.org_] *On Behalf Of *Timothy Deboer*
> Sent:* Tuesday, June 28, 2005 6:19 PM*
> To:* _wtp-dev@eclipse.org_ <mailto:wtp-dev@xxxxxxxxxxx>*
> Subject:* [wtp-dev] Stability of I-builds during release shutdown
>
>
> Hi everyone,
>
> The 0.7 release candidate builds are proposed for the 13th and 22nd,
> with the release at the end of the month. What I haven't seen is what
> happens between these times - presumably we are going to run nightly
> I-builds or something similar. My concern is that with constant
> I-builds typically comes constant churn - every team ends up taking a
> turn at breaking the build and we have brown-outs that are nobody's
> fault - most of the build is good, but there are some less stable
> parts which change from build to build. Our past I-builds are a good
> example - although they are fairly stable on Tuesday, we typically
> have a series of build and JUnit failures until sometime Thursday when
> everyone starts focusing on it.
>
> To ensure that we have frequent, high-quality builds that can be used
> for testing and external teams building on top of WTP, I propose one
> of the following:
>
> 1. We do less frequent I-builds (e.g. every 3-4 days), but use warmups
> like we have done with previous I-builds. For example, we could start
> spinning I-builds on every Monday morning and spin every 4 hours until
> we get it clean, then do it again on Thursday.
>
> 2. We stay vigilant - every nightly I-build should be as good as the
> previous milestone/release candidate. Any compile errors or JUnit
> failures are deemed critical and must be fixed asap by a respin.
>
> Thought? Opinions? Other suggestions?
>
> Thanks,
> Tim deBoer
> WebSphere Tools - IBM Canada Ltd.
> (905) 413-3503 (tieline 969)_
> __deboer@xxxxxx.com_ <mailto:deboer@xxxxxxxxxx>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> wtp-dev mailing list_
> __wtp-dev@eclipse.org_ <mailto:wtp-dev@xxxxxxxxxxx>_
> __https://dev.eclipse.org/mailman/listinfo/wtp-dev_
>
>
>
> --
> Naci Dai,
> eteration a.s.
> Inonu cad. Sumer sok. Zitas D1-15
> Kozyatagi, Istanbul 34742
> +90 (533) 580 2393 (cell)
> +90 (216) 361 5434 (phone)
> +90 (216) 361 2034 (fax)_
> __http://www.eteration.com/__
> __mailto:nacidai@acm.org__
> __mailto:naci@eteration.com________________________________________________
> wtp-dev mailing list
> wtp-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/wtp-dev
> _______________________________________________
> wtp-dev mailing list
> wtp-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/wtp-dev
>
>------------------------------------------------------------------------
>
>_______________________________________________
>wtp-dev mailing list
>wtp-dev@xxxxxxxxxxx
>https://dev.eclipse.org/mailman/listinfo/wtp-dev
>
>
--
Naci Dai,
eteration a.s.
Inonu cad. Sumer sok. Zitas D1-15
Kozyatagi, Istanbul 34742
+90 (533) 580 2393 (cell)
+90 (216) 361 5434 (phone)
+90 (216) 361 2034 (fax)
http://www.eteration.com/
mailto:nacidai@xxxxxxx
mailto:naci@xxxxxxxxxxxxx
_______________________________________________
wtp-dev mailing list
wtp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/wtp-dev
