[eclipse-dev] more green builds

["Walter Harley" <wharley@xxxxxxx>]

In today's arch call we discussed how to have more builds that are coded
as "green."  A green build does not necessarily imply 100% perfection,
just that the build is good enough to be used and that there's no need
for a rebuild.

Seems like these are the criteria:

1. A good-enough build should be marked as green.
2. If a build is not good-enough, there should be an opportunity to fix
it, which implies a maximum time for build/test.
3. There should be no randomly-failing tests that can make the build go
red.
4. Implementation of any build/test changes should not require a lot of
releng team time or new hardware.
5. There should be no category of tests whose failures are ignorable
enough that they routinely fail.

Here's my synthesis of a suggestion I heard, with some added
embellishment:

1. Teams divide their tests into "must-pass" and "should-pass".  How do
teams decide that?  I'd suggest working it out on the basis of a
total-time budget, perhaps weighted by the amount of code being tested.
E.g., say that we have one hour to run the must-pass tests; if JDT Core
has 10% of the code in the SDK, then the JDT Core "must-pass" tests get
6 minutes.  If a team's must-pass tests run past their limit, the tests
still run but the team gets a nastygram.  There is no hard limit on time
for should-pass tests.

The time allotment does not need to be recalculated frequently.  Perhaps
some teams feel that their must-pass tests cannot fit within the
allotted space; one can imagine, then, some sort of chocolate-based
auction system, similar to pollution cap-and-trade schemes.

2. The build cycle looks like this: first build; then run the must-pass
tests.  If the build or the must-pass tests fail, the build is red.  If
they pass, the build is green; but then the should-pass tests proceed to
run.  If any should-pass tests fail, the failure is reported, but the
build is still green, and there is no rebuild.  (Or it could be yellow,
as has occasionally been suggested.)

Since a rebuild would only happen if the build is red (ie if build or
must-pass tests fail), the extra time taken by should-pass tests is not
a problem.  But every team still gets the benefit of having all their
tests run on every platform on every build.


Personally I don't see any problems with that approach; it seems to meet
all the criteria, and it seems it should not be too hard to implement.
Thoughts?  If it's worth further discussion I can send to eclipse-dev,
but I thought I'd ping you guys first.

  -walter
_______________________________________________________________________
Notice:  This email message, together with any attachments, may contain
information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated
entities,  that may be confidential,  proprietary,  copyrighted  and/or
legally privileged, and is intended solely for the use of the individual
or entity named in this message. If you are not the intended recipient,
and have received this message in error, please immediately return this
by email and then delete it.