Re: [eclipse-dev] more green builds

[Mike Wilson <Mike_Wilson@xxxxxxxxxx>]

For comparison purposes, here's my recollection
of what we decided in the call:

- We like the fact that "*all*
the tests pass" is used to decide whether the build is green or not,
but are frustrated that running them takes a long time. 

- As a longer term goal we ought to
find ways to make our test infrastructure more parallelizable, but that
would be a real development effort.

- What we *don't* want is for a test that is known to intermittently fail
(for example, because of network timing) to be the cause of a red build.
All of those tests should be rewritten to "pass" for the intermittent
cases, and just log the behavior instead. For each such test, it is the
responsibility of the test writer to track its output and verify that no
hidden failures have arisen.

- We also decided that everyone *should*
use the rel. eng. tool to do their build submissions, but I don't believe
we reached consensus that they *must*. However, as a group we are going
to be extremely intolerant of build failures caused by bogus, manual submissions.

On more thing...

<rant>
None of us are children. We (or our
components) are not going to be punished for breaking a build. Nobody is
going to say "I'm sorry, you get a 1 week timeout from submitting".
But we all need to understand that gratuitously breaking a build is _A_BIG_DEAL_.
I swear, if I see one more "that was a problem in the test" message
(for a test that would have failed if you had run it in your workspace
before submitting), my head is going to explode. [Yes, I know that would
be entertaining. No, you shouldn't try to make it happen.] This process
is too important for it to be failing due to shear laziness.
</rant>

McQ.

"Walter Harley" <wharley@xxxxxxx>
Sent by: eclipse-dev-bounces@xxxxxxxxxxx

02/14/07 17:08

Please respond to
"General development mailing list of the Eclipse project."        

To	"General development mailing list of the Eclipse project." <eclipse-dev@xxxxxxxxxxx>
cc
Subject	[eclipse-dev] more green builds

In today's arch call we discussed how to have more
builds that are coded
as "green."  A green build does not necessarily imply 100%
perfection,
just that the build is good enough to be used and that there's no need
for a rebuild.

Seems like these are the criteria:

1. A good-enough build should be marked as green.
2. If a build is not good-enough, there should be an opportunity to fix
it, which implies a maximum time for build/test.
3. There should be no randomly-failing tests that can make the build go
red.
4. Implementation of any build/test changes should not require a lot of
releng team time or new hardware.
5. There should be no category of tests whose failures are ignorable
enough that they routinely fail.

Here's my synthesis of a suggestion I heard, with some added
embellishment:

1. Teams divide their tests into "must-pass" and "should-pass".
 How do
teams decide that?  I'd suggest working it out on the basis of a
total-time budget, perhaps weighted by the amount of code being tested.
E.g., say that we have one hour to run the must-pass tests; if JDT Core
has 10% of the code in the SDK, then the JDT Core "must-pass"
tests get
6 minutes.  If a team's must-pass tests run past their limit, the
tests
still run but the team gets a nastygram.  There is no hard limit on
time
for should-pass tests.

The time allotment does not need to be recalculated frequently.  Perhaps
some teams feel that their must-pass tests cannot fit within the
allotted space; one can imagine, then, some sort of chocolate-based
auction system, similar to pollution cap-and-trade schemes.

2. The build cycle looks like this: first build; then run the must-pass
tests.  If the build or the must-pass tests fail, the build is red.
 If
they pass, the build is green; but then the should-pass tests proceed to
run.  If any should-pass tests fail, the failure is reported, but
the
build is still green, and there is no rebuild.  (Or it could be yellow,
as has occasionally been suggested.)

Since a rebuild would only happen if the build is red (ie if build or
must-pass tests fail), the extra time taken by should-pass tests is not
a problem.  But every team still gets the benefit of having all their
tests run on every platform on every build.


Personally I don't see any problems with that approach; it seems to meet
all the criteria, and it seems it should not be too hard to implement.
Thoughts?  If it's worth further discussion I can send to eclipse-dev,
but I thought I'd ping you guys first.

  -walter
_______________________________________________________________________
Notice:  This email message, together with any attachments, may contain
information  of  BEA Systems,  Inc.,  its subsidiaries
 and  affiliated
entities,  that may be confidential,  proprietary,  copyrighted
 and/or
legally privileged, and is intended solely for the use of the individual
or entity named in this message. If you are not the intended recipient,
and have received this message in error, please immediately return this
by email and then delete it.
_______________________________________________
eclipse-dev mailing list
eclipse-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/eclipse-dev