[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [wtp-dev] Java EE smoke passes with 5 bugs!


I guess we should write this down, if we haven't already, but I think for a weekly I-build that a "fail" means a blocking bug or bad regression and we would not promote this build to the download site, and we would instead fix and respin, as soon as possible. (It would be an exception to the rule, to "fail", not promote, and wait till the following week to produce a usable build).

I think "pass" means the build is worthy of using as a development target for the coming week. To some degree, it also means adopters and early testers could use it to do or continue testing. But, that degree (that is, the testers expectations) varies quite a bit during a milestone (and during a release) and I think at this point, it would take quite a pretty bad build to interfere with early testing. Oh, and most of all ... "pass" means we would not be embarrassed with what those adopters or earlier testers saw :)  and they would not be surprised by what they saw.

That leads to the middle ground of "pass, with restrictions" (or, what ever we call it). This means that it is worthy to use as a development target, but earlier testers should be aware that some areas have problems.

As for your direct question, I always think its a good idea to check logs and problems view but suspect very few of those types of bugs would be "blocking". But if new such problems were found right before release or perhaps even a milestone, it might indicate a regression that would need to be investigated before promotion.

Hope this helps and is not too far from what people have been thinking a smoke test means. If this doesn't match what others think the smoke tests mean, please comment, and once we reach consensus, we will document it :)

P.S. while we're add it ... please check your logs collected during JUnit tests ... there's sometimes lots of errors in there ... and I think one enterprising team created some JUnit tests which actually checked the logs! (I'm not sure of details, though, or if that test is a good example for others to follow).






From: "Mitov, Kiril" <k.mitov@xxxxxxx>
To: "General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
Date: 10/10/2008 12:54 PM
Subject: [wtp-dev] Java EE smoke passes with 5 bugs!





Hi, I have just executed the java ee smoke test for

Build
http://build.eclipse.org/webtools/committers/wtp-R3.1-I/20081010055538/I
-3.1-20081010055538/

There were no problems with the functionality declared in the smoke
test. All the verify statements have "passes" and the provided
fuctionality is working.

There are, on the other hand, a number of issues (5) connected with
errors in the error log or warning in the problems view. I have created
the appropriate bugs
http://wiki.eclipse.org/WTP_Smoke_Test_Results_R31_101008

Because of this issues I was thinking of declaring the smoke test as
"failing".
But it is more appropriate to first extend the checks in the smoke test
so that they include
"Verify there are no errors in the error log".

At least for the maintenance release this might be appropriate.

What do you think? Should we extends the smoke tests to check the error
log and problems view?

Best Regards,
Kiril


_______________________________________________
wtp-dev mailing list
wtp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/wtp-dev