Re: [wtp-dev] Java EE smoke passes with 5 bugs!

[David M Williams <david_williams@xxxxxxxxxx>]

I guess we should write this down, if
we haven't already, but I think for a weekly I-build that a "fail"
means a blocking bug or bad regression and we would not promote this build
to the download site, and we would instead fix and respin, as soon as possible.
(It would be an exception to the rule, to "fail", not promote,
and wait till the following week to produce a usable build). 

I think "pass" means the build
is worthy of using as a development target for the coming week. To some
degree, it also means adopters and early testers could use it to do or
continue testing. But, that degree (that is, the testers expectations)
varies quite a bit during a milestone (and during a release) and I think
at this point, it would take quite a pretty bad build to interfere with
early testing. Oh, and most of all ... "pass" means we would
not be embarrassed with what those adopters or earlier testers saw :)  and
they would not be surprised by what they saw. 

That leads to the middle ground of "pass,
with restrictions" (or, what ever we call it). This means that it
is worthy to use as a development target, but earlier testers should be
aware that some areas have problems. 

As for your direct question, I always
think its a good idea to check logs and problems view but suspect very
few of those types of bugs would be "blocking". But if new such
problems were found right before release or perhaps even a milestone, it
might indicate a regression that would need to be investigated before promotion.


Hope this helps and is not too far from
what people have been thinking a smoke test means. If this doesn't match
what others think the smoke tests mean, please comment, and once we reach
consensus, we will document it :) 

P.S. while we're add it ... please check
your logs collected during JUnit tests ... there's sometimes lots of errors
in there ... and I think one enterprising team created some JUnit tests
which actually checked the logs! (I'm not sure of details, though, or if
that test is a good example for others to follow).

From:	"Mitov, Kiril" <k.mitov@xxxxxxx>
To:	"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
Date:	10/10/2008 12:54 PM
Subject:	[wtp-dev] Java EE smoke passes with 5 bugs!

---

Hi, I have just executed the java ee smoke test for

Build 
http://build.eclipse.org/webtools/committers/wtp-R3.1-I/20081010055538/I
-3.1-20081010055538/

There were no problems with the functionality declared in the smoke
test. All the verify statements have "passes" and the provided
fuctionality is working.

There are, on the other hand, a number of issues (5) connected with
errors in the error log or warning in the problems view. I have created
the appropriate bugs 
http://wiki.eclipse.org/WTP_Smoke_Test_Results_R31_101008

Because of this issues I was thinking of declaring the smoke test as
"failing". 
But it is more appropriate to first extend the checks in the smoke test
so that they include
"Verify there are no errors in the error log".

At least for the maintenance release this might be appropriate. 

What do you think? Should we extends the smoke tests to check the error
log and problems view?

Best Regards,
Kiril


_______________________________________________
wtp-dev mailing list
wtp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/wtp-dev