Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [wtp-dev] Clarification of priorities and measurements


Jeffrey Liu shared with some of us today a little known secret of achieving "quality == test results" and "api coverage == coverage reports" at the same time.

You can have JUnit tests in the build, which are simply not ran -- just omit the test from the test.xml file (the component tool, somehow, does a search of binaries/source? to do its counting, not what's actually ran.

While not all component leads were present, those that were agreed this was doable, so we volunteered the rest to go along :)

So, now there will be no excuse for failing tests and all must be fixed (or, if nonessential, problem-with-the-test-only) can be commented out until the test is fixed.

Not sure do-able by this week (today) ... so component lead wtp-dev list forgiveness may still be granted today only ... but as we move forward we'll expect 100%

Component leads also expressed a glitch in our process that notification of build failures, compile failures, and test failures is spotty or nonexistent, and difficult for them to monitor build page closely. So .. I've entered

90711

  Build and Test failures need to notify immediately
https://bugs.eclipse.org/bugs/show_bug.cgi?id=90711

and hopefully the build team can improve the auto-notification system for all three ... build failures, compile failures, and JUnit failures.



So .. component leads please monitor the last I-build of today, starting at Thursday, 3:30 EDT and inform us all of status if you have failing tests.
Using my magical predictive abilities ... I predict the following failures

Chuck
org.eclipse.jst.j2ee.tests_6
org.eclipse.jst.validation.test_1

Naci
org.eclipse.jst.servlet.tests_15

Tim
org.eclipse.wst.server.core.tests_1
org.eclipse.wst.server.ui.tests_1


Hope I'm wrong :)

Thanks all ... for continuing to improve our processes.






"Ted Bashor" <tbashor@xxxxxxx>
Sent by: wtp-dev-bounces@xxxxxxxxxxx

04/07/2005 02:02 PM

Please respond to
"General discussion of project-wide or architectural issues."

To
"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
cc
Subject
RE: [wtp-dev] Clarification of priorities and measurements





I think it's very important that we produce a weekly integration build that buildmasters feel it's worth spending time trying to adopt.  We need a flag on the download site that indicates the build has achieved "quality" success.

I understand that we also need "api coverage" success for M4.

So I see two options for the short term:
1)  quality = test results + component lead email
    api coverage = test coverage
2)  quality = test results
    api coverage = test coverage + component lead email

I personally think 2) is a more practical and enforceable policy.  1) is fine with me, if there's a flag on the download site that indicates the quality result (and assuming you can get timely component lead email).

Another comment on the phrase "we'd expect JUnit success to always be high".  I believe we should expect JUnit success to always be 100%.  It already seems pretty lenient to only enforce this once a week.

-Ted

                -----Original Message-----
                From: wtp-dev-bounces@xxxxxxxxxxx on behalf of David M Williams
                Sent: Thu 4/7/2005 8:18 AM
                To: General discussion of project-wide or architectural issues.
                Cc:
                Subject: [wtp-dev] Clarification of priorities and measurements
               
               

                Thanks Chuck. just to be clear, the priorities are definitely (still) to 1) define high quality API (or decide something can not be API this release), and 2) produce a high
                quality milestone (in that order, but both high priority).
               
                It just seems to me many of the failing tests didn't have to do with the defining high quality API process, so that's what I was asking component leads to clarify.
               
                So if those are tests failing simply due to the "defining high quality API" process, then its fair for you
                to simply say so (for now, until M4) under the rules established by project leads, in order to have those
                counted in the coverage report.
               
                My concern about, our process, in general, is that if we have lots of tests failing due to the "defining high quality API in advance" process,
                then they are not suddenly going to go away overnight after M4, so we'll have this problem of using JUnits to judge quality of the build for a long time.
               
                So ... you are right, using JUnits to judge quality of the build (the traditional use) and using JUnits to judge API definition in advance of being implemented
                (the proposed new use we are trying out) are in conflict -- we'd expect them to merge overtime as implemented .... but that leaves a fair amount of time of
                ambiguity. Communication is the key, so I appreciate your (continued) comments.
               
                I'll make this concrete proposal -- that is, and is intend to be, consistent with our priorities and goals --
               
                let's shoot for complete coverage of overview docs and JavaDoc for proposed API for M4, and then use the
                JUnit coverage *trend*, as implemented (not defined in advance) to judge quality during the M5 timeframe.
                That is, we'd expect JUnit success to always be high, and coverage to steadily increase as we approach release of 1.0.
                I'm pretty sure this is consistent with project leads and PMC's view and quality goals, but they can verify it if others think
                this is a good idea.
               
                If anyone has any other suggestions to improve our process to accomplish our goals
                (high quality API, high quality implementation) and measure our progress towards them,
                then I think we are all open to suggestions.
               
               
               
               
               
Chuck Bridgham/Raleigh/IBM@IBMUS
Sent by: wtp-dev-bounces@xxxxxxxxxxx

04/07/2005 09:49 AM
Please respond to
"General discussion of project-wide or architectural issues."


To
"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
cc
Subject
RE: [wtp-dev] Thursday I-build schedule                

                                 




               
                I will sustain from voting on this issue, but I feel we need to clarify priority for this milestone.
               
                Our main focus is to define quality api, and build tests that cover this api. - We have been directed to build on this coverage in this milestone.
               
                So I will go ahead and remove our failing tests, which will damage our coverage in the api report.
               
                Just wanted to bring up these conflicting policies, so we can have a clear understanding of whats expected.
               
                Thanks - Chuck
               
                Rational J2EE Tooling Team Lead
                IBM Software Lab - Research Triangle Park, NC
                Email:  cbridgha@xxxxxxxxxx
                Phone: 919-254-1848 (T/L: 444)
               
               
               
David M Williams/Raleigh/IBM@IBMUS
Sent by: wtp-dev-bounces@xxxxxxxxxxx

04/06/2005 07:58 PM

Please respond to
"General discussion of project-wide or architectural issues."

To
"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
cc
Subject
RE: [wtp-dev] Thursday I-build schedule                


                                 


               
               
               
               
                +1 for "Build will not be declared a success if there are any test errors"
               
                The "easy way out" was intended to be a very temporary condition, just until M4 is complete (which is not that far off!) intended to make it easier for developers to specify proposed API, provide a JUnit test for it in advance so it could be counted by the test counter, and have the proposed-api-junit failure not prevent the build from being declared a success.
               
                But, its not working. I'm not sure if its just a bad idea or if component leads just haven't been communicating well. (And, I say this knowing full well I'm one of the culprits, allowing failing xml validation tests last week to go unmentioned and unexplained -- they are non critical but should have been removed until fixed).
               
                So, fellow component leads .. please make sure tests are clean this week, or communicate clearly if they are failing due to the "provide JUnit tests in advance" policy.
                That way, we'll be able to easily decide next week if a change in policy/procedure is needed. (my current impression is we do not have any that are failing due to the "specified in advance" policy)
               
                And, IMHO, I think the main criteria of success of an I-build is if it can be used as a target for further development, so, in many cases, its legitimate to remove failing tests if they are failing due to testing some fringe cases, etc. Might it even be possible to provide the 'proposed-api-tests" and rig things to always have them "fake" success? Long long term, it would be great if the JUnit framework provided a way to specify the "level" of the test, right in the test ... critical, noncritical, stress-only, performance, etc. .... but that's a Release 2 item, if even then.
               
                Thanks for the comments Ted -- it helps to have another perspective.  And three cheers for continuos improvement through continuous iteration :)
               
                Other's opinions welcome.
               
               
               
"Ted Bashor" <tbashor@xxxxxxx>
Sent by: wtp-dev-bounces@xxxxxxxxxxx

04/06/2005 01:32 PM

Please respond to
"General discussion of project-wide or architectural issues."




To
"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
cc
Subject
RE: [wtp-dev] Thursday I-build schedule                




                                 


               
               
               
               
                "Component owners must justify test errors if they want to declare it a pass despite errors."
                Is there any way we can change that policy to "Build will not be declared a success if there are any test errors"?
               
                Seems to me that either the error is critical, and the build is unstable, or it's non-critical, and the test should be commented out until there's time to fix it.  The impression one gets is that failing tests are allowed to hang out indefinitely through the milestone.
               
                This "successful with errors" state is really inconvenient for someone trying to determine whether to adopt the build.  Ideally one should be able to look at the download site and see a green check or red X and get a general sense of stability.  Obviously there are going to be bugs and unfinished work in an integration build, but I think we really need a binary thumbs up or thumbs down that doesn't require digging through email.  
               
                If people really don't want to comment out tests, we could have two status flags, one for "no test errors" and one for "test errors but declared sucessful", but to me that indicates the need for separate test suites.
               
                Another recommendation is that test results start going out with the first candidate build (that builds successfully).  I know that's more spam to the mailing list, but I think it will give us a better shot at achieving a clean I-build sometime between Tues and Thurs.
               
                Thanks, Ted
               
               
                              -----Original Message-----
                              From: wtp-dev-bounces@xxxxxxxxxxx on behalf of David M Williams
                              Sent: Wed 4/6/2005 9:35 AM
                              To: wtp-dev@xxxxxxxxxxx
                              Cc:
                              Subject: [wtp-dev] Thursday I-build schedule
                             
                             
               
                              Thanks goes to Ozgur for confirming. And thinking generally that if I'm confused others might be (which ... I know ... is seldom the case :) I thought I'd re-post the I-build schedule for Thursday. And, now that most of us have gone through daylight-savings-time change, I will even do the math for EDT:
                                      midnight, 11:30 AM, 6:30 PM.
                             
                              And, BTW, no need to wait till asked for "Component owners must justify test errors" part. Let us all know status of failed tests here on wtp-dev.
                             
                              Thanks,
                             
                              = = = = =  original post = = = = =
                              Final-Thursday:
                              Builds are started on 4:00 AM GMT,  15:30 GMT, 22:30 GMT
                              Failure Policy:
                              Rebuild until success.  (Success will be achieved when there are no compile/testing errors).  Component owners must justify test errors if they want to declare it a pass despite errors.  All other unfixed errors will be the source of public humiliation. You must fix your errors.  This becomes the top priority on thursdays.
               
                _______________________________________________
                wtp-dev mailing list
                wtp-dev@xxxxxxxxxxx
                https://dev.eclipse.org/mailman/listinfo/wtp-dev
                _______________________________________________
                wtp-dev mailing list
                wtp-dev@xxxxxxxxxxx
                https://dev.eclipse.org/mailman/listinfo/wtp-dev
                _______________________________________________
                wtp-dev mailing list
                wtp-dev@xxxxxxxxxxx
                https://dev.eclipse.org/mailman/listinfo/wtp-dev
               
               

_______________________________________________
wtp-dev mailing list
wtp-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/wtp-dev


Back to the top