[stp-dev] RE: Topic for agenda of next IRC

["Trieloff, Carl" <CTRIELOFF@xxxxxxxx>]

Comments on this thread we need to consider.

Carl.




-----Original Message-----
From: Stefan Daume [mailto:Stefan.Daume@xxxxxxxxxxxxx]
Sent: Tuesday, January 31, 2006 8:43 AM
To: Trieloff, Carl
Cc: Antony.Miguel@xxxxxxxxxxxxx; Karl Reti (E-mail); Christophe Ney
(E-mail)
Subject: Re: Topic for agenda of next IRC


Carl,

something that I think needs clarification for the further discussions 
is how we imagine this coverage metrics to be produced.

Further to your comments I would like to be a bit more specific about 
concerns in the area of B2J. My concern is not really GUI testing (as 
you referred to). Large areas of the B2J plugin are only active when 
running as part of a distributed engine.  Things like varying the 
distribution, testing over heterogeneous environments and ensuring that 
the distribution of threads is correct are extrememly hard and time 
consuimg to test with an automated test. We do have a produced a number 
of automated tests. I am not sure though that there is a tool we can use 
to check code coverage when they are running (it would have to work 
across multiple JVMs potentially on multiple machines).

So this (B2J) would be an example where we do have to make an exception. 
If we can agree on this we should be fine to proceed.

    Stefan


Trieloff, Carl wrote:

>Stefen,
>
>I agree that at time test coverage goals do not help the process, but I
>think that these should be exceptions made. The great thing with test
>coverage is that it makes it very easy for someone to be comfortable to merge
>if the tests pass. We all know that is this not fail proof, but then if an
>issue is uncovered the practice should be to add a test for that case.
>
>The goal with this is to make it easier to merge and not have to manually 
>retest. We have found both internally in IONA and with Celtix, the higher
>the test coverage, the faster the project can be delivered with more people as
>there are low counts of unforeseen circumstances.
>
>Sometimes we bring code in with low coverage for some reason, but indefinitely
>we find that at some point it is lower cost just to stop network and get the tests
>in, else you never know if the new work has broken something.
>
>I know GUI's are harder than runtime, so tests don't cover the visual aspects well, and
>test can be bogus - but I think the statement is to set a culture of "test your own work"
>and automate it. (Any suggesting in stating the goal differently would be appreciated)
>
>
>Does that help
>Carl.
>
>
>-----Original Message-----
>From: Stefan Daume [mailto:Stefan.Daume@xxxxxxxxxxxxx]
>Sent: Friday, January 27, 2006 11:10 AM
>To: Trieloff, Carl
>Cc: antony.miguel@xxxxxxxxxxxxx
>Subject: Topic for agenda of next IRC
>
>
>Carl,
>
>Antony and myself had a chat regarding one of the topics of the last 
>weeks IRC namely the issue of STP coding policies and guidelines in 
>general and the test coverage in particular. We would like to put this 
>on the agenda for next week.
>
>After the last IRC we discussed this particular issue on the basis of 
>our experiences in TPTP. We are concerned about enforcing a fixed test 
>coverage rate (70%). Coverage is an interesting measure but should not 
>be a static threshold Our main concerns would be
>
>- that it will stiffle overall STP progress; experience from other 
>projects shows that this particularly true if you enforce dedicated test 
>periods that do not match up with the progress in the individual 
>components and subprojects
>- I think we all know that this measure is not a reliable indicator for 
>stability or quality; a measure incorporating the time spent bugfixing 
>vs new enhancements in relation to the severity and frequency of 
>reported bugs would be more helpful - but it will be difficult to 
>capture that in a single metric as it essentially captures the 
>efficiency of the whole process
>- there are cases where tests cannot be automated and manual testing is 
>required; this has a higher cost associated with it that is not captured 
>in the coverage metric either
>
>I think we will both be able to attend the IRC next week.
>
>Talk to you later,
>
>    Stefan
>  
>