[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [stp-dev] STP Committer/Developer commit process page

Hi Antony,

My concern is around areas of B2J such as the distributed engine where automated tests and coverage results will be very hard to acheive (e.g. testing broken network link reconnection or proper thread distribution across multiple nodes in a distributed engine).

I can absolutely see the issue here - and having been in the position of developing the same kind of distributed software for more years than I care to remember, I'm familiar with the complexity of testing software like this :)

My aim is not to have untested and unstable code in STP but to avoid sweeping limits which may be unrealistic or stifle progress for certain components.

As you say though I am sure we can come to an understanding on the IRC chat tomorrow.

It could be useful for us to categorize our tests. Those kind of tests that need to run for cooperating processes, which may not be cohosted, are a very different thing to fine-focus testing, where interface-level functionality gets tested. Coverage on the latter kind of tests is relatively straightforward to measure and indicates that all of the fine-grained participant reactions to input have been tested. Developers go straight in at this level, so having good coverage implies that I can make updates and know pretty quickly that I may have contravened expected behaviours, because all code branches have test drivers. These I refer to as units, and I think we should have a target coverage figure for these.

Of course, you can have 100% coverage on all your unit tests, and the
system can still not work :) So at the next level of testing, coverage
of APIs is not as vital - instead it's interactions between functional
components that you need to cover, especially the behaviours in the
presence of badness. So, I agree that the rules of engagement for these
kind of tests need to be different to the units (BTW - I've usually
called these system tests, even though they generally just involve
different parts of the system).

The amount of testing automation we can achieve within STP will
differ depending on which kind of test we need to run. So while its
a good thing to run the units nightly after commits, it may be
the case that more complex testing needs to be run offline by
the committer.

Anyhow - there's some material for the IRC :)