[hyades-dev] black- and white-box, was: integration vs interfacing

[Thomas L Roche <tlroche@xxxxxxxxxx>]

"Saliba, James" Fri, 17 Sep 2004 17:13:20 -0400
>>> From my point of view the larger portion our audience is not doing
>>> "green fields" development; people are extending and build on
>>> existing products/projects. Therefore we must make Eclipse/TPTP
>>> attractive enough for people to want to switch. I believe this
>>> would include items from three general areas:

<big snip>

>>> 3) The ease of writing new test cases with Eclipse/TPTP 

Tom Roche 19 September 2004 20:21
>> I would add one more

>> 4) The ease of maintaining test cases developed with Eclipse/TPTP

>> FWIW 3 and 4 are my primary concerns: specifically, reducing the total
>> lifecycle cost of testing for *developers*, in order that they and
>> their management will see the (admittedly overhead) cost as
>> acceptable, and begin to consistently/continually do automated
>> testing, rather than pushing it off to "dedicated" or "3rd party"
>> acceptance-type/black-box testers. But that's a topic for another
>> thread ... suffice to say that, IMHO, while black-box tools have made
>> progress in lowering the cost of _creating_ tests, tests thus
>> created/run will never be as maintainable (or even, in many cases, as
>> easy to develop) as those developed using xUnit-based tools that

>> * allow developers to create tests as (or even before) they develop
>>   their sources

>> * leverage developers' {access to, knowledge of} their sources

>> * are more easily integrated into development tools and processes

>> <off soapbox/>

Michael.Norman Tue, 21 Sep 2004 20:36:08 +0100
> From the size of the soap box obviously this white-box vs. black-box
> thing exercises a lot of people, but personally I've always seen
> value in both approaches, and I think amongst Hyades launch goals
> was to be methodology-neutral.

Agreed. To express the above better, v0.1:

AFAICS most Hyades developers come from a black-box background, and
are mostly interested in doing more/better black-box tooling.
Black-box is not bad! but IMHO

0 The nature of defect-cost increase ensures that (ceteris paribus)
  there are more potential cost savings to be gained from increased/
  improved testing by developers (and architects, but that's another
  thread) than from increased/improved testing by traditional,
  acceptance-style testers.

At some point current terminology, which assumes all testing is
manual, and that developers don't test, will need to change--if one
develops automated stress tests for application(s) to which one lacks
access to sources, is one a "tester" or a "developer"? But for now,
when I say "developer" I mean someone whose primary responsibility is
to deliver sources (no matter how much testing of those sources s/he
might do) and by "tester" I mean someone whose primary responsibility
is to verify the quality of an application or its deployment (no
matter how much coding s/he might do) and who lacks access to the
target's sources.

1 Presently testers are far better served by current tooling than are
  developers. There is already a lotta black-box tooling out there.
  Certainly much more time/energy has been invested in developing
  black-box tooling (for dynamic testing) than has been invested in
  white-box tooling. (Static analysis is a separate matter, and not
  what I mean by "white-box" or "developer-time" testing.)

Therefore 

2 Presently the marginal benefit likely derivable from investment in
  tooling intended to assist developers to do more/better automated
  testing greatly exceeds the marginal benefit likely derivable from
  investment in tooling intended to assist testers to do more/better
  automated testing.

How to do that?

3 "White-box" or "source-enabling" tools (like JUnit and its
  extensions) that

* allow developers to create tests as (or even before) they develop
  their sources

* leverage developers' {access to, knowledge of} their sources

* are more easily integrated into development tools and processes

  are more likely to realize the potential savings of developer-time
  testing. IMHO, while black-box tools have made progress in lowering
  the cost of _creating_ tests, tests thus created/run will never be
  as maintainable (or even, in many cases, as easy to develop) as
  those developed using xUnit-based tools possessing the properties
  listed.

This does not mean that JUnit extensions cannot be used to
acceptance-test or drive deployments, only that black-box tools are
(IMHO empirically) suboptimal for developer-time testing.

Conclusion:

4 I would like to see more Hyades effort devoted to assisting
  *developers* to do more/better automated testing using JUnit
  and its extensions.