[news.eclipse.tools.hyades] whitebox and GUI testing

[Tom Roche <tlroche@xxxxxxxxxx>]

Tom Roche wrote:
>>>> Will Hyades provide functionality for GUI test automation?

sluiman@xxxxxxxxxx (Harm Sluiman) Sun, 22 Dec 2002 18:46:54 +0000
>>> One of the most important things we will provide is an EMF model
>>> of the test case structure, and a way to plug in (with good
>>> examples) of how to generate runtime required artifacts.

Tom Roche replied:
>> Great, but allow me to suggest that your initial efforts not be too
>> entirely "top-down."

Or blackbox-focused (more below)

>> It seems to me that Hyades wants most to enable a mode of development
>> in which test is fully integrated into design model, so that one's
>> test artifacts could be generated from The Model along with code,
>> documentation, etc. This is certainly a valid vision.

<But>

>> by providing GUI automation sooner, rather than later, you would
>> also be making Hyades relevant to XP-style developers. [And] the
>> entire Eclipse community's interests are served by making SWT/JFace
>> as testable as AWT/Swing. As you note

>>> I think getting Swing, AWT, SWT, HTML as GUI environments covered,
>>> as well as non-visual environments like Java and Web Services will
>>> be some of our top priorities. After all these are all environments
>>> we all want to improve the quality of and automation of these
>>> componentized environments is key to making that happen in a
>>> productive fashion.

Harm Sluiman replied:
>> When I mentioned the EMF model, this is not just for top down
>> usage. We will use models to capture the results of execution as
>> well as the configuration and definition of the test case. Behind
>> every UI is a model. A very tiny model in some cases.

>> There are in fact already some tools that can adapt to various GUI
>> component models. Rational Robot for one, which can be used for SWT
>> as well as AWT and Swing.

While RobotJ is an interesting tool (which I am beginning to use for
lack of alternatives), it is unfortunately a prime example of blackbox-
centricity. I would vastly prefer something like Rational TestRT,
provided that a launched Eclipse plugin, including its GUIs, be a
supported target.

Mike Norman <mgn@xxxxxxxxxxxxx> Mon, 20 Jan 2003 13:20:26 +0000 (UTC)
> I'd like to enforce

or reinforce :-)

> Harm's point about the top vs. bottom issue. There is (or should be)
> a balance in the Hyades material about these two approaches
> reflecting the different world views of the respective vendors.

Sometimes held by the same vendor: see Rational's "RUP/XP Plug-In"
(aka BDUP4XP :-)

> The key here is that the test model and the SUT model are different.
> The test model describes the test; the SUT model describes the
> system under test.

> One important area of hyades is that by mapping through a standard
> form of test model, the scripts become interoperable so that any
> hyades-compliant tool can manipulate them.

IMHO you're beginning to sound a bit too blackbox. What I mean:

I'm a coder, therefore I have access to my own code, therefore my unit
testing is whitebox. As a Java coder, I already have a test-scripting
framework, JUnit. This allows me to write my own scripts (JUnit test
cases and suites), which any JUnit-compatible test runner can
manipulate. As an Eclipse coder, I even have excellent JUnit support
... except that I code SWT GUIs, and we don't have SWT test automation
support.

If we had that, we could utilize JUnit to the extent that our AWT
colleagues can. They already have open, interoperable tools (see

http://www.superlinksoftware.com/cgi-bin/jugwiki.pl?TestingGUIs

esp Abbot and JFCUnit)--we just want to be able to use them.

_That_ is the problem in my life (or just the problem relevant to this
discussion :-), not the absence of test and SUT models. (To the extent
that such models are required *for whitebox*, models should be easily
obtainable from one's code and one's testcases.)

> Do you think what I'm suggesting does actually solve the problem you
> are concerned with? Have I explained it?

Can you explain the relevance of your proposals (which I can see would
have value in the blackbox or traditional-QA world) for whitebox?

Not that I am disinterested in your quest. Again I suggest that,
to the extent that one's JUnit test cases and suites do not already
constitute your "test model," tools should be able to easily create
one by parsing them. (Or parsing the code itself: e.g. one could adapt
a tool like JUnitDoclet to generate not only JUnit testcase stubs (for
those whose development is not test-driven), but also test or SUT
models.) But

* all the models you can devise will be as naught to SWT GUI coders if
  we cannot exercise our SUTs.

* blackbox-centric tooling will be at best tolerated, and at worst
  fought, by coders. A unified testing framework must be useful to all
  members of the software development community.
Hyades newsgroup: respond to Mike Norman, post re TestRT

Newsgroups: eclipse.tools.hyades
Subject: Re: swt.Robot, was: automate GUI testing?
Date: Mon, 20 Jan 2003 13:20:26 +0000 (UTC)
Mike Norman <mgn@xxxxxxxxxxxxx>


Tom Roche wrote:
>>>> Will Hyades provide functionality for GUI test automation?

sluiman@xxxxxxxxxx (Harm Sluiman) Sun, 22 Dec 2002 18:46:54 +0000
>>> One of the most important things we will provide is an EMF model
>>> of the test case structure, and a way to plug in (with good
>>> examples) of how to generate runtime required artifacts.

Tom Roche replied:
>> Great, but allow me to suggest that your initial efforts not be too
>> entirely "top-down."

Or blackbox-focused (more below)

>> It seems to me that Hyades wants most to enable a mode of development
>> in which test is fully integrated into design model, so that one's
>> test artifacts could be generated from The Model along with code,
>> documentation, etc. This is certainly a valid vision.

<But>

>> by providing GUI automation sooner, rather than later, you would
>> also be making Hyades relevant to XP-style developers. [And] the
>> entire Eclipse community's interests are served by making SWT/JFace
>> as testable as AWT/Swing. As you note

>>> I think getting Swing, AWT, SWT, HTML as GUI environments covered,
>>> as well as non-visual environments like Java and Web Services will
>>> be some of our top priorities. After all these are all environments
>>> we all want to improve the quality of and automation of these
>>> componentized environments is key to making that happen in a
>>> productive fashion.

Harm Sluiman replied:
>> When I mentioned the EMF model, this is not just for top down
>> usage. We will use models to capture the results of execution as
>> well as the configuration and definition of the test case. Behind
>> every UI is a model. A very tiny model in some cases.

>> There are in fact already some tools that can adapt to various GUI
>> component models. Rational Robot for one, which can be used for SWT
>> as well as AWT and Swing.

While RobotJ is an interesting tool (which I am beginning to use for
lack of alternatives), it is unfortunately a prime example of blackbox-
centricity. I would vastly prefer something like Rational TestRT,
provided that a launched Eclipse plugin, including its GUIs, be a
supported target.

Mike Norman <mgn@xxxxxxxxxxxxx> Mon, 20 Jan 2003 13:20:26 +0000 (UTC)
> I'd like to enforce

or reinforce :-)

> Harm's point about the top vs. bottom issue. There is (or should be)
> a balance in the Hyades material about these two approaches
> reflecting the different world views of the respective vendors.

Sometimes held by the same vendor: see Rational's "RUP/XP Plug-In"
(aka BDUP4XP :-)

> The key here is that the test model and the SUT model are different.
> The test model describes the test; the SUT model describes the
> system under test.

> One important area of hyades is that by mapping through a standard
> form of test model, the scripts become interoperable so that any
> hyades-compliant tool can manipulate them.

IMHO you're beginning to sound a bit too blackbox. What I mean:

I'm a coder, therefore I have access to my own code, therefore my unit
testing is whitebox. As a Java coder, I already have a test-scripting
framework, JUnit. This allows me to write my own scripts (JUnit test
cases and suites), which any JUnit-compatible test runner can
manipulate. As an Eclipse coder, I even have excellent JUnit support
... except that I code SWT GUIs, and we don't have SWT test automation
support.

If we had that, we could utilize JUnit to the extent that our AWT
colleagues can. They already have open, interoperable tools (see

http://www.superlinksoftware.com/cgi-bin/jugwiki.pl?TestingGUIs

esp Abbot and JFCUnit)--we just want to be able to use them.

_That_ is the problem in my life (rather, the problem relevant to this
discussion :-), not the absence of test and SUT models. (To the extent
that such models are required *for whitebox*, models should be easily
obtainable from one's code and one's testcases.)

> Do you think what I'm suggesting does actually solve the problem you
> are concerned with? Have I explained it?

Can you explain the relevance of your proposals (which I can see would
have value in the blackbox, QA-intensive world) for whitebox?

Note that I am disinterested in your quest. Again I suggest that,
to the extent that one's JUnit test cases and suites do not already
constitute your "test model," tools should be able to easily create
one by parsing them. (Or parsing the code itself: e.g. one could adapt
a tool like JUnitDoclet to generate not only JUnit testcase stubs (for
those whose development is not test-driven), but also test or SUT
models.) But

* all the models you can devise will be as naught to SWT GUI coders if
  we cannot exercise our SUTs.

* blackbox-centric tooling will be at best tolerated, and at worst
  fought, by coders. A unified testing framework must be useful to all
  members of the software development community.

* given the mindshare of JUnit (and the other children of sUnit),
  Hyades must work either with it or around it.