[tptp-testing-tools-dev] Abbot and SWT testing support in TPTP

[Thomas L Roche <tlroche@xxxxxxxxxx>]

Per suggestion in the 4.1 AG review today (H 2 Jun 05), I'd like to
inquire about getting--and exploiting--better support for Abbot

http://sourceforge.net/projects/abbot

in the test tooling, specifically WRT testing SWT GUIs.

Abbot's community (both users and committers) continues to grow, and
continues to use EJUA (the Eclipse/JUnit/Abbot test platform) in ways
I know little about, so I'm hesitant to speak for it. I know better
the pilot group I'm with @ IBM, which uses Abbot's APIs to write test
methods targeting the Eclipse plugins we write with SWT (for Rational
Application Developer), so here I'm mostly addressing problems in that
space. But the majority of Abbot users test Swing, most of them using
Abbot's XML/JUnit-based record/edit/play tool rather than its APIs:
they should feel free to append their needs, as should anyone with
SWT-testing needs that I have missed (or with different solutions for
the problems below).

Furthermore my current test-tooling experience is nearly all with the
JDT/PDE JUnit runners (esp the latter, more below), so some of my
concerns may be already solved in the TPTP JUnit runners. But I'm sure
that the abbot-users and the tptp-testing folks will correct me where
wrong :-) and in any case I'm seeking here just to get discussion
started. (I would also appreciate pointers to working TPTP JUnit
runners which I and the pilot group could try out.)

Some areas in which I'd currently like to see TPTP's JUnit support
improve over JDT/PDE's JUnit support are

0 provide the ability to run tests (and the test runner itself) off
  the UI thread. AFAIK this is not a problem on the Swing side, but
  for SWT testing thread access is crucial. Fortunately we (esp
  Richard Birenheide of SAP) have written enough code to workaround
  many problems, but we continue to encounter problems using the PDE
  runner that I suspect would go away if the underlying problem was
  solved. Note that we are currently pursuing this via

https://bugs.eclipse.org/bugs/show_bug.cgi?id=90625

1 better support for iterative test development. This usage may not
  make much sense to folks who do traditional recorder-based testing:
  what I seek is better support for testing _by_ developers during the
  development process using code-centric tools, what Kent Beck et al
  are calling "developer testing." (For a recent SD Times article, see

http://68.236.189.240/article/special-20050601-01.html

  For a Beck talk on the topic see

http://www.itconversations.com/shows/detail301.html

  ) Consider the case of a developer-tester of an Eclipse (or RCP)
  plugin: s/he writes tests as, or before, s/he writes function. This
  would not seem to be a problem: write function and test, launch a
  runtime workbench (runbench), test function, loop. That manner of
  iteration works well enough for small products (on the scale of the
  base Eclipse) but not for large products (on the scale of Rational
  Application Developer, where one's target build contains literally
  thousands of plugins), because launching a large runbench takes
  minutes even on {high-speed, large-memory} hardware.

  Currently the non-JUnit PDE runner (Run>Eclipse Application) allows
  one to iteratively develop function via hotswapping: one can often
  make changes in one's launching workbench (devbench), build, and see
  those changes reflected in the behavior of the runbench. This mode
  of development has limitations:

- it encourages manual testing, not automated testing: IIUC there is
  no currently-supported runner for automated tests that runs _from_ a
  runbench (more below)

- reliability is low: eventually hotswap fails and one needs to
  restart one's runbench (more below)

- in works only in Debug mode (not really a problem: Run mode ~= Debug
  mode minus breakpoints)

  But if our developer-tester (testing plugins in-the-large) prefers
  to write automated tests, as s/he should, s/he has a problem:

- in Run mode the PDE JUnit runner, unlike the non-JUnit runner, tears
  down the runbench at the end of the test session.

- in Debug mode one can specify that the PDE JUnit runner not
  terminate the runbench. Unfortunately this does not imply that one
  can rerun, much less iterate development of, one's test: in my
  GUI-writing/testing experience the runbench hangs at the end of the
  test, and tests cannot be rerun from the devbench.

  How to improve iterative test development? Some examples OTTOMH:

+ reduce the time required to launch a very large runbench. Feels like
  a difficult problem to me; OTOH solve that and the general problem
  falls away (and every plugin developer will bless you :-)

* support running of tests _from_ (not just in) a runbench. IBM has a
  TestCollector with a UI (dialog) that can be launched from a running
  runbench, which can then run arbitrary tests selected from that UI.
  Unfortunately IIUC it is currently-unsupported, and the last version
  with which I worked

cvs.opensource.ibm.com:/cvsroot/corona:j2ee-tools/com.ibm.etools.commontests.testcollector

  has not supported UI testing since eclipse 3.0 (due to invalid
  thread access). I believe the code has since been moved to WTP; in
  any case, it is very likely already OSS, and the owners would very
  likely transfer it to TPTP as a more appropriate locus for
  maintenance and further development. (I can assist with
  communication if desired.)

* provide support in the TPTP JUnit runners to not teardown the
  runbench at the end of a test run, and to run tests (from the
  devbench) in an existing runbench.

2 shutdown problems. Eclipse plugin testers often see exceptions and
  errors thrown on shutdown of the runbench. This is very likely due
  to the way that the PDE runner shuts down, and likely due to the
  runner running in the UI thread. An RCP plugin tester also reports

Matthias Küspert 06/02/2005 09:48:29 AM:
> I just noticed that when my test finished and I start using the app
> again, I get a SWTException that says: Widget is disposed.

3 relax restrictions on the types of tests one can run. Currently the
  JDT/PDE runners allow one only to run a subclass of
  junit.framework.TestCase or a class which implements public static
  Test suite(). One should also be able to run a subclass of
  TestSuite.

4 relax restrictions on the location of tests one can run. Currently
  one can only run tests that reside in one's workspace, but a plugin
  developer will often want to run tests in her/his target build (or
  Target Platform), e.g. to verify the functionality of dependencies.
  One should be able to run tests whether source or binary, whether in
  the workspace proper or the Target Platform, or wherever in the
  filespace. Such functionality would obviate the need to temporarily
  {import projects, adjust classpaths} solely in order to test.

5 correct test reporting. Currently the PDE runner reports GUI tests
  passing even before they run. E.g. our example
  abbot.swt.eclipse.tests.JavaProjectTester: JPT's first test is
  headless (creating a Java project using API), the second tests an
  Eclipse wizard (generating a Java class into that project). I have
  observed, with 3.1-{M6, M7, RC1} that the first test does not show
  as passed until it runs, then the runner greenbars and shows "Runs:
  2/2" as the wizard launches.

This is (at least part of) what SWT developer-testers want in order to
improve our testing, and therefore our development. I hope that you
are sufficiently interested in this market segment (which includes
many of your co-workers) to devote the necessary effort.

I'd also like to raise, if only to "scratch the surface," a larger
topic: support by TPTP for developer testing. As the SD Times article
discusses, developer testing is quite the hot area in the test world
today. Tooling for developer testing is much less mature--much more of
an "open field"--than tooling for traditional end-of-the-cycle
blackbox record/play testing.

But developer testing, by limiting defect-cost increase, promises
greater ROI than traditional testing (hence the interest in, and
spending on, products like Agitator). Furthermore, developer testing
already has one foundation (*Unit) which is "real-world proven" in
wide-scale use, has extensions in many domains (web, Java, .Net, GUIs,
databases, performance/load, etc), and has large mindshare. (There are
also newer frameworks (e.g. FitNesse, TestNG) with buzz.)

TPTP is well on its way to providing common frameworks, primitives,
services, and UIs for traditional test functionality, and that's good.
But, frankly, traditional tools have yet to be significantly adopted
by rank-and-file developers for various reasons (notably brittleness
of tests and learning curve for test tools and scripts). TPTP could
bring real value to the larger development community by integrating,
leveraging, and supporting existing developer-testing tools, enabling
developers to write (and more importantly maintain) headless,
function, and performance tests in the languages we use now, with the
(Eclipse-based) tools we use now, for the domains for which we code
now.

Better yet, Abbot's record/edit/play tool (Costello) demonstrates that
one can "have it both ways": one can, using just XML and JUnit (with
no platform-specific bits), drive tests either from API or from a
recorder. It Should Be Possible to create portable tools that are
useful by both developer-testers and traditional testers from the same
substrate. But now I am too far off-topic :-) Thanks in advance for
your support for API-driven developer-testing of SWT GUIs via EJUA.