[hyades-dev] Re: [JUnit extensions] goals for script/record/play

[Thomas L Roche <tlroche@xxxxxxxxxx>]

Tom Roche 14 September 2004 18:32
>>>> I'm wondering how folks feel about the following:

>>>> 0 The importance of record/play. My impression is that (ceteris
>>>>   paribus) we prefer JUnit extensions that support record/play
>>>>   to tools that do not (e.g. that only provide API with which
>>>>   users can write JUnit test methods). I would further assume
>>>>   that, c.p., we would prefer tools that provide *both* API and
>>>>   record/play to tools that provide [either] only API or only
>>>>   record/play. Does anyone disagree? If so, why?

Until someone disagrees, I'm assuming universal agreement on that ...

>>>> 1 The importance of human-writable formats. I consider
>>>>   script/record/play to be 3 sides of the same "aspect," since I
>>>>   assume that

>>>> * one's recorder will hafta record (write) to some format

>>>> * one's player will play (read) from that format

>>>> * that format should be read/writable by humans, so that folks
>>>>   can (minimally) edit their scripts for maintenance or
>>>>   (maximally) write their own scripts without needing to record
>>>>   everything

>>>>  Does anyone disagree? If so, why?

... and that. However ...

>>>> 2 The desirability of supporting script/record/play with
>>>>   procedural languages other than Java. My bias is toward
>>>>   persisting to either Java (since that's what the workbench
>>>>   currently best supports) or XML (since it's

>>>> * easily played-back by JUnit: e.g. Abbot provides a
>>>>   ScriptTestSuite and ScriptFixture that do this

>>>> * more cross-compatible: i.e. one can use XSLT to generate some
>>>>   procedural language from XML more easily than one can
>>>>   translate one procedural language into another

>>>>  ). I'm wondering, are there compelling reasons to persist to a
>>>>  non-Java procedural language?

... is more open, but see Hyades Trace Model below. (I'm assuming
that's EMF, no?) Even more open:

>>>> 3 The desirability of supporting one script/record/play format
>>>>   for all supported tools. Should we prefer to support tools
>>>>   that share some common persistence format? Should we provide
>>>>   some common "hub" format to which tools could plug in? Or
>>>>   should format not be a consideration, at least at this stage?

Michael.Norman 09/15/2004 06:52 AM 
>>> THis kind-of relates to an old issue of the replayability of the
>>> Hyades trace model - the old trace/test duality.

>>> There was a lot of discussion of this and I think the point we
>>> got to was that the current Hyades Trace Model would be extended
>>> to cover method data, C/C++ and protocols such as HTTP and would
>>> become the recording persistence format for the various sample
>>> tools.

>>> Then, the traces themselves wouldn't necessarily be used as
>>> replayable test models, there would be a process of trace-test
>>> conversion which mapped between them, and obviously

!-)

>>> allowed you to hang datapools and other things around them.

Just to clarify: I don't work in a Hyades group, and my "real job" is
not as a developer for test tooling. (Not that I wouldn't like to do
that, but until I can get a paycheck out of it, test tool development
is just an all-too-infrequent-engaged hobby.) In relation to Hyades,
think of me as a (hopeful) user, i.e. someone whose knowledge of "the
old trace/test duality" and the Hyades Trace Model (which I'll call
"HTrM" below) is miniscule.

Pointers to more information would be appreciated, esp regarding

* the Hyades Trace Model

* the Hyades Test Model

* how they differ

sluiman 15 September 2004 15:18
>> Yes a review of these in the f2f

* the "Hyades glossary" :-)

>> is very appropriate.

>> There are calls being run by Marius on the subject of adding
>> protocol etc. support to the trace model, starting with required
>> parameter capture. As that comes on line the http recorder will
>> have to migrate to that model. The base spec is due in the 3.1
>> time frame, and the impl is targeted in 3.2 based on our plan.

When you say "the http recorder," does that refer to one in Siefkes'
tooling, or to one included in one of the URL test tools under
evaluation?

>> UML2 is still a long way out with what adoption really means and
>> U2TP to be spec'd for 3.1. This work is not on track at this time.
>> The UML2 model project is also not yet confirmed with dates so
>> this topic is going to remain long range.

>> Dominique is driving the alignment of our test environments to
>> JUnit for the Java space with the first cut spec in 3.1 and the
>> impl currently targeted for 3.3.

>> So once the model is figured out for at least capture we can start
>> the work to record to the model and then the steps of creating
>> tests from recordings. Each "recorder" that is built to create the
>> skeleton of a test needs to be done in this context and any
>> requirements they drive will need assessment. It is this
>> multiplicity that makes this a somewhat tedious at times effort of
>> compromise ;-)

Michael.Norman Wed, 15 Sep 2004 16:43:15 +0100
> I think I understand Harm's response and I think it's what I thought
> the plan was from the Hyades project side. In other words the sample
> tools record into the trace model, the tests are generated from the
> traces, the editor and the codegen operate relatively generically on
> the tests.

> So this would give us a generic way of dealing with at least some of
> these types of tools e.g. Abbot & Costello by mapping their existing
> persistence formats

Just to clarify: Costello is the recorder, which "has" the format.
Abbot contains API (for writing JUnit test cases, and allowing a JUnit
test runner to run Costello scripts) as well as Costello.

> through to trace models describing interactions with the APIs and
> then running our generic editor and generic codegen on them. All we
> need is to get the Hyades trace stuff to act as a generic recorder
> and we've subsumed the tools. We can then release the thing inside
> RCP and call it Abbot & Costello ;-)

> However, there's another way of doing this which is to take the
> tools more or less as is with their own editors recorders and replay
> engines and integrate them as external behaviours which happen to
> fire some stuff back into the execution history.

OK, barring further correction, it sounds to me like

* We prefer JUnit extensions with recorders to those that don't. (I'd
  still like to know how folks feel about script format.)

* For now, if a JUnit extension has its own recorder, we use it as is.

* At some point, we should decide what is preferable between

· "recorder interfacing": i.e. providing an interface between an
  external script tool (EST) and the HTrM. One way to do this is to
  facilitate "[integration of ESTs] as external behaviours which
  happen to fire some stuff back into the execution history." Another
  way might be to read/write or import/export ("trace-test conversion")
  the scripts generated by an EST. In either case, the user would
  presumably be interacting separately with the EST (to record/play)
  and Hyades (for other magic).

· "recorder integration": i.e. "[getting] the Hyades trace stuff to
  act as a generic recorder [and subsuming] the [ESTs]."

· something else entirely. Am I missing one or more alternatives?

Is that correct?

> Which of these were you thinking of?

Recorder interfacing, specifically via read/write/import/export (see 
"hub format" above). This is because

* I didn't know about the HTrM (and don't know enough about it now :-)

* it seems like that would get us "off the ground" quicker, at least
  for tools that script in XML (which I prefer anyway, for the reasons
  above).

* it seems more extensible. IMHO there are always gonna be other
  record/play tools, just as there are JUnit extension domains that we
  have not tackled. (Not to mention xUnit domains.) If we create a
  clean HTrM interface, we can point other tools at it and say,
  "here's how you can Play Well With Hyades."

This is not to say that I oppose recorder integration, just that it
seems like more work. Am I missing something?

> Your postings to hyades-dev and abbot-users give entirely different
> impressions of this!

Can you cite the posts? Because IIRC I've not discussed integrating
Costello with Hyades outside this thread. What I _have_ previously
discussed is creating a plugin that would emit CBEs on Abbot-specific
events (e.g. {Component, Widget}NotFoundException, 
Multiple{Component, Widget}sFoundException), which seems to me like a
separate issue (i.e. logging).

Is that the same issue? Or were you referring to something else?

TIA, Tom Roche, IBM Rational Developer Model2 Tooling, abbot admin