Re: [hyades-dev] Re: [Hyades-team] Test and trace model alignment - cla

[Ian Hunter <ian@xxxxxxxxxxxxx>]

Hi Allan

> The traces we're talking about modeling are highly detailed, potentially
> representing every function call in the whole program. (More likely it'll
> be filtered by class or by triggers that start and stop tracing, but you
> get the idea.) What a Scapa person thinks of when you say "trace" is much
> more coarse-grained, showing the way high-level components (beans) are
> stimulated, how they interact, and the responses they give. If you capture
> this as a trace, it looks almost exactly like the corresponding
> stimulus-response test.

I think of the function calls which we in Scapa are interested in as being a
"stack slice" - if

foo1 calls  foo2 calls foo3
foo4 calls foo5
foo1 calls foo2 calls foo6

then we would only use the knowledge of one level of these (perhaps, but not
necessarily, foo1-foo4-foo1) in our replay.  However the extra information
about subsidiary calls in a trace would not out of place in our combined
trace-test model, as long as we could discern from the trace that we should
not be calling the subsidiary calls.  What information might we need for
this?  Such things as process id, thread id and entry and exit timing.  To my
mind, the records we need from the trace seem to be similar to the needs which

you describe, but we would filter the information differently.


>
> So I will answer the question "Why are trace and test different?" by
> saying "They might be similar, but in general a trace is much more
> fine-grained and exact, while a test is more coarse-grained and abstract."

We are likely to need less information than you, but this should just be a
case of different filtering.  I'm not sure that any difference in coarseness
is justification for significant divergence between the trace and test models.

> Also, the data you attach to something like a "function invocation" in the
> model is different: trace data includes a lot of information about entry
> and exit timing, for example, which makes no sense for a test model.

We have a concept called "mediation", which is the process of converting a
trace into a test.  Even if the test itself did not need entry and exit
timing, mediation might well.  And in fact, for runtime monitoring, having a
record of how long a particular function took is very useful, as if a replayed

function starts to take a good deal longer than its captured counterpart, then

that is a good indication that a system is under stress, and can be used for
runtime metrics.

>
> Conversely, a test model contains data tables for supplying arguments,
> while a trace captures the actual data that was observed during one
> specific execution of the SUT.

Certainly the trace contains the captured data, but the captured value is the
seed of the argument used at replay time.  Often the captured value is put
unchanged into the replay.  We sometimes use a process of generalisation to
generate plausible alternatives to the captured value (which may, of course,
include the captured value itself).  These may in one form manifest themselves

as values in the data table, though we prefer to think of them as inputs into
a generalisation function which chooses which value to pass to the replayed
function.  But the captured value is of central importance, and generalisation

can be thought of something which gets added to the trace as part of the
process of making it into a test.  But it should not mutate the test model to
the extent that it should be significantly different from the trace model.


> (If the trace captures data at all, that
> is.)

If the trace captures no data at all, then we couldn't generalise, but neither

could we replay.  That would simply be an unusable task.  We are not saying
that we would be able to replay all traces, but only that we believe trace and

test are close enough that a common format should be supportable and
supported.

Ian Hunter
Product Development Manager
Scapa Technologies