[News.eclipse.test-and-performance] Re: Will this project be geared to s

[mgn@xxxxxxxxxxxxx (Mike Norman)]

Bryan,

It's maybe time we got you off the project creation newsgroups and plugged
into the Hyades newsgroups and developer lists instead.  

It sounds as though you might gain some benefit from some of the things in
Hyades but your core test execution engine would remain.  The most common
mode for existing test tools vendors adopting the framework is invoking
their existing engine as an external behaviour, whilst adopting the
top-level structural elements of the model (testSuite, Test Case, Test
Component etc.) to provide consistency of management with other tools.

You might also find it useful to pick up the test execution history &
reporting for verdicts and/or the statcon for runtime or post-mortem
visualization of test activity (assuming your engine reports things at
runtime), and you may even choose to drive your engine off the placement
stuff and the datapool stuff in the models (the latter probably
corresponds quite well to your sets of parameter sets).  Post 3.0 these
models are under review, so contributions to those discussions are
welcome. 

The data volumes question is actually in the remit of EMF rather than
Hyades, but we push requirements into them.  There is some support for
relational backends in Hyades 3.0 and this will be extended.

Software tests don't either pass or fail consistently by the way.

Bryan Hunt wrote:

> > This is the basic concept spot on.  At some point the behaviours will 
> > be able to be very generic, and the codegen will interface to any SUT
> > provided a thin testability interface is provided to it, and the
> > behaviours will also be able to control any logging monitoring or
> > parameter tuning agents that are kicking about in the environment.
> > 
> > However, at the moment the codegen is specific to certain SUTs and the
> > behaviours are limited to certain constructs.  You can take one of 
> > these and extend it, or bolt in your own codegen through the UI 
> > extension point, and there is a "cat flap" to let in proprietary 
> > behaviours (i.e. non-UML-derived scripts) at this point.

> Ok, while this is a very interesting concept, it won't work for us 
> anytime in the near future.  Our environment effectively consists of a 
> single test component that generates all behaviors at run-time.  We do 
> this because it requires billions and billions (as Carl Segan would say) 
> of behaviors to fully test a SUT.  Our test component is on the order of 
> 1 million lines of C++.  A testcase for us is simply a set of 
> configuration parameters for the test component.

> To execute these tests, we use thousands of machines.  These machines 
> must be shared with non-test workloads.  This requirement does not allow 
> us to use a dedicated testing daemon like RAC.  Jobs are dispatched to 
> these machines from a central server that manages a queue.  Any user may 
> queue any job.

> To maximize throughput of tests, we use another server that supplies 
> test jobs to the queue described above.  Users construct testcaess and 
> make the test server aware of the testcases.  The test server will 
> submit these testcases to the queue 24/7 until a set of specified 
> conditions is met.  As we have thousands of testcases to choose from, 
> the test server is able to keep the queue full of tests.

> It is my understanding (please enlighten me if I'm misinformed) that 
> with software testing if you run a testcase more than once, it will 
> always produce the same behavior and same fail for a given (defective) 
> SUT.  This is not the case for us.  Whenever we run a testcase, a new 
> behavior is created and thus the possibility for a new fail condition.  (
> Note that this feature factors into the use of the test server described 
> above.)  For example, if I run a testcase 100 times on a given SUT, it 
> is possible to get 100 unique fail verdicts.  This situation typically 
> poses a challenge to fail management not seen in software testing.

> You mention the test execution history model.  I'm assuming this is an 
> EMF model that needs to be loaded.  How is the data displayed once the 
> model is loaded?  To manage our fails, we require the ability to make 
> user defined queries to a database, and to display the data in a format 
> as specified by the user.  Right now, the only given is that the fail 
> data will be displayed in tabular format.  Everything else is 
> configurable by the user.

> I'd be interested to know more about how you handle artifact data.  We 
> can easily generate hundreds of gigabytes of artifact data per day.

> There is also a major performance issue for us that gets into a 
> discussion of testcase grouping and data localization, but I'll save 
> that for later.

> Bryan