[News.eclipse.test-and-performance] Re: Will this project be geared to s

[Bryan Hunt <bhunt@xxxxxxx>]

> This is the basic concept spot on.  At some point the behaviours will 
> be able to be very generic, and the codegen will interface to any SUT
> provided a thin testability interface is provided to it, and the
> behaviours will also be able to control any logging monitoring or
> parameter tuning agents that are kicking about in the environment.
> 
> However, at the moment the codegen is specific to certain SUTs and the
> behaviours are limited to certain constructs.  You can take one of 
> these and extend it, or bolt in your own codegen through the UI 
> extension point, and there is a "cat flap" to let in proprietary 
> behaviours (i.e. non-UML-derived scripts) at this point.

Ok, while this is a very interesting concept, it won't work for us 
anytime in the near future.  Our environment effectively consists of a 
single test component that generates all behaviors at run-time.  We do 
this because it requires billions and billions (as Carl Segan would say) 
of behaviors to fully test a SUT.  Our test component is on the order of 
1 million lines of C++.  A testcase for us is simply a set of 
configuration parameters for the test component.

To execute these tests, we use thousands of machines.  These machines 
must be shared with non-test workloads.  This requirement does not allow 
us to use a dedicated testing daemon like RAC.  Jobs are dispatched to 
these machines from a central server that manages a queue.  Any user may 
queue any job.

To maximize throughput of tests, we use another server that supplies 
test jobs to the queue described above.  Users construct testcaess and 
make the test server aware of the testcases.  The test server will 
submit these testcases to the queue 24/7 until a set of specified 
conditions is met.  As we have thousands of testcases to choose from, 
the test server is able to keep the queue full of tests.

It is my understanding (please enlighten me if I'm misinformed) that 
with software testing if you run a testcase more than once, it will 
always produce the same behavior and same fail for a given (defective) 
SUT.  This is not the case for us.  Whenever we run a testcase, a new 
behavior is created and thus the possibility for a new fail condition.  (
Note that this feature factors into the use of the test server described 
above.)  For example, if I run a testcase 100 times on a given SUT, it 
is possible to get 100 unique fail verdicts.  This situation typically 
poses a challenge to fail management not seen in software testing.

You mention the test execution history model.  I'm assuming this is an 
EMF model that needs to be loaded.  How is the data displayed once the 
model is loaded?  To manage our fails, we require the ability to make 
user defined queries to a database, and to display the data in a format 
as specified by the user.  Right now, the only given is that the fail 
data will be displayed in tabular format.  Everything else is 
configurable by the user.

I'd be interested to know more about how you handle artifact data.  We 
can easily generate hundreds of gigabytes of artifact data per day.

There is also a major performance issue for us that gets into a 
discussion of testcase grouping and data localization, but I'll save 
that for later.

Bryan