[News.eclipse.test-and-performance] Re: Will this project be geared to s

[mgn@xxxxxxxxxxxxx (Mike Norman)]

Bryan

Generally-speaking there is nothing in the models that says the SUT is
hardware or software, but the execution framework and codegen is
specifically targeted to certain environments. Interesting this came up in
the context of hardware testing, but you are raising some very important
issues which cut to the heart of what we are trying to do.  

I think some of this is already being addressed in the Hyades project
although it may not be entirecly obvious how it fits together.  The
current test model has no real concept of a boundary, that is a model of
the SUT (or other elements of the test environment) against which the test
interaction is defined - a testability interface, conceptually equivalent
to TRI in the TTCN3 world. We intend to put this in place and work on this
is an early priority once the PMC is running. 

Then, beyond that there is the concept of arbitrary behaviour in the test
case.  Most people are bridging out to external behaviours (e.g.
proprietary scriptng languages) at this point in the adoption of Hyades,
but Hyades does provide a limited behaviour, and implementations thereof
(editors, codegens etc), based upon some concepts we pulled from UML 1.4. 
The plan is to expand this behaviour set, and provide an implementation of
it which, I think would allow you to express your queueing logic, retry
logic etc. in the Hyades behavioural model, and then the Hyades execution
framework would run it.

Now, there are few subtleties to all of this, and I do want to stress that
this stuff is in the design phase. At the moment the plan is for the
behaviour that the execution engine codegens from to be an EMF mapping of
the BPEL standard being developed for web services, and for the
testability interface to be WSDL.  Then, the various agents in Hyades
would also provide WSDL interfaces choreographable from inside BPEL.  The
execution framework would exercise these interfaces alongside interfaces
to the SUT (typically through native bindings rather than SOAP).

Work on this stuff is going on in the M8 Sandbox branch in CVS. 

I expect this is a reponse that is going to elicit more comment...

Mike Norman,
Scapa Technologies
(Old) Hyades Project Lead
(New) Hyades Test Project lead designate

Bryan Hunt wrote:

> Great, here is some more info:

> My current preceived problem with Hyades is in the execution model.  It 
> appears to me that the current execution model relies on the ability to 
> choose the machine(s) on which the test(s) executs, and the requirement 
> of running a daemon on the test machine.  This model does not work when 
> you have thousands of machines that execute tests 24/7 and must be 
> shared with other jobs.

> Our existing environment uses a queueing system to dispatch jobs to a 
> machine for execution.  The job may be a test, or any other type of task.  
> Our system submits tests to this queue for execution 24/7 and results 
> are stored in a database.  Our tests are designed to produce stiumuls 
> real-time in response to a SUT based on a set of configuration 
> parameters.  This allows us to test different functions of a SUT simply 
> by running a test multiple times.  Various configuration parameters 
> control how tests are submitted to the queue.

> Here is a sample use case:

> Bring up eclipse.
> Query the status of the tests ran / failed in the last 24 hours.
> Re-run a failing test with additional parameters to generate additional 
> debug information.
> Debug test.
> Submit all tests with same fail symptom for re-run on new SUT or fixed 
> configuration parameters.
> Shut down eclipse and go home while tests run.

> There's obviously a lot more to our system than I've described above, 
> but this will hopefully get a good dialog started.

> Bryan

> In <cdfo95$78j$1@xxxxxxxxxxx> Tyler Thessin wrote:
> > Large-scale testing of software and/or hardware systems is a target of 
> > the Test & Performance Project (and the Hyades platform developed 
> > therein).  Let us know the specific constraints you have perceived and 
> > we can provide more specific insights for applicability to your 
> > environment.
> >
> > --tyler