RE: [hyades-dev] Interest in contributing a test execution framework

[Michael.Norman@xxxxxxxxxxxxx]

Mike
 
There is a guiding principle in this Hyadfes stuff.  We need to leave 
space so we can all release commercial tooling built on top of the 
framework. This means that above all the project is about tools 
interoperability and not about providing a best-of-breed toolset into 
the public domain, and the decisions to put specific tools into the 
public domain are very sensitive. Some clarity over the natural boundary 
between paid-for and public domain tooling will emerge over the next six 
months or so.
 
On this basis I think it is unlikely that your entire framework would 
make sense in the open source project, at least in the short term.  
However, it would be useful to identify a cleavage point in your 
architecture, the layers  below which are supported by the public domain 
architecture, and the layers above which are your value-add.  If 
modifications are required to allow the public domain architecture to 
support your tooling, these can form your contribution to the open 
source effort.
 
What I will say is that although a a trace model will be released quite 
soon, the test model is not yet in a state where we can release it. The 
initial cut is based upon the OMG RFP revised initial response which, 
although not yet available from the OMG is to be found at 
http://www.fokus.gmd.de/u2tp/Related_Documents/related_documents.html#Latest%20Document 
Beyond this there will be certain modifications and extensions, which 
are due to be thrashed out amongst the initial participants at the end 
of the month. At that point something will be put into the public domain 
and you can start playing.
 
I hope this makes sense.
 
I'd be happy to talk to you about this stuff if you want.  Please 
suggest some times for a call.  I'm GMT, so avoid too late in the 
afternoon.
 
Regards,
 
Mike Norman,
CEO Scapa Technologies
Hyades project lead
 
 
 
 

-----Original Message-----
From: msilverstein [mailto:msilverstein@xxxxxxxxxxxxxx]
Sent: Friday, January 03, 2003 7:32 PM
To: hyades-dev
Subject: [hyades-dev] Interest in contributing a test execution 
framework



Hello: 

We'd like to gauge interest in a potential contribution to the Hyades 
project by SilverMark. We've been creating object testing tools since 
1996 and have a framework that may be of use to the project. I'll 
briefly describe it without burdening you all with a lot of details at 
the outset.

This is a test execution framework implemented in Java, based on 
concepts described in "Automated testing of Object Oriented-Components 
Using Intelligent Test Artifacts", 
http://servlet.java.sun.com/javaone/javaone2000/pdfs/TS-708 and used as 
a basis for the SilverMark's Test Mentor - Java Edition ( 
www.javatesting.com) commercial product. The essence of it is this:

*	The framework provides objects that each possess a type of testing 
behavior. These objects are called steps. 
*	Tests are composed of instances of steps, configured to perform 
specific testing tasks. 
*	The tests themselves may be considered "executable models". One of 
our areas of interest is to use the framework as basis for an executable 
form of the UML Testing Profile. 
*	There are a variety of types of steps, and it is fairly easy to 
extend the framework to provide additional behaviors. Some examples of 
the behavior provided by specific step types are: 

*	calling methods and constructors 
*	setting/getting fields 
*	parsing and iterating over test data 
*	referencing test assets (asset reuse) 
*	sequencing other test steps 
*	running a piece of Java code (script) 
*	recursively extracting object state 
*	validation (assertion-like behavior) 
*	calling out to a JUnit test 
*	accessing distributed objects such as EJBs or through distribution 
schemes such as RMI or CORBA 

*	Steps also share common properties, such as: 

*	name and description, for documentation 
*	expected exception (class of exception expected to be thrown by 
executing the step) 
*	number of times to execute the step 
*	performance criterion (maximum time step should take to execute) 
*	precondition 
*	postcondition 
*	failure (step to execute if the owning step fails for any reason) 
*	invariant (step to execute, which performs an assertion on some 
state that is assumed to be true under all circumstances) 
*	maximum execution time 

*	All of these common properties except for “name” are optional, 
which reduces the overhead of step configuration. Individual step types 
have their own configuration properties. 
*	Steps know how to represent themselves as XML. The objects that 
represent the results of executing steps also know how to represent 
themselves as XML. 
*	The framework includes mechanisms for reusing and passing 
parameters to test assets, as well as iterating over composite 
parameters 
*	Test steps implement the JUnit junit.framework.Test interface, so 
they look and smell to JUnit like JUnit tests, and can even be run under 
JUnit 


Some benefits of using the framework are (in no particular order): 

*	You already have a robust, flexible, well tested framework for 
composing and executing a variety of types of tests 
*	At execution time test steps use reflection to interact with 
components under test, so complete tests can be implemented independent 
of the presence of the system under test 

*	Steps may be configured incrementally, as more information about 
the system under test becomes known. 

*	Test steps can conditionally override Java's private member access 
checks 
*	Each test step measures its own execution time, so fine-grained 
performance metrics are gathered 
*	Tests and test results are independent of presentation, so a 
variety of views can be created for them, specific to the target 
audience 
*	The framework can be easily extended to support additional testing 
behaviors 
*	The framework can be easily extended to gather additional 
fine-grained metrics during execution. For example, it would not be 
difficult to extend the steps' behavior to measure memory consumption or 
code coverage during individual method calls, given the presence of 
callable API for those metrics. 


I've gone on enough. Please let me know if any of this would be of 
interest to the group. 

- Mike 

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Mike Silverstein                        voice:  (919) 858-8300 x29 
SilverMark, Inc.                        fax:    (919) 858-8308 
5511 Capital Center Drive               mobile: (919) 669-5906 
Suite 510, Raleigh, NC 27606            toll free: (888) 588-0668 
http://www.silvermark.com               msilverstein@xxxxxxxxxxxxxx