RE: [hyades-dev] Meeting minutes for discussion on proposed extensions t

[Antony.Miguel@xxxxxxxxxxxxx]

Hi everyone,

I wasn't able to attend this meeting but here are my 
comments/suggestions. 

I am generally in agreement with these suggested changes but I have a 
few comments in one or two areas:

> Several proposed organizational extensions:
> 
> -Provide a root-level /org.eclipse.hyades.use.cases/ directory for
> functional test suites for use cases.
> -The current organization at the plug-in level does not account for
> functional test cases that cross plug-in boundaries.
> -Hierarchical test suite definitions (e.g. AllTests.testsuite) for
> fine-grained test suite execution and results reporting.
> -Reflect the subdirectory organization of the JUnit and 
> manual test suite
> directories in the JUnit and manual results directories.
> -Internally organize the lowest level JUnit and manual 
> results directories
> by platform (e.g. Windows, Linux, AIX, AS/400, HP-UX, Linux 
> 390, OS/390,
> Solaris) to persist multi-platform test execution results in CVS.
> -Provide a /dependencies/ subdirectory for test suite 
> dependencies such as
> common utility classes, required JARs and datapools.
> -Utilize a standardized namespace (e.g. plug-in ID) when naming and
> identifying test resources.
> -Eclipse Java project configuration files (e.g. .project and 
> .classpath)
> identifying dependencies and the source trees to compile and 
> run JUnit Java
> code.



The hierarchical testsuite definitions are there to a certain extent as 
we have AllTests.testsuite for the whole of Hyades and an 
AllTests.testsuite for each plugin.  To gain better granlarity, the 
plugin provider could organise their own hierarchy of tests which their 
plugin's AllTests.testsuite is the root node for but I'm open to 
suggestions of more mandated structures.  I'm not sure I entirely 
understand this suggestion.

I think reflecting the subdirectory organisation of the JUnit tests and 
manual tests is not necessarily a good idea.  The JUnit tests 
subdirectory organisation might naturally have a different focus to the 
manual test subdirectory organisation.  The JUnit tests will probably be 
organised based on the implementation whereas the manual tests will be 
based more on the UI organisation.  I'm saying this because I can't see 
any obvious benefit from requiring matching folder structures for 
different test types but again if we gain something from this then I'm 
open to the possibility.



> Several proposed procedural extensions:
> 
> -A terse but effective formal test process documentation 
> specifying how
> Hyades plug-ins and functionality are to be tested by 
> utilizing the Hyades
> testing resources.
> -A standardized structured manual test case description 
> including context,
> dependencies, ordered steps of execution and expected results 
> (see Appendix
> A).
> -Utilize formal software quality metrics in measuring test coverage by
> providing documentation on how to execute test suites with 
> the necessary
> code coverage profiling and filter set.
> -Persist test execution reports (e.g. summary and coverage 
> statistics) at
> the highest directory level containing all of the test suite execution
> results.
> -Committers are responsible for generating and checking-in 
> test execution
> reports for their components at predetermined milestones in 
> the test pass.
> -Each committer provides a summary test execution report for their
> components to be used in a web project accessible from the 
> Hyades site.
> -Committers are responsible for checking-in test execution 
> results to CVS
> for their components.
> -Multi-release test suite execution results organized in CVS 
> by release
> branches (e.g. v3_1_0) with HEAD containing the current release.
> 
> Discussion: 
> ...
> -Bruce was concerned with checking-in all the execution 
> result files into
> CVS would be costly especially if the entire source tree is 
> extracted at
> build time.
> -Paul offered the solution that only execution result files for a
> particular release would be associated to a particular branch 
> in CVS.  This
> still left the outstanding issue of the HEAD stream in CVS with no
> alternatives offered.
> ...



All this sounds OK to me.  I strongly agree with Joe that the standard 
structural test case description components should be part of the model.

As far as checking in results files into CVS, I can see we have a 
potential bloat problem and having results files associated with a 
particular release seems like a good idea.  I don't see what the problem 
with the HEAD stream is here though.  I would think people would run 
tests and store the results in HEAD as they ran them, then when a branch 
was created, a branch of the results files would also exist.  Then after 
each branch people could delete the old results files from their 
workspace and check the changes into HEAD (thus keeping them in the 
pertinent branch of CVS but removing them from HEAD).  This would mean 
we'd have only the latest up to date results files all the time in HEAD 
but also have only the results files for a particular release under a 
particular branch.



> Future considerations for Hyades Test:
> 
> -A preference pane or extension point (e.g. wizard pane) for users to
> provide an organization-specific template for manual test case
> descriptions.
> -Capture contextual execution information such as tester, platform and
> build driver in the test suite execution results.
> -Provide additional verdict classifications such as n/a (not 
> applicable)
> and n/r (not ran).
> -Stopping and restarting of test suite executions by providing
> functionality to execute test suites with partially completed 
> execution
> results.
> -Simple summary results reporting for one or more execution result(s).
> -More complex statistical analysis on test suites execution results.
> 

I again agree with Joe here that the extra contextual information should 
be part of the model.

Other than that, everything looks good.  Hopefully we can get some 
discussion going about these issues over hyades-dev.


cheers

Antony