[hyades-dev] Reponse to Antony's comments on the proposed extensions to

[Paul Slauenwhite <paules@xxxxxxxxxx>]

Hi Team,
      In response to Antony's comments on the proposed extensions to the
existing Hyades testing resources (org.eclipse.hyades.tests):

[Antony's comments on the several proposed organizational extensions]:

1) The hierarchical testsuite definitions are there to a certain extent as
we have AllTests.testsuite for the whole of Hyades and an
AllTests.testsuite for each plugin.  To gain better granularity, the
plugin provider could organize their own hierarchy of tests which their
plugin's AllTests.testsuite is the root node for but I'm open to
suggestions of more mandated structures.  I'm not sure I entirely
understand this suggestion.

2) I think reflecting the subdirectory organization of the JUnit tests and
manual tests is not necessarily a good idea.  The JUnit tests
subdirectory organization might naturally have a different focus to the
manual test subdirectory organization.  The JUnit tests will probably be
organized based on the implementation whereas the manual tests will be
based more on the UI organization.  I'm saying this because I can't see
any obvious benefit from requiring matching folder structures for
different test types but again if we gain something from this then I'm
open to the possibility.

[Paul's response]:

1) This suggestion covers the use case where tester A has been assigned a
subset of test suites from a plugin and they do not want to run all of the
assigned test suites individually.  After further considering, this type of
granularity may not be useful when organizing test results.  Ideally, we
would like to have an one-to-one coupling of lowest-level test suites (e.g.
only containing test case(s)/invocation(s)) and result files so reporting
and
test execution histories are similar and maintainable from release to
release.
I would be content with the existing granularity of hierarchical testsuite
definitions.  The end result is that we should maintain analogous test
suite and
test result structures.

2) I agree.

[Antony's comments on the several proposed procedural extensions]:

1) All this sounds OK to me.  I strongly agree with Joe that the standard
structural test case description components should be part of the model.

2) As far as checking in results files into CVS, I can see we have a
potential bloat problem and having results files associated with a
particular release seems like a good idea.  I don't see what the problem
with the HEAD stream is here though.  I would think people would run
tests and store the results in HEAD as they ran them, then when a branch
was created, a branch of the results files would also exist.  Then after
each branch people could delete the old results files from their
workspace and check the changes into HEAD (thus keeping them in the
pertinent branch of CVS but removing them from HEAD).  This would mean
we'd have only the latest up to date results files all the time in HEAD
but also have only the results files for a particular release under a
particular branch.

[Paul's response]:

1) I have open a Hyades enhancement for this requirement:

76688: Extend the test case model object with contextual information.

2) This suggestion makes more sense if we have one test pass per release.
Multiple test passes cause the same problem, albeit at a smaller scale.  I
believe we still need to maintain separate versions of each test result for
each release for historical analysis.  As such, a hybrid approach with
keeping result
files for a release only in that release (include HEAD) and leveraging a
naming
convention for multiple result files within a release would permit ease in
reporting
and analysis.  We should still need to execute a manual step for every
release to 'refresh'
the result files in HEAD.

[Antony's comments on the future considerations for Hyades Test]:

1) I again agree with Joe here that the extra contextual information should

be part of the model.

[Paul's response]:

1) Here is a list of opened Hyades Bugzilla enhancements:

76686: Provide a preference pane for users to provide an
organization-specific template for test case descriptions.
76688: Extend the test case model object with contextual information.
76689: Extend the test result model object with contextual information.
76691: Expose the reason field in the manual test client.
76692: Allow the reason field in the manual test case result be extensible.
76694: Permit stopping and starting a test suite execution without loosing
state.
76699: Provide simple accumulative verdict summary results reporting.
76700: Provide statistical analysis on test suites execution results.

PS

---------------------------------------------
Paul Slauenwhite
IBM Toronto Lab, Canada

Internet: paules@xxxxxxxxxx
Telephone: (905) 413-3861
Tie Line: 969-3861
Fax: (905) 413-4920
---------------------------------------------