Hi B., Yahya
I have to admit that your concerns are very valid. I’ll look
into a way to incorporate the requirements mentioned by Barbara into our test
guidelines. Please read this document (I’ll mail it to the list in a few
minutes) and suggest where we can add an interface to a tool like Testopia. I
have a few ideas myself but I want your opinions first.
Achim
From:
ormf-dev-bounces@xxxxxxxxxxx [mailto:ormf-dev-bounces@xxxxxxxxxxx] On Behalf
Of Yahya ÖZTÜRK
Sent: Tuesday, October 21, 2008 8:32 AM
To: The Open Requirements Management Framework project development list
Subject: Re: [ormf-dev] Testopia
In addition I have attached a part of testopia documantation
White
box and Automated Testing
Though not
specifically designed to handle white box tests (which are often automated),
Testopia
can still
provide a repository for test results. Each test case can have an associated
script and test
logs can be
attached to a test case to show the results of testing.
I thought we can use testopia and GUIDancer for powerful test envirenment. In
my opinion coverage is very important knowledge for release and change management.
2008/10/20 Barbara Rosi-Schwartz <Barbara.Rosi-Schwartz@xxxxxxxxx>
Hi.
I understand the general wisdom of not denormalising
information, as you clearly explain in your message, Achim. In my fucntional
testing, I certainly suffered from the pain of having to keep my test scripts,
maintained with an automated tool, and my related test cases and procedures,
maintained with a text editor and/or a spreadsheet, in sync.
However I would like to share the reasons why I found the
ancillary textual and tabular information invaluable:
1) Tracking: the automated test tools I used did not have
any traceability maintaining and stats producing facilities; just staring at a
large number of test script certaimly could not and did not give me an overall
view of where the testing effort was at and what needed doing. In other words,
a high level management perspective was totally lacking.
2) Monitoring of state of development: different test
scripts were in different states in their life cycle (designed, implemented, in
regression, etc.) and it would have been impossible for me to figure out which
test script was in which state of development if I had not had some external
control documents.
3) Coverage: this is really a subset of traceability, which
I mentioned above. Since our development was (and still is) use case driven,
all functional tests were based on use case flows and scenarios. Figuring out
which use case was completely tested in which suite (especially in the case of
included or extending use cases) would have been a nightmare without external
matrices to specify it.
4) Bug identification: when a script failed in regression, I
would go through the test case manually by repeating the steps as defined in
the script. Having a textual document that clearly specified startup, steps and
all conditions made the task of spotting the bug much much easier.
5) Test script reproducibility: If for any reason my test
script got corrupted and I had to reproduce it, again having a detailed test
case document to use as my guideline was pretty invaluable.
I understand when you talk about the different consumers of
test cases. However I think that, even when an automatic test tool is chosen,
as is our case, there are always circumstances in which human interaction or
intervention is needed, as I have tried to highlight in my points above. This
is why, at my first superficial reading of Testopia, I thought it might have
quite a good role to play.
I just thought I would share my findings and my
concerns. As I have said many times, my experience of functional testing
is limited. Also it may well be that GUIDancer solves all issues I have
mentioned above. I am looking forward to Achim's strategy document and his
expertise to learn how the issues I have faced and tried to tackle in a simple,
time consuming and perhaps naive, way, can be solved in a more automated, less
error prone and more elegant fashion.
On 19 Oct 2008, at 15:16, Achim Lörke wrote:
You shall not have different sources for information!
(Especially for something as time consuming as test descriptions and test
results.)
From our experience one has to make a decision: use a tool to
describe test cases for consumption by human testers or design automated tests
for consumption by a test tool. If you try to do both you will end up with
transferring requirements from one database to the other and
results/modifications vice versa (just like UML diagrams and source code if not
tightly integrated) . The two repositories will invariably drift apart.
I have to admit that GUIdancer is not the best tool to describe
test cases for humans (documentation support is planned for next year) but it's
doable. Therefore I wouldn't introduce another tool for test tracking.
[Mat - please nix this up front if there is no chance, otherwise
we will make an initial evaluation.]
Yahya
brought up the subject of Testopia last week at the team meeting but I have not
had the chance to take a look at it until this evening. I for
one misunderstood what is used for. It is not a testing tool, but a
test tracking system from Mozilla. From the website:
Testopia
Testopia is a test case management extension for Bugzilla. It is
designed to be a generic tool for tracking test cases, allowing for testing
organizations to integrate bug reporting with their test case run results.
Though it is designed with software testing in mind, it can be used to track
testing on virtually anything in the engineering process.
It
looks as if it may be a useful addition to our testing toolkit, but the problem
is that it requires patching Bugzilla. Achim/B. if you want to take a very
quick look at it and if you come up with a positive verdict, I can ask the EF
Web Masters to consider making it available. As that is a long shot, I would
not spend a lot of time up front.
###########################################
This message has been scanned by F-Secure Anti-Virus for Microsoft Exchange.
For more information, connect to http://www.f-secure.com/ _______________________________________________
_______________________________________________
ormf-dev mailing list
ormf-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ormf-dev
--
Saygılarımla
Yahya ÖZTÜRK
###########################################
This message has been scanned by F-Secure Anti-Virus for Microsoft Exchange. For more information, connect to http://www.f-secure.com/
|