[cdt-dev] Junit testing of CDT components

[Peter Graves <pgraves@xxxxxxx>]

Hey all,
I have been working on the CDT JUnit test suites, and wanted to put a 
question out to the community.

My question related to test cases that trigger bugs that have bugzilla 
reports against them, but will not be fixed for a while. In this case 
with the current setup these failures will show up every time the tests 
are run, and make it a bit harder to pull out the new failures. What do 
people think we should do about this? Some options that have come to 
mind are:

1 - Do nothing, accept that this is how the results look, and take the 
time to pull out new failures.

2 - Reverse the failure/assertion condition, and put a bugzilla id into 
the output string.  This would hide the failure until the bug was 
fixed..  Pros of this are that it would not show up in the results, so 
any failures or asserts would have to be new issues and make it easy to 
catch things as they broke.  As well, when the issue was fixed the it 
would show up in the logs and be easy to track back to bugzilla, and 
close the pr/update the test.  The cons here is that we would manually 
have to go through and update the test cases when things changed, and it 
would be hard to keep track of the open issues that are being tested for 
as they would not show up in the test results.

3 - Post process out the failures with an associated bugzilla id.  We 
could setup the results processing  to search for failures where the 
output message started with PR:  and then assume they are known issues. 
We would then have to make sure when problems were fixed, that the PR 
number was removed from the output message. Pros: Fairly easy to 
maintain, and we can get a bit more information on the number of problem 
reports being tested. Cons: If a problem was fixed, and the test case 
was not updated to remove the PR number, if the bug resurfaced it would 
not show up in the logs. As well if the test was not updated it would be 
hard to find it to clean it up after the fact.

4 - Modify junit to have an idea of expected failures. Pros - get 
exactly what we want/need   Cons - way more work then I think we want to 
get into..

5 - Something else?

Any comments/ideas?

Thanks
-Peter