Bug 342087 - [JUnit] Compare results of different JUnit executions
Summary: [JUnit] Compare results of different JUnit executions
Status: ASSIGNED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: UI (show other bugs)
Version: 3.7   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: ---   Edit
Assignee: JDT-UI-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords: helpwanted
Depends on:
Blocks:
 
Reported: 2011-04-06 18:38 EDT by Nitin Dahyabhai CLA
Modified: 2012-03-20 06:44 EDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nitin Dahyabhai CLA 2011-04-06 18:38:52 EDT
In WTP Source Editing we spend time, when we find it, bolstering our compatibility with the W3C DOM unit test suite (bug 214439).  Currently we fail nearly 100 of the 499 tests, and I find it difficult making forward progress as it is easy to make changes that (hypothetically) let us pass 5 tests while causing 4 tests that *used* to pass to now fail.  The overall numbers are better but it's basically a regression at that point, and one that's hard to figure out unless there's a way for me to compare the results of the suite executions before and after my changes (not comparing the results in a single execution as is offered for expected/actual).

Ideally when I bring up the History of runs, there'd be a new button to compare the exactly two runs I've selected so I can know both the net pass/fail change as well as easily spot tests that previously passed but now fail, and vice-versa.
Comment 1 Dani Megert CLA 2011-04-07 02:24:41 EDT
Nitin, maybe a workaround for you might be to export the results to XML (via JUnit view's history drop down menu) and then use text / XML compare?
Comment 2 Markus Keller CLA 2011-04-07 08:22:17 EDT
Good idea, I'll keep this in mind for 3.8. This would also be nice item for a new contributor to work on.

Apart from a Compare... button in the history dialog, we should also add Import.../Export... buttons (bug 185056) to make it easier to compare external test runs (e.g. from a build server).