Community
Participate
Working Groups
This is the top-level enhancment that is used to track all of the other enhancments and defects that are needed to improve our test reports, e.g. renaming test suites, adding summaries to the top of test reports.
Adding 110071 to this enhancement list even though we didn't discuss this issue at the F2F. During some of the Project meetings in 3.3/4.0, there were delays interpreting the test reports because the execution history was missing comments (e.g. bugzilla id) and we didn't know who to ask.
Reassigning to the Project Lead of the Test project.
This is really a P1 for the TPTP project own use of the test reports.
P3 => P1
proposed to be planned for 4.2 but need to clarify the work to be done 4.2 and sizing (investigate keyword added)
Target set to 4.2
Assigned to Jerome.
Found my notes from the Sept F2F that listed the priorities. Here's the priorities, with 1 being the highest priority and 6 being the lowest. P1 1 - need a summary to show which execution histories the attempted status (wedge in pie chart)? % of attempted and not attempted and links to each. P1 2 - platform coverage. Refactor test suites and have a report that can display them. P1 3 - some sort of way to track which build or series of build that it was run on. Have to figure out if the user is asked which build target. Feed from our build results? Impl details up to owner. P1 4 - consolidated lists of exceptions (defects that are blocking test success and inconclusive results, both of those by test across the project). Add the bugzilla priority and resolution status so that you have one nice page, "this is a blocking issue & here's its bugzilla status" P2 5 - hierarchy of test suites (by project, also summaries of tests vs. the long detailed) P3 6 - dashboard. project health page that would combine bugzilla and test results. Summary of numbers by severity/priority. Use BIRT template. - me to open separate enhancements
Reassigning to Sheldon. New implementation will use BIRT.
I am updating this enhancement to include comments from Harm and Tyler. P1 1 - need a summary to show which execution histories the attempted status (wedge in pie chart)? % of attempted and not attempted and links to each. I will provide a new report type that will show a tabular list of test execution histories. The user will have ability to drill down on each test execution history. This report is similar to the existing TPTP tabular report. I will attach a screen capture of the tabular report that will be provided. P1 2 - platform coverage. Refactor test suites and have a report that can display them. This work was already done by another enhancement. P1 3 - some sort of way to track which build or series of build that it was run on. Have to figure out if the user is asked which build target. Feed from our build results? Impl details up to owner. Currently, there's no way to figure out the build target. The test execution histories do not capture this information. I am not sure how this can be done until we change the test execution model to capture this meta data. Unless we require the testers to enter the build id in the comment and tell them to follow a naming convention. This feature will not implement this enhancment since the models do not support this. I will open an enhancement on the test models to capture build id information. P1 4 - consolidated lists of exceptions (defects that are blocking test success and inconclusive results, both of those by test across the project). Add the bugzilla priority and resolution status so that you have one nice page, "this is a blocking issue & here's its bugzilla status" I will create another report type that will build ontop of the tabular report mentioned in sub enhancment 1. The test execution histories structure does not capture bug information. Currently, testers should be entering the bugzilla number in the comments of the test execution if the test has failed. However, this is a guideline when entering test cases. I will create a report based on the bugs noted in the test execution histories however it won't be accurate. This is the best that can be done right now. I can also list the tests that have failed that don't have bugzilla entries. Note that this report will be specific to TPTP Bugzilla naming convetion. This report type should live outside of the TPTP runtime. I will attach a screen capture of this report type. Also note that the report has to connect to the TPTP specific bugzilla server. This is another reason why this report type will be specific to TPTP. P2 5 - hierarchy of test suites (by project, also summaries of tests vs. the long detailed) This requirement will be fullfilled by the new tabular report type I mentioned in the previous comments. P3 6 - dashboard. project health page that would combine bugzilla and test results. Summary of numbers by severity/priority. Use BIRT template. There is another feature that discusses a dashboard. Enhancement 109656.
Created attachment 36764 [details] Tabular report that shows bugs and test executions
Created attachment 36765 [details] Tabular Report
Added tabular report in cvs. Still need to create automation service for test reports.
Please set the target milestone to proper iteration. Also indicate if the investigation is completed.
Added two test reports as outlined in the previous comments.
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this originator of this enhancement/defect has an inactive Bugzilla account and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
