Bug 183338 - [perfs] Too many JDT/Core performance tests are yellow
Summary: [perfs] Too many JDT/Core performance tests are yellow
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.3   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 3.3 RC1   Edit
Assignee: Frederic Fusier CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 184015 185580
Blocks:
  Show dependency tree
 
Reported: 2007-04-20 04:42 EDT by Frederic Fusier CLA
Modified: 2007-05-21 03:26 EDT (History)
0 users

See Also:

Attachments
Proposed patch for perf_32x branch (73.25 KB, patch)
2007-04-20 04:50 EDT, Frederic Fusier CLA 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Fusier CLA 2007-04-20 04:42:50 EDT 
Since a while too many JDT/Core perf tests are yellow in releng perf results HTML page.

Although there's a problem with the method used to decide a test must be yellow or not (Student's t-test), we must try to tune our tests to make them as green as possible...
Comment 1 Frederic Fusier CLA 2007-04-20 04:45:28 EDT 
Looking closely at measures done within each test, it seems that for some of them, warm-up is not enough. So, the JIT continues to optimize code during the measure and make them too dispersed...
Comment 2 Frederic Fusier CLA 2007-04-20 04:50:32 EDT 
Created attachment 64398 [details]
Proposed patch for perf_32x branch

This patch increase the warm-up for all search, build and model tests as most of them are yellow. Note that global time to run tests will not be increased as I fixed a problem on project indexing test which made 100 measures instead of 10!

Also improve local perf tests storage and display to have a precise idea of what's wrong on measures when a test becomes yellow...
Comment 3 Frederic Fusier CLA 2007-04-26 09:27:39 EDT 
When bug 184015 will be fixed, we should only have approx.7 yellow tests for whole JDT/Core perfs tests on all boxes...
Comment 4 Frederic Fusier CLA 2007-05-16 05:59:20 EDT 
Fix for bug 185580 was released yesterday => consider this one as fixed for 3.3 RC1
Comment 5 Olivier Thomann CLA 2007-05-17 10:35:34 EDT 
We still have 12 tests as yellow and 2 red.
Not sure this should be considered as verified for RC1.
Comment 6 Frederic Fusier CLA 2007-05-21 03:21:36 EDT 
In 3.3 M6 we had 55 yellow tests and 5 red ones. The goal was not to remove all yellow tests but try to understand why we had so many. I think we reach this goal as we have now less than 5% of yellow tests instead of more than 20%.
I'll still try to reduce this number - and this would be feasible as most of the error are less than 4% (current threshold is at 3%) - but I definitely consider this bug as verified for 3.3 RC1.
Comment 7 Frederic Fusier CLA 2007-05-21 03:26:57 EDT 
I've opened bug 188043 for the too big standard error on FullSourceWorkspaceModelTests#testCloseProjects() test...