Community
Participate
Working Groups
I20110329-0800 http://download.eclipse.org/eclipse/downloads/drops/I20110329-0800/performance/eplnx2/Scenario415.html The regression can be seen on all platforms. Also TypeHierarchyPerfTest#testOpenCollHierarchy() shows erratic behavior on all platforms.
ypeHierarchyPerfTest#testOpenObjectHierarchy2() also has a regression http://download.eclipse.org/eclipse/downloads/drops/I20110329-0800/performance/epwin2/Scenario417.html
Numbers for TypeHierarchyPerfTest#testOpenObjectHierarchy2() on my machine *3.6 till 3.7M4* Scenario 'org.eclipse.jdt.ui.tests.performance.views.TypeHierarchyPerfTest#testOpenObjectHierarchy2()' (average over 400 samples): Used Java Heap: 857.24K (95% in [-79.1K, 1.75M]) Measurable effect: 1.84M (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set: 124.39K (95% in [111.29K, 137.49K]) Measurable effect: 26.41K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Committed: 107.26K (95% in [84.87K, 129.65K]) Measurable effect: 45.14K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set Peak: 6.37K (95% in [-2.82K, 15.56K]) Measurable effect: 18.54K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 216ms (95% in [213ms, 219ms]) Measurable effect: 6ms (0.2 SDs) Kernel time: 152ms (95% in [149ms, 156ms]) Measurable effect: 7ms (0.2 SDs) Page Faults: 152 (95% in [139, 164]) Measurable effect: 24 (0.2 SDs) (required sample size for an effect of 5% of mean: 4206) CPU Time: *250ms* (95% in [245ms, 255ms]) Measurable effect: 10ms (0.2 SDs) GDI Objects: 40 (95% in [40, 40]) *3.7 M6 till now* Scenario 'org.eclipse.jdt.ui.tests.performance.views.TypeHierarchyPerfTest#testOpenObjectHierarchy2()' (average over 400 samples): Used Java Heap: 366.04K (95% in [-580.09K, 1.28M]) Measurable effect: 1.86M (0.2 SDs) (required sample size for an effect of 5% of stdev: 6400) Working Set: 131.41K (95% in [83.14K, 179.68K]) Measurable effect: 97.32K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6400) Committed: 134K (95% in [83.74K, 184.26K]) Measurable effect: 101.33K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Working Set Peak: 665 (95% in [-291, 1.58K]) Measurable effect: 1.88K (0.2 SDs) (required sample size for an effect of 5% of stdev: 6401) Elapsed Process: 156ms (95% in [152ms, 159ms]) Measurable effect: 6ms (0.2 SDs) Kernel time: 105ms (95% in [101ms, 108ms]) Measurable effect: 7ms (0.2 SDs) (required sample size for an effect of 5% of mean: 706) Page Faults: 178 (95% in [164, 192]) Measurable effect: 28 (0.2 SDs) (required sample size for an effect of 5% of mean: 4135) CPU Time: *186ms* (95% in [180ms, 192ms]) Measurable effect: 12ms (0.2 SDs) (required sample size for an effect of 5% of mean: 721) GDI Objects: 44 (95% in [44, 44]) This is opposite of the trend seen http://download.eclipse.org/eclipse/downloads/drops/I20110329-0800/performance/epwin2/Scenario417.html and http://download.eclipse.org/eclipse/downloads/drops/I20110329-0800/performance/epwin3/Scenario417.html I am stumped now :/
I'll try to look at this tomorrow.
See comments in bug 270791: we had problems with those tests before. Also note that the test itself is meant to fail when exceeding an absolute band but that is currently not supported by the reports (see bug 89804). Investigating further...
See also bug 323461.
The test is bogus and outdated: it was used to measure how long it takes to compute and show the type hierarchy (TH) for 'Object' but since 3.6 the TH is computed in the background which means the test barely measures something useful. Moreover, it affects the next test because the job which computes the TH might still be running as canceling doesn't happen immediately and not in the UI thread. We should implement it correctly by waiting for the job to finish. Doing this is to late now. For 3.7 we will remove the 'TypeHierarchyPerfTest'. Filed bug 346088 to revive the test in 3.8. Regarding the numbers: On my Windows XP machine I don't see any difference between a 1.5 and 1.6 JRE but I do see a constant difference between 3.6.2 (elapsed time: 250ms) and I20110512-2000 (297ms) which means a degradation of around 19%. I can't see the big 100% diff reported on the releng machines. Deepak's numbers are based on a tweaked test that repeats the test. When I do this I also get the same trend (i.e. faster in 3.7) - this is because the job that computes the TH is canceled faster now.
Removed the test from /org.eclipse.jdt.ui.tests/test.xml.
Verified in performance results of I20110519-1138.
