Community
Participate
Working Groups
3.2 M3 rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close] This test is running too fast and failing on most test machines
testRestoreOpen fails as well. The failures begin on build 20051123 Here are the 20051116b numbers Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 218.58K Working Set: 643.6K Committed: 780.8K Working Set Peak: 568.8K Elapsed Process: 311 ms Kernel time: 106 ms Page Faults: 172 CPU Time: 306 ms GDI Objects: 68 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: 118.17K Working Set: -65536 Committed: -83148 Working Set Peak: 17.6K Elapsed Process: 28 ms Kernel time: 10 ms Page Faults: 5 CPU Time: 31 ms GDI Objects: -67 And here are the M4 numbers Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 1.03M Working Set: 1.24M Committed: 874.4K Working Set Peak: 1.17M Elapsed Process: 526 ms Kernel time: 115 ms Page Faults: 335 CPU Time: 501 ms GDI Objects: 72 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: 121.27K Working Set: -61030 Committed: -39731 Working Set Peak: 18.4K Elapsed Process: 46 ms Kernel time: 12 ms Page Faults: 6 CPU Time: 35 ms GDI Objects: -72
And here is the 20051122 numbers. The issue is apparently between 1116 and 1122 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 677.59K Working Set: 1.15M Committed: 945.2K Working Set Peak: 1.07M Elapsed Process: 544 ms Kernel time: 95 ms Page Faults: 318 CPU Time: 534 ms GDI Objects: 67 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: 213.84K Working Set: -62259 Committed: -22528 Working Set Peak: 14K Elapsed Process: 59 ms Kernel time: 9 ms Page Faults: 9 CPU Time: 40 ms GDI Objects: -67
If you replace 20051129 with the workbench from 20051122 you get these numbers Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 601.04K Working Set: 989.2K Committed: 1,000.4K Working Set Peak: 905.2K Elapsed Process: 385 ms Kernel time: 78 ms Page Faults: 262 CPU Time: 393 ms GDI Objects: 70 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: 162.43K Working Set: -74137 Committed: -166297 Working Set Peak: 4.8K Elapsed Process: 29 ms Kernel time: 7 ms Page Faults: 3 CPU Time: 28 ms GDI Objects: -69
If you replace the 1122 builds workbench, jface and core commands you get the following Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 330.45K Working Set: 633.2K Committed: 690K Working Set Peak: 545.6K Elapsed Process: 278 ms Kernel time: 85 ms Page Faults: 171 CPU Time: 278 ms GDI Objects: 70 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: -105416 Working Set: -49152 Committed: -142950 Working Set Peak: 19.6K Elapsed Process: 31 ms Kernel time: 4 ms Page Faults: 9 CPU Time: 28 ms GDI Objects: -69
Bringing in the layout code gives the following (30% performance degradation) Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [open]' (average over 10 samples): Used Java Heap: 331.91K Working Set: 641.2K Committed: 561.2K Working Set Peak: 554K Elapsed Process: 364 ms Kernel time: 107 ms Page Faults: 173 CPU Time: 320 ms GDI Objects: 70 Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest#testRestore() [close]' (average over 10 samples): Used Java Heap: -104212 Working Set: -48742 Committed: 78.8K Working Set Peak: 19.2K Elapsed Process: 28 ms Kernel time: 7 ms Page Faults: 9 CPU Time: 23 ms GDI Objects: -69 When you bring in keys, the workbench, actions, menus, commands, handlers and services you get about the same result. I am going to move this to Eric as I think the TrimLayout performance is worth looking at but I suspect the IDE has some of the issues too.
I think a bunch of the OpenClose Window, Workbench, and Perspective tests are affected by this as well. See bug 116238 for more details.
Also note these tests are running in less than 100ms on 4 of 5 machines for the M5 performance results.
Tod, I'll look into this as part of the cleanup work on the trim to determine how much has been intro'd by the new creation/layout code.
The performance on this test is getting steadily worse with the same spike at 20060407 we had before.
This is still a problem in the performance test results for 3.2
And in 3.3 http://fullmoon.torolab.ibm.com/downloads/drops/I20070424-0930/performance/eclipseperflnx1_R3.3/org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest.testOpen()%20%5Bclose%5D.html They also run too fast - we want to make the test meaningful first and then we should see what the results are like
*** Bug 116282 has been marked as a duplicate of this bug. ***
*** Bug 116283 has been marked as a duplicate of this bug. ***
There is an apparent slowdown in shutdown but it is a degradation from 45ms to 70 ms so I don't think it is something we need to worry about. I would add a degradation comment but it will require us to rework the test. That will cost us all of our history in 3.3 so I think we should do that early 3.4.
Tod, it's early in 3.4 now...;-). Is there anything I should be looking at?
org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest.testOpen() is where you should look
Created attachment 132499 [details] Patch The test appears as green / yellow in 3.5 vs 3.4. However, the shutdown times measured by the test on all test machines are < 50ms. To make some sense from the results, they have to be at least 100ms, see: http://wiki.eclipse.org/Performance_Test_Fingerprint_Guidelines The patch modifies the test to only record the startup times that are on the order of about 300ms - 500ms. The shutdown times are verified to only be < 120ms. Should shutdown times exceed 120ms test will assert and it will be time to check it. The patch also removes some unused code.
Oleg, I've looked at the patch and it looks fine (now that you've explained how the testing works...;-). The only nit is that you have used the word 'inherit' rather than 'inherent' in the comment about the close times. Note that we will have to apply the same patch into the 3.4 'performance' branch at the same time we commit it here to maintain parity between the two tests. Do we have to do anything special to regenerate the baseline 3.4 once we've done this? Boris, what did we decide on for bumping the version? Since this is test work it's not under the same constraints as the released code so I'm going to re-target this to RC1 to allow for more time for work on the release code. Right now (for as yet undetermined reasons) I can't seem to get the tests to run. one set consistently gives 'out of handles' and the other (the revamped ones) fail because 'workbench already exists'. Oleg found a suspicious singleton somewhere (Oleg?). This is one main reason I'm deferring this until RC1.
(In reply to comment #18) > Right now (for as yet undetermined reasons) I can't seem to get the tests to > run. one set consistently gives 'out of handles' and the other (the revamped > ones) fail because 'workbench already exists'. The "IllegalStateException: Workbench already exists" is caused by runing the test in UI mode while it is expected to run in a healdess mode. (The RCPPerformanceTestSuite is described in the "core-test" Ant target which makes testing framework run those tests in the headless mode.) I can't duplicate the running out of handles error; could you try it again on a recent clean build?
Boris, the patch as is won't apply due to changes in the OpenWorkbenchIntervalMonitor & RestoreWorkbenchIntervalMonitor. I've tried it by removing the two imports you added and applying, then fixing the import error (I didn't get one from the OWIM?). Could you review this just to ensure that both fixes work together nicely?
Created attachment 134453 [details] Updated patch The RCPTestWorkbenchAdvisor was moved into a different place which caused changes in "import" statements. Attaching updated patch (+ fixed the typo in the comment).
Committed in >20090508. Applied Oleg's last patch to the 3.5 stream. Committed in >20090508 (3.4 stream): - Branched org.eclipse.ui.tests.rcp and applied Oleg's original patch (since the refactoring done in 3.5 wasn't there). - Ran the releng tool to tag the changes with "M20090508-perf34x". I'll post the change to releng-dev to re-spin the baseline.
Here's the results I get on an 'ad hoc' test...the real 'test machine' results may vary...;-). Scenario 'org.eclipse.ui.tests.rcp.performance.PlatformUIPerfTest #testCreateAndDisposeDisplayX100()' (average over 10 samples): 3,4 Elapsed Process: 56ms (95% in [32ms, 80ms]) 3.5 Elapsed Process: 54ms (95% in [20ms, 89ms]) Scenario 'org.eclipse.ui.tests.rcp.performance.PlatformUIPerfTest #testRunAndShutdownWorkbench()' (average over 10 samples): 3.4 Elapsed Process: 323ms (95% in [100ms, 546ms]) 3.5 Elapsed Process: 407ms (95% in [89ms, 725ms]) Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest #testOpen() [open]' (average over 25 samples): 3.4 Elapsed Process: 178ms (95% in [167ms, 190ms]) 3.5 Elapsed Process: 213ms (95% in [201ms, 225ms]) Scenario 'org.eclipse.ui.tests.rcp.performance.EmptyWorkbenchPerfTest #testRestore() [open]' (average over 25 samples): 3.4 Elapsed Process: 214ms (95% in [200ms, 228ms]) 3.5 Elapsed Process: 234ms (95% in [219ms, 249ms])
Verified for 3.5RC1 using I20090512-2000 results
Thanks Frederic...
