Community
Participate
Working Groups
Verifying results for build I20090331-0901, it appears that there's a regression on all OpenMultipleEditorTest.* tests. All those tests are currently commented but for a regression between 3.2 and 3.3. So, this comment must either be removed and the regression investigated again or updated to explain the difference between 3.4 and 3.5...
The build machine test results show 10% - 110% depending on the configuration. The Windows test machine seems to have consistent 110% slowdown. However, I can not recreate those results on my Windows computer. (Using testing framework with a similar VM.) On my computer tests show about 10% slowdown in 3.5M6 vs 3.4, consistent across all configurations and VMs I tried. I tracked part of the slowdown to the bug 11001 that added extra processing for parts being hidden. If I unapply patch from that bug, I get about 3 - 5% improvements in performance for those tests. I suspect that the rest of the slowdown is a results of multiple bugs, with bug 226746 being another suspect. Summary so far: - I can't recreate resutls from the test machine - On my computer I see slowdown of about 10% - The slowdown seems to be a result of multiple changes, with fix for bug 11001 contributing about half of it.
Eric, could you try to run this test on your XP box?
Oleg, I get the following results: Scenario 'org.eclipse.ui.tests.performance.OpenMultipleEditorTest#testOpenMultipleEditors:java[closeAll]()' (average over 1 samples): 3.4 Elapsed Process: 13.76s (no confidence) Elapsed Process: 16.03s (no confidence) Elapsed Process: 14.22s (no confidence) ------ Average 14.67s 3.5 Elapsed Process: 13.83s (no confidence) Elapsed Process: 14.22s (no confidence) Elapsed Process: 13.81s (no confidence) ------ Average 13.953s 3.4 CPU Time: 19.77s (no confidence) CPU Time: 23.89s (no confidence) CPU Time: 20.19s (no confidence) ------ Average 21.28s 3.5 CPU Time: 19.03s (no confidence) CPU Time: 19.38s (no confidence) CPU Time: 19.5s (no confidence) ------ Average 19.30s
On my WinXP box, comparing 3.4 vs. latest 3.5 I-build: 3.4,s 3.5,s Elapsed process 13.15 14.94 CPU time 14.15 14.80 There is about 10% increase in elapsed time and 3% increase in CPU time. This correspond well to both Linux test environments.
When I tried to run combination of tests to find out if there is some side effects from another test run, some combinations of tests resulted in the out of memory exceptions. Some time before out-of-memory exceptions were thrown, the tests considerably slowed down, presumably as GC tried to scrap all remaining memory bits. Doubling the memory allocation (-Xms128m -Xmx512m) eliminated the out-of-memory error. Hence, a hypothesis: some prior tests (ActivitiesPerformanceSuite ?) allocate large amount of memory that is either impossible or hard to garbage collect. Getting closer to the memory limit intensifies GC activities thus slowing down the test.
Interesting theory! Do we have a way to measure the number of garbage collections during a performane test run?
What made you mention 'ActivitiesPerformanceSuite', does is do a lot of allocations ?
(In reply to comment #7) > What made you mention 'ActivitiesPerformanceSuite', does is do a lot of > allocations ? I listed it because when runing self-hosting performance tests it is enough to have those two test suits to cause out-of-memory exception: UIPerformanceTestSuite with ActivitiesPerformanceSuite, and EditorPerformanceSuite (everything else commented out). When run under profiler, with 512Mb max memory allocated, the total memory consumed by those two test suits is about 336Mb, despite numerous GCs. (The default max, I believe, is 256Mb.) I'll see if we can run the test battery on the Windows test machine twice, once with current memory allocation, and ones with 512Mb max. (The memory allocated at the end of the test run does seem to have strange spots. For instance, I see 19Mb of HashMap entries allocated from the class MutableActivityManager. It would be very useful to see where memory goes, but that will take time and will be 3.6 thing.)
There are two dimensions to this bug: A) The 10% slowdown in the opening of editors B) The discrepancy between test machine results and local tests for Windows XP While (A) is unfortunate, most of the difference has been tracked down to fixes for the bug 11001 and bug 226746. Those differences have been accepted as a tradeoff for the improved functionality. The (B) is why this bug stayed open for so long. Several people tried to reproduce results from the test machine and none could. We used the same test framework to drive tests, and I used similar VM and did both full performance test runs and partial runs. In all cases the difference we observed was around 10%; nowhere near the 100% difference that test machine reports. One suspicious area we found during those attempts was the heap memory consumption. I opened: bug 278057 to consider increasing max heap size for performance tests (they currently run into the max heap size), and bug 278062 to investigate memory consumption in opening and activating editors. Another observation is that for some reason the actual machine that runs WinXP tests is very sluggish. From the hardware specs it should be fine, yet in actual use it feels very slow. While that should impact similarly 3.4 (baseline) and 3.5, there is always a chance that whatever makes it slow is used more often in the 3.5. It would be nice to remove this concern from the equation. (Also when tests take 5+ hours to run it becomes impossible to try multiple configurations to check for cross-test interactions.) I don't think there is anything more we can do for this bug in 3.5 other than update the comment on the test regression to point to this big.
Created attachment 137409 [details] Change to the degradation comment
good for me...
Released to HEAD.
