| [platform-ui-dev] Performance regressions |
Hi all, (and in particular, hi Oleg and Hitesh, I need your help :-)
We have a couple of performance regressions in the 3.5 stream. To see them, go to http://download.eclipse.org/eclipse and click on a recent build's clock symbol. This should bring you to the performance results main page. Scroll down and click on org.eclipse.ui, then look at the table for all 141 scenarios.
The regressions are in EditorSwitchTest (two scenarios), OpenCloseEditorTest (three scenarios), and OpenMultipleEditorTest (six scenarios). Note that the last one already had a regression in the 3.4 cycle but we were not able to fix it for 3.4, and we are now seeing further regression in 3.5. Also note that we have been told that the results for the Windows machine currently cannot be trusted.
I have spent the last days trying to reproduce the measurements, on a freshly installed Linux machine, with no success. I was not able to reproduce slowdowns in the same precentage range as shown on the performance result pages when comparing 3.4.0 with newer 3.5 builds.
Our goal in the short term should be to understand what is going on - I will have to explain this to the other component leads at the next weekly planning meeting on Wednesday.
Of course, the real goal is to either (1) fix the performance problem, (2) determine that it is not a serious problem worth fixing, (3) be able to explain the performance regression by a bug we fixed that could not have otherwise be fixed, or (4) find someone else to blame (e.g. a lower level component, or a change in the test setup that is not in our control). Obviously, (1) would be the best outcome. :-)
Oleg has already helped a lot by writing a wiki page on how to approach performance testing (http://wiki.eclipse.org/Platform_UI/Performance_Tests) and by implementing a mechanism to run a single performance test case (https://bugs.eclipse.org/bugs/show_bug.cgi?id=271456). I hope that on Monday, I can work with Oleg on the reproducability problem (the short term goal).
Hitesh, could you please run YourKit on the scenarios mentioned above, against both a recent 3.5 build and a 3.4.0 build, and compare the resulting traces? See the wiki page Oleg wrote for steps to follow. Any insight into what has changed between the two would be very useful. You may want to run the profiler in the mode where it records invocation counts, in case this makes the comparison easier. If you find anything that can be optimized easily, feel free to experiment if you can improve the performance, but I am not expecting any easy wins.
If anybody else has additional ideas or would like to help, let us know!
Thanks,
Boris.
P.S. Note that some of these tests have rather strange performance numbers. For example, some have *way* more elapsed time than CPU time. Others show a jump in both the 3.5 and the baseline (3.4.0) numbers, some in December and some in January, pointing to a cause outside of the Platform UI code. Also, make sure that you look at the raw data and not only the pretty graphs (click on the "Raw data and Stats" link on the individual scenario pages). We even have tests that sometimes go completely off rails, taking more than a minute for what used to take less than a second.