Community
Participate
Working Groups
I20071009-2225 The organize imports performance test is very red. This started after 3.3. Investigation indicates that fix for Bug 196340 is causing this. The fix forces the save of all cus which are not open in an editor, which is correct. The 3.3 performance test is written such that no cus was saved when organizing import. With the fix all cus are saved leading to disk IO causing this regression. Fix is to change the performance test in the 3.3 performance stream such that all cus are saved.
Created attachment 80127 [details] fix
fixed in HEAD and perf_33x streams
Looking at N20071021-0010 performance results, the problem still exists...
Created attachment 81470 [details] yourkit (In reply to comment #3) > Looking at N20071021-0010 performance results, the problem still exists... Mmm, I'm running out of ideas here to be honest. One can clearly see, that the fix we made was correct: N20070909-0010 is almost as fast as 3.3 now. But something must have changed between N20070909-0010 and N20071015-0010 which made the test slower again. But I can't find out what it was. This are 6 (sic!) weeks between the two performance test runs, a lot must have happen between the two. I can see the difference when I run yourkit on 3.3 and N20071015-0010 (6796 ms vs. 7906 ms in OrganizeImportsOperation#run) But the trace just indicates that almost everything got a little bit slower. Any idea on how to track this down is highly appreciated...
(In reply to comment #4) > Created an attachment (id=81470) [details] > yourkit > > (In reply to comment #3) > > Looking at N20071021-0010 performance results, the problem still exists... > > Mmm, I'm running out of ideas here to be honest. One can clearly see, that the > fix we made was correct: N20070909-0010 is almost as fast as 3.3 now. But > something must have changed between N20070909-0010 and N20071015-0010 which > made the test slower again. But I can't find out what it was. This are 6 (sic!) > weeks between the two performance test runs, a lot must have happen between the > two. I can see the difference when I run yourkit on 3.3 and N20071015-0010 > (6796 ms vs. 7906 ms in OrganizeImportsOperation#run) But the trace just > indicates that almost everything got a little bit slower. > > Any idea on how to track this down is highly appreciated... > The dates look similar to those I noticed in bug 206601 comment 3... Perhaps your test is also impacted by the regression introduced at this time by Platform/Resources !?
(In reply to comment #5) > The dates look similar to those I noticed in bug 206601 comment 3... Perhaps > your test is also impacted by the regression introduced at this time by > Platform/Resources !? There seams to be a pretty clear pattern: A lot of our test went from green to red between N20070909-0010 and N20071015-0010. It is very likely that it was something below us or in our infrastructure (maybe even the performance test infrastructure?) that got slower. I need more results to narrow down the point when it got slower. I guess I need to setup the testing framework on my machine and test the builds between N20070909-0010 and N20071015-0010. Oh boy, this may take a while... Frederic, is there any good documentation on how to do that?
(In reply to comment #6) > (In reply to comment #5) > > The dates look similar to those I noticed in bug 206601 comment 3... Perhaps > > your test is also impacted by the regression introduced at this time by > > Platform/Resources !? > > There seams to be a pretty clear pattern: A lot of our test went from green to > red between N20070909-0010 and N20071015-0010. It is very likely that it was > something below us or in our infrastructure (maybe even the performance test > infrastructure?) that got slower. I need more results to narrow down the point > when it got slower. I guess I need to setup the testing framework on my machine > and test the builds between N20070909-0010 and N20071015-0010. Oh boy, this may > take a while... > > Frederic, is there any good documentation on how to do that? > There's some documentation on Platform/Releng project page (http://www.eclipse.org/eclipse/platform-releng/index.html) but it may differ from the reality as it was written two years and half ago... By the way, I do not think you really need to setup the entire testing framework (espcially performance database storage). As the delta is quite big (30% - 40%), I think that analyzing the difference using yourkit should be enough. You just need to run the OrganizeImportsPerfTest test suite with the yourkit plug-in, take snapshots both in HEAD and perf_33x branch and look at the difference... Then, you should see who is responsible of the regression and finally understand it happened between 2007/09/09 and 2007/10/15. Note that I tried to revert the Platform/Resources change and run your test, but I didn't get the previous results. So, it seems that this is not the origin of the problem...
Created attachment 82681 [details] graph showing regression In order to track this down I run the performance tests 10 times on each nightly build between 0904 and 1105 on my dev machine.
Created attachment 82682 [details] graph showing findings Then I tried to compare 0909 and 0915 with the profiler, without any success. I could not find a difference which seems to be relevant. Then I took 0909 and started to add plugins from 0915 until "after a while" I could see the regression. Here are my findings: elapsed process average N20070909-0010 1015 N20070909-0010-plus 1253 N20070915-0010 1243 where N20070909-0010-plus is N20070909-0010 plus following plugins from 0915: org.eclipse.jface_3.3.0.N20070915-0010.jar org.eclipse.swt.win32.win32.x86_3.4.0.N20070915-0010.jar org.eclipse.swt_3.4.0.N20070915-0010.jar org.eclipse.ui.ide_3.3.100.N20070915-0010.jar org.eclipse.ui.workbench_3.3.0.N20070915-0010.jar org.eclipse.ui_3.3.0.N20070915-0010.jar removing any of this plugins makes the regression go away (or prevents the workbench from starting)
First of all, the test does not measure what it should i.e. Organize Imports. This is captured in bug 209611. This will be resolved once we've found out the regressions for the current version of the test.
Created attachment 82747 [details] Measurements This attached data has been captured by runnining the org.eclipse.test.uitestapplication with org.eclipse.jdt.ui.tests test plug-in and org.eclipse.jdt.ui.tests.performance.views.OrganizeImportsPerfTest test class on the corresponding build from the command line on a T43p WinXP. Benno has a dual core machine. On my machine the tests are quite unstable at least when looking at ELAPSED time and hence the regression is not so clear. When looking at CPU time the regression is more obvious and around 200ms. Just a guess: this might be due to different class loading (e.g. some plug-ins loaded later due to less aggressive plug-in activation by Platform UI).
Created attachment 82765 [details] vista results I can reproduce this even if I: - Delete the workspace before each test - Use the 0909 and 0915 performance tests instead of head - Use a 1.4.2_16 jre instead of 1.5.0_12 (the regression is smaller, but still there) - Use the default performance meter (instead of my one) - Run the tests on vista machine (see graph, it's a dual core as well)
Created attachment 82766 [details] test plugins for 0909
Created attachment 82767 [details] test plugins for 0915
Created attachment 82768 [details] script to run organize imports test (windows)
Moving to platform/ui. see comment 9 for the reason. I hope you have an explanation for this.
Benno when you say any of these plug-ins on comment 9 I assume you mean all of these plug-ins? Or does the addition of any one of them cause this issue?
(In reply to comment #17) > Benno when you say any of these plug-ins on comment 9 I assume you mean all of > these plug-ins? > > Or does the addition of any one of them cause this issue? > You need to add _all_ the plugins from comment 9 to see the regression.
In the ui.workbench and jface plugins, that time period includes: Bug 73872, which adds a part listener for ObjectPluginAction. They are datafilled when you actually pop up the context menu which calls setActivePart(*), and that takes special code to do so programmatically. Bug 136397 comment #48 in that time period includes changes that will not force DebugUIPlugin to start simply by opening a window. The MenuManager only updates when it is about to show, and windows no longer do udpateAll(true) so it doesn't happen when the window opens. Bug 202994 which now disposes all ActionContributionItems when they have dispose() called, when ACIs used to wait for their associated Menu to fire an SWT.Dispose. These all have the potential to change timing in tests, but they also seem to me to need the pre-condition of opening a menu somewhere. PW
Close as WONTFIX as this was on old versions issue...
>Close as WONTFIX as this was on old versions issue... We don't close all our bugs once the next release comes, do we? ;-) And hence just because it no longer appears red in the next release doesn't mean the problem is solved.
(In reply to comment #21) > >Close as WONTFIX as this was on old versions issue... > > We don't close all our bugs once the next release comes, do we? ;-) And hence > just because it no longer appears red in the next release doesn't mean the > problem is solved. > I didn't close it as FIXED but as WONTFIX... and for two main reasons: 1) there's nothing new on this since more than 16 months (which usually means that it has been garbaged - at least in developers mind...) 2) the problem is only visible while comparing 3.2 to 3.3. As we're now in 3.5, I frankly doubt that someone takes time to install an debug environment with these old versions to investigate this regression... So, I just thought that we could honestly admit that this problem will not be investigated in 3.5 and not adding this bug to the root bug 270824 I opened yesterday... Hence, WONTFIX appears to me to be best resolution for this bug, but I agree I should ask before doing it...
Yes, it doesn't need to go to the 3.5 effort but should stay open as the described performance degradation probably didn't go away.
(mass update) Removing target milestone, this bug was targeted at a milestone in the past.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.
