Community
Participate
Working Groups
Scenario: org.eclipse.core.tests.runtime.perf.StartupTest.testApplicationStartup shows now a significant regression on all boxes. Last results for I20080115-0800 build are: RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB): -5.9% Win XP Sun 1.4.2_10 (3 GHz 2 GB): -5.5% RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB): -7.7% Win XP Sun 1.4.2_10 (2 GHz 512 MB): -4.9% RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB): -133.4% (not reliable)
Around Nov 28, 2007 source bundles were changed to have one per bundle. This almost doubled the number of bundles in the SDK. The performance results do indicate a consistently higher startup time since around Nov 28th. Past performance analysis has indicated that each bundle installed in the system increases startup time by 1/2 - 1 ms. This corresponds the slower startup time for all the test machines except the machine with RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB). Here the CPU time is somewhat consistent but the overall Eclipse elapsed time is very sporadic. Perhaps the disk is very fragmented which is causing inconsistent IO? I thought there was another bug open to have either p2 or old update not install the source bundles.
bug 202464 is open against p2 to not install source bundles.
I opened bug 215491 against update to address handling source bundles.
I released a degradation comment for the core headless startup test (StartupTest.java) which points to this bug to explain the degradation.
fwiw this test is bogus because it depends on the number of bundles in the configuration and that varies over time. A real test would have a fixed and known data set.
The same is true for the UIStartupTest test. It is not as pronounced there because we are running a bunch of other code in the test which offset the 50-100 ms penalty from the extra source bundles.
We should have a test that measures the time to startup the Eclipse IDE. That is one of the things that end users care about. However, the core startup tests are not in that area. Here we really just care about bundle overheads etc. Having said that, we should still address this issue for 3.4
Frederic, could you please detail more on how the measurement is done? Is the test framework dropped into the plugins or dropins folder?
- With the latest p2 we should no longer be installing the source bundles which doubled the number of bundles in our test. Theoretically this should bring the numbers of the startup tests back down to reasonable levels. The test results still show a big slowdown - Bug 224177 was causing update.configurator to attempt to reconcile the installed bundles each startup. This maybe causing a big slowdown since it will always detect that the source bundles need to be installed after p2 determined they should be uninstlled.
(In reply to comment #8) > Frederic, could you please detail more on how the measurement is done? > Is the test framework dropped into the plugins or dropins folder? > I have no idea where the test framework is dropped to before running performance tests. Kim, may you give some clue here? Thanks
Regarding comment #10, the test framework and the test plugins are put in the dropins folder before running the perf tests.
The problem with installing the test fwk by putting it in the dropins is that it causes a lot of work to be done on startup (some metadata generation, followed by the reconciliation). This work is not typical of what a user would see when the installation is fresh out of the zip. Indeed this test is measuring the performance of the dropins behavior, which we know is in regression since 3.3 because of the new way of handling reconciliation. There has been some improvements but given the new architecture I don't think we will be as performant as we use to be. Now what I would be really interested in seeing is the difference in the *real* startup time of an unmodified (or at least with nothing dropped) 3.3 and unmodified 3.4, where normally we should see some improvements in 3.4 since less work as to happen.
I could be wrong but I thought that when we measured the time for startup, it was the second startup so we could avoid the time to read all the plugin.xml files to build the registry, etc? I believe that a measurement of the second time startup with p2 is really what we want.
DJ, you are correct. We should be running from a cached configuration to run the performance tests. We need to confirm that a form of bug 224177 is not cropping up in the performance tests. Perhaps they do not use the same setup as the other session tests do?
3.5 perf test results also shows a regression and the comment applied for 3.4 version on this test is still activated: "Performance decrease caused by the increase in the number of bundles. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=215464 for details." Is the comment still valid for the observed 3.5 regression? FYI, here are the numbers for 3.5 performance tests results: Win XP Sun 1.5.0_10 (2 x 3.00GHz - 3GB RAM) I20081118-1720 -5.0% I20081119-1600 -5.3% N20081120-2000 -5.0% I20081125-0840 -5.0% N20081127-2000 -5.0% N20081129-2000 -4.8% I20081202-1812 -23.0% N20081204-2000 -7.5% N20081206-2000 -23.7% I20081209-0100 -4.3% I20081210-0100 -3.9% I20081211-0100 -4.3% I20081211-1105 -6.3% M4 I20081211-1908 -4.8% N20081213-2000 -4.8% I20081216-0800 -3.6% N20081218-2000 -3.9% N20081220-2000 -5.5% N20090101-2000 -5.2% N20090103-2000 -5.5% I20090106-0800 -4.6% N20090108-2000 -5.5% N20090110-2000 -5.1% SLED 10 Sun 1.5.0_10 (2 x 3.00GHz - 3GB RAM) I20081118-1720 -4.5% I20081119-1600 -4.7% N20081120-2000 -5.0% I20081125-0840 -4.6% N20081127-2000 -4.3% N20081129-2000 -4.3% I20081202-1812 -4.9% N20081204-2000 -4.7% N20081206-2000 -5.1% I20081208-0921 -4.8% I20081209-0100 -4.9% I20081210-0100 -5.2% I20081211-0100 -6.3% I20081211-1105 -5.4% M4 I20081211-1908 -5.4% N20081213-2000 -5.7% I20081216-0800 -5.2% N20081218-2000 3.1% N20081220-2000 -17.4% N20081225-2000 -6.5% N20081227-2000 -7.0% N20090101-2000 -6.6% N20090103-2000 -6.6% I20090106-0800 -6.0% N20090108-2000 -6.4% N20090110-2000 -6.6% RHEL 5.0 Sun 6.0_04 (2 x 3.00GHz - 3GB RAM) I20081118-1720 -6.4% I20081119-1600 -5.6% N20081120-2000 -7.6% I20081125-0840 -5.5% N20081127-2000 -5.8% N20081129-2000 -6.7% I20081202-1812 -6.3% N20081204-2000 -5.7% N20081206-2000 -5.1% I20081208-0921 -6.4% I20081209-0100 -5.9% I20081210-0100 -6.1% I20081211-0100 -7.7% I20081211-1105 -14.8% M4 I20081211-1908 -6.3% N20081213-2000 -6.2% I20081216-0800 -5.9% N20081218-2000 -6.0% N20081220-2000 -7.1% N20081225-2000 -6.2% N20081227-2000 -4.2% N20090101-2000 -2.7% N20090103-2000 -6.8% I20090106-0800 -8.4% N20090108-2000 -23.1% N20090110-2000 -5.0%
I think so. We increased the number of bundles in both releases: 3.3->3.4 - increased by 19 bundles 3.4->3.5 - increased by 13 bundles So I would expect some slowdown to occur in this testcase when comparing 3.5 with 3.4.
Tom, I think the new regression should be closer investigated because it is much bigger from 3.4 to 3.5 than it was from 3.3 to 3.4 and according to your investigation we added less bundles this time. Core startup 3.3 to 3.4: 3.1 - 7.5%, 44 - 104 ms slower Core startup 3.4 to 3.5: 33.6 - 50.1%, 371 - 590 ms slower In an old comment I see: >Past performance analysis has indicated that each bundle installed in the >system increases startup time by 1/2 - 1 ms This seems no longer be the case i.e. adding bundles seem to be more expensive. Do we have numbers whether this cost is O(n) or more, or less? This would also be interesting for products sitting on top of us.
I agree with Dani that there is a recent regression that is different from the general degradation over time from increased number of bundles, etc. I have entered bug 270815 for this.
(set target milestone and priority on wrong bug)
(In reply to comment #17) > In an old comment I see: > >Past performance analysis has indicated that each bundle installed in the > >system increases startup time by 1/2 - 1 ms > This seems no longer be the case i.e. adding bundles seem to be more expensive. > Do we have numbers whether this cost is O(n) or more, or less? This would also > be interesting for products sitting on top of us. > It should be O(n). I did some testing with installing many bundles and the numbers are identical when launching from a cached state for both 3.4.2 and 3.5 M6: On my WinXP laptop I have the following numbers: bundles init time ---------------------------- 10 430 ms 5000 1000 ms 10000 1600 ms This is only measuring the overhead of the framework to reify the installed bundles from a persistent cached state. No additional code is actually run in any of the installed bundles in this measurement. The numbers above suggest that there is an overhead of ~400 ms just to get an empty framework. If we subtract that from the numbers then we have the following: bundles init time --------------------------------------- 10 430 - 400 = 30ms 5000 1000 - 400 = 600ms 10000 1600 - 400 = 1200ms The rate of growth in relation to the number of bundles is linear and is around 0.12 ms on my laptop. I am convinced the slowdown is not because of changes in the framework since these numbers are identical between 3.4 and 3.5 M6.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
At this point I am closing as worksforme. The framework has greatly changed from when the discussion happened in this bug. Any future investigation in startup performance will need to be measured against the latest.
