Community
Participate
Working Groups
Looking at the Scenario Status Table of the JDT/UI component performance tests, it appears that most of the tests have only one run (eg. standard error = '[n/a]'). This has a strong impact on the tests reliability and make the verification hard to be accurate. The global result is that no regression was noticed on this component although the fingerprints looks not so good; a large number of tests in the fingerprints are red or yellow. IMO, this may give to users a negative feeling about the global performance of this component although there's no real performance issue...
All these tests fall under 2 categories: - cold tests, i.e. tests to verify that we're not unacceptably slow the *first* time we perform an operation - very short tests for high-level operations like refactorings, which would need a big number of repetitions (often > 100) to become reliable For both of these categories, our main interest is to ensure that the scenarios stay in a reasonable absolute range. A 20% difference in the 100ms region is not a problem for the user in these scenarios. I believe that the jitter is mostly due to unpredictable disk I/O. Best would be if we could just change these tests to only assert that we stayed in a given bound (disregarding small absolute jumps in execution time). This is bug 89804.
I suggest we change the tests to assert absolute values and revive bug 89804. Frédéric, can you take a look at this?
I've removed the OrganizeImportsPerfTest#testOrganizeImport() from the summary for now, since it usually takes less than 100ms and is thus not accurately measurable. We don't plan to do anything else here, unless the issues in bug 89804 get addressed (which could also be done in 3.6, IMO).
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're closing this bug. If you have further information on the current state of the bug, please add it and reopen this bug. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
