Bug 217127 - Regression in performance test PerfContextualLaunchMenu#testContextualLaunchMenu()
Summary: Regression in performance test PerfContextualLaunchMenu#testContextualLaunchM...
Status: CLOSED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Debug (show other bugs)
Version: 3.4   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 3.4 RC1   Edit
Assignee: JDT-Debug-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance, test
Depends on:
Blocks:
 
Reported: 2008-01-30 12:42 EST by Frederic Fusier CLA
Modified: 2008-12-08 15:16 EST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Fusier CLA 2008-01-30 12:42:21 EST
Looking closely at Scenario Status Table of JDT/Debug component, I saw that the scenario testContextualLaunchMenu() of org.eclipse.jdt.debug.tests.performance.PerfContextualLaunchMenu
got a noticeable regression since build I20070925-0800.

Here are the numbers:
RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB): -118.6% [±13.3]
Win XP Sun 1.4.2_10 (3 GHz 2 GB):      -45.1% [±10.0]
RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB):    -30.6% [±6.6]
Win XP Sun 1.4.2_10 (2 GHz 512 MB):   -121.7% [±39.6]
RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB):  -21.7% [±17.9]

Even if the standard error is quite big, the results are still negative using the optimistic value (results+|stderr|)...

Sorry for reporting this so late, but I missed it as this was not a fingerprint test...
Comment 1 Darin Wright CLA 2008-01-30 16:33:44 EST
Investigate for 3.4
Comment 2 Darin Wright CLA 2008-03-31 14:31:04 EDT
Marking as M7 (although, we do need preformance tests to run again :-)
Comment 3 Curtis Windatt CLA 2008-04-25 16:25:14 EDT
Performance was degraded after we fixed Bug 181204 (contextual launching of non-resource based objects).  We added the following code to ContextualLaunchAction.fillMenu():

in 3.4 we are correctly passing the IEditorPart and ISelection, so we have to perform 
some sneekyness to make sure the IEditorInput is passed to the eval expressions
for backwards compatibility
Object o = ss.getFirstElement();
if(o instanceof IEditorPart) {
   selection.set(0, ((IEditorPart)o).getEditorInput());
}

When I took out this code and reran the test, the fillMenu() method took about half the time.
Comment 4 Darin Wright CLA 2008-05-09 17:26:31 EDT
Marking as fixed.

The problem is that this test was improved 10x during 3.3 (vs 3.2). The test went from @500ms to @50ms. This test has become so much faster that it is unreliable. In 3.3, we could fill the context menu 40 times in 61ms. In 3.4, we can fill the context menu 40 times 81ms. This is not a real degradation - the menu is less than 1ms slower to fill.

I have added a comment to the test, and we will replace or remove the test in 3.5.
Comment 5 Darin Wright CLA 2008-12-08 15:16:17 EST
Closing.