Bug 217127

Summary: Regression in performance test PerfContextualLaunchMenu#testContextualLaunchMenu()
Product: [Eclipse Project] JDT Reporter: Frederic Fusier <frederic_fusier>
Component: DebugAssignee: JDT-Debug-Inbox <jdt-debug-inbox>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: curtis.windatt.public, darin.eclipse
Version: 3.4Keywords: performance, test
Target Milestone: 3.4 RC1   
Hardware: PC   
OS: Windows XP   
Whiteboard:

Description Frederic Fusier CLA 2008-01-30 12:42:21 EST 
Looking closely at Scenario Status Table of JDT/Debug component, I saw that the scenario testContextualLaunchMenu() of org.eclipse.jdt.debug.tests.performance.PerfContextualLaunchMenu
got a noticeable regression since build I20070925-0800.

Here are the numbers:
RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB): -118.6% [±13.3]
Win XP Sun 1.4.2_10 (3 GHz 2 GB):      -45.1% [±10.0]
RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB):    -30.6% [±6.6]
Win XP Sun 1.4.2_10 (2 GHz 512 MB):   -121.7% [±39.6]
RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB):  -21.7% [±17.9]

Even if the standard error is quite big, the results are still negative using the optimistic value (results+|stderr|)...

Sorry for reporting this so late, but I missed it as this was not a fingerprint test...
Comment 1 Darin Wright CLA 2008-01-30 16:33:44 EST 
Investigate for 3.4
Comment 2 Darin Wright CLA 2008-03-31 14:31:04 EDT 
Marking as M7 (although, we do need preformance tests to run again :-)
Comment 3 Curtis Windatt CLA 2008-04-25 16:25:14 EDT 
Performance was degraded after we fixed Bug 181204 (contextual launching of non-resource based objects).  We added the following code to ContextualLaunchAction.fillMenu():

in 3.4 we are correctly passing the IEditorPart and ISelection, so we have to perform 
some sneekyness to make sure the IEditorInput is passed to the eval expressions
for backwards compatibility
Object o = ss.getFirstElement();
if(o instanceof IEditorPart) {
   selection.set(0, ((IEditorPart)o).getEditorInput());
}

When I took out this code and reran the test, the fillMenu() method took about half the time.
Comment 4 Darin Wright CLA 2008-05-09 17:26:31 EDT 
Marking as fixed.

The problem is that this test was improved 10x during 3.3 (vs 3.2). The test went from @500ms to @50ms. This test has become so much faster that it is unreliable. In 3.3, we could fill the context menu 40 times in 61ms. In 3.4, we can fill the context menu 40 times 81ms. This is not a real degradation - the menu is less than 1ms slower to fill.

I have added a comment to the test, and we will replace or remove the test in 3.5.
Comment 5 Darin Wright CLA 2008-12-08 15:16:17 EST 
Closing.