Bug 334757 - [perfs] Performance regression in WorkflowTests#testBigWorkflowNoUI()Tag1
Summary: [perfs] Performance regression in WorkflowTests#testBigWorkflowNoUI()Tag1
Status: RESOLVED WORKSFORME
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Team (show other bugs)
Version: 3.7   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Platform Team Inbox CLA
QA Contact:
URL: http://download.eclipse.org/eclipse/d...
Whiteboard:
Keywords: performance, test
Depends on:
Blocks: 313891
  Show dependency tree
 
Reported: 2011-01-19 06:20 EST by Satyam Kandula CLA
Modified: 2011-03-22 06:52 EDT (History)
2 users (show)

See Also:

Attachments
Full log (52.17 KB, text/plain)
2011-02-02 09:40 EST, Tomasz Zarna CLA 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Satyam Kandula CLA 2011-01-19 06:20:57 EST 
There is a 50-100% regression in the test org.eclipse.team.tests.ccvs.ui.benchmark.WorkflowTests#testBigWorkflowNoUI()Tag1. The regression seems to have got introduced between 11th and 13th January builds.
Comment 1 Tomasz Zarna CLA 2011-01-20 06:10:20 EST 
It looks bad indeed. I can see the regression in N20110115-2000 and I20110118-0800. What's interesting, on the results page for I20110118-0800[3] you can see that I20110114-1330[2], which had no contribution from the team, already had the regression. I20110111-0800[1] looks fine. I will keep an eye on the next perf tests results.

Workspace contribution:
[1] Bug 315694 and Bug 333812
[2] nothing
[3] Bug 294925 and Bug 332728
Comment 2 Satyam Kandula CLA 2011-01-26 01:53:30 EST 
There is also around 25% performance regression in org.eclipse.team.tests.ccvs.ui.benchmark.SyncTests#testSync100NoUI()ModifiedFiles on both RHEL and SLED machines. There is also 25% regression in SyncTests#testSync100NoUI()RemovedFiles on RHEL -- this test is good on SLED. 
This regression also seems to have got introduced between 11th and 13th Jan.
Comment 3 Tomasz Zarna CLA 2011-01-28 07:12:16 EST 
Examining perf results from http://fullmoon.ottawa.ibm.com/downloads/drops/I20110126-1800/performance/eplnx1/Scenario511.html I can see that WorkflowTests#testBigWorkflowNoUI()Tag1 results were pretty good for I20110124-1345 and I20110125-2012 but then it peaked again. Satyam, do you have any idea why did this happen?
Comment 4 Tomasz Zarna CLA 2011-02-02 09:40:35 EST 
Created attachment 188145 [details]
Full log

I've run the perf tests locally for I20101208-1300 (3.7M4) and I20110127-2034 (3.7M5). Here is an excerpt for the scenarios for which a regression was indicated on the server:

== ModifiedFiles (3.7M4 vs 3.7M5)
Elapsed Process:        6.18s vs 5.92s
CPU Time:                1.2s  vs 1.04s

== Tag1
Elapsed Process:        12.3s vs 15s
CPU Time:                490ms vs 425ms

Only Elapsed Process metric for Tag1 showed that the scenario ran longer (ca -21%?). Full log attached.
Comment 5 Tomasz Zarna CLA 2011-02-10 11:34:56 EST 
Last week the results for org.eclipse.team.tests.ccvs.ui.benchmark.WorkflowTests#testBigWorkflowNoUI()Tag1 skyrocketed to -314.7 %. Kim do you have any idea why this may be happening? Is it us or the server?
Comment 6 Tomasz Zarna CLA 2011-03-22 06:52:45 EDT 
The results are back to normal. They have been stable for the last couple of builds[1][2]. The only problem remaining is the fact that baseline hasn't been measured for the long time - filed bug 340639 for that.

[1] http://download.eclipse.org/eclipse/downloads/drops/I20110315-0800/performance/epwin2/Scenario511.html
[2] http://download.eclipse.org/eclipse/downloads/drops/I20110315-0800/performance/epwin3/Scenario511.html