Bug 269037 - [perfs] Regression on 'Incremental Build - All' test
Summary: [perfs] Regression on 'Incremental Build - All' test
Status: VERIFIED FIXED
Alias: None
Product: PDE
Classification: Eclipse Project
Component: API Tools (show other bugs)
Version: 3.5   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 3.5 M7   Edit
Assignee: Michael Rennie CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance, test
Depends on: 233643
Blocks: 270824
  Show dependency tree
 
Reported: 2009-03-17 13:32 EDT by Frederic Fusier CLA
Modified: 2009-05-06 05:40 EDT (History)
2 users (show)

See Also:

Attachments
fix for exceptions (1.25 KB, patch)
2009-03-30 23:36 EDT, Michael Rennie CLA 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Fusier CLA 2009-03-17 13:32:44 EDT 
Looking at 3.5M6 performance results, I can say now that there's a small regression on IncrementalBuildTests#testIncrementalBuildAll() test.

Here are the numbers:

Win XP Sun 1.5.0_10 (2 x 3.00GHz - 3GB RAM)
	N20090307-2000	-9.8%
	I20090308-2000	-8.6%
	I20090309-0100	-9.3%
	I20090310-0100	-7.2%
	I20090311-0100	-8.5%
	I20090312-0100	-7.6%
	I20090313-0100	-11.3%


SLED 10 Sun 1.5.0_10 (2 x 3.00GHz - 3GB RAM)
	I20090217-2200	-3.3%
	I20090224-0800	-5.3%
	N20090226-2000	-12.5%
	N20090228-2000	-5.1%
	I20090303-0800	-5.7%
	N20090305-2000	-6.2%
	N20090307-2000	-7.8%
	I20090308-2000	-9.7%
	I20090309-0100	-5.7%
	I20090310-0100	-8.9%
	I20090311-0100	-7.1%
	I20090312-0100	-14.9%
	I20090313-0100	-7.0%

RHEL 5.0 Sun 6.0_04 (2 x 3.00GHz - 3GB RAM)
	I20090217-2200	-3.5%
	N20090219-2000	-7.7%
	I20090224-0800	-3.1%
	N20090226-2000	-7.4%
	N20090228-2000	-7.4%
	I20090303-0800	-6.1%
	N20090305-2000	-8.2%
	N20090307-2000	-6.9%
	I20090308-2000	-5.6%
	I20090309-0100	-6.6%
	I20090310-0100	-8.5%
	I20090311-0100	-6.2%
	I20090312-0100	-7.6%
	I20090313-0100	-8.1%

Note that it makes only a difference less than 100ms...
Comment 1 Michael Rennie CLA 2009-03-20 09:26:40 EDT 
the regression can be accounted for in that we are doing a lot more work in builder than in the baseline release. 

We have added:

1. Java 5 support
2. unsupported tag detection
3. unused filter detection
4. system library use detection

with fixes to bugs 233643 and 231936 this regression should be fixed.
Comment 2 Michael Rennie CLA 2009-03-30 23:36:40 EDT 
Created attachment 130341 [details]
fix for exceptions

This patch fixes up the exceptions found in the console, which were caused by trying to add the Enum and Annotation tests to the test suite more than once...
Comment 3 Michael Rennie CLA 2009-03-31 09:18:09 EDT 
applied patch to HEAD
Comment 4 Darin Wright CLA 2009-04-22 10:03:11 EDT 
The tests are all gree in the latest build - I20090421-0930.

Marking fixed.
Comment 5 Frederic Fusier CLA 2009-04-29 05:30:54 EDT 
(In reply to comment #4)
> The tests are all gree in the latest build - I20090421-0930.
> 
> Marking fixed.
> 
The results seem to be negative again since build I20090426-2000...
Could you have a look again on this?
Thanks
Comment 6 Michael Rennie CLA 2009-04-29 09:49:22 EDT 
(In reply to comment #5)
> (In reply to comment #4)
> > The tests are all gree in the latest build - I20090421-0930.
> > 
> > Marking fixed.
> > 
> The results seem to be negative again since build I20090426-2000...
> Could you have a look again on this?
> Thanks
> 

I think Darin meant that the tests all pass without exception now. 

With the latest results from http://fullmoon.ottawa.ibm.com/downloads/drops/I20090428-0100/performance/org.eclipse.pde.api.tools.php?fp_type=0 we have made a significant improvement (especially on windows).
Comment 7 Frederic Fusier CLA 2009-05-06 05:40:48 EDT 
Verified for M7 using I20090430-2300 (the numbers are now around 50ms)