Bug 266886 - [perfs] Possible regression on 'Rename package 1000 CUs, 10 Refs' test
Summary: [perfs] Possible regression on 'Rename package 1000 CUs, 10 Refs' test
Status: VERIFIED WORKSFORME
Alias: None
Product: JDT
Classification: Eclipse Project
Component: UI (show other bugs)
Version: 3.5   Edit
Hardware: PC Linux-GTK
: P3 normal (vote)
Target Milestone: 3.5 M7   Edit
Assignee: Markus Keller CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance, test
Depends on:
Blocks: 270824
  Show dependency tree
 
Reported: 2009-03-03 11:38 EST by Frederic Fusier CLA
Modified: 2009-05-06 05:29 EDT (History)
2 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Fusier CLA 2009-03-03 11:38:43 EST 
The perf results verification for N20090228-2000 build show that a
regression may have occurred for RenamePackagePerfTests1#test_1000_10() 
scenario since N20090212-2000 build.

It's only clear on one machine (REHL 5.0), the test is really unstable on other machines, hence the regression cannot be confirmed.

Here are the numbers for REHL 5.0 machine since 3.5M5:
	builds		Rename package - 1000 CUs, 10 Refs
M4	I20081211-1908	-3.4%
M5	I20090202-1535	1.5%
	I20090203-1200	-1.4%
	N20090205-2000	-2.1%
	I20090210-0950	-0.4%
	N20090212-2000	-2.6%
	I20090217-2200	-3.7%
	N20090219-2000	-3.8%
	I20090224-0800	-3.4%
	N20090226-2000	-4.4%
	N20090228-2000	-6.4%

I decided to open a bug, even if it's not confirmed because the regression is over 400ms on this machine, hence may be now noticeable by users...
Comment 1 Frederic Fusier CLA 2009-03-10 14:13:18 EDT 
FYI, the same potential issue may also occurs for the scenario org.eclipse.jdt.ui.tests.refactoring.reorg.RenameMethodPerfTests1#test_1000_10()

The numbers are:
M4	I20081211-1908	-1.2%
M5	I20090202-1535	-3.9%
	I20090203-1200	-11.5%
	N20090205-2000	-9.0%
	I20090210-0950	-2.4%
	N20090212-2000	-1.8%
	I20090217-2200	-3.9%
	N20090219-2000	-2.8%
	I20090224-0800	-2.7%
	N20090226-2000	-4.6%
	N20090228-2000	-3.7%
	I20090303-0800	-3.1%
	N20090305-2000	-3.0%
	N20090307-2000	-3.7%
	I20090308-2000	-3.9%
	I20090309-0100	-9.0%

But I'm also not sure whether there's something real here or not and if it's related to the Rename Package test...
Comment 2 Frederic Fusier CLA 2009-03-10 14:44:41 EDT 
May be also a consequence of fix for bug 250454...?
Comment 3 Dani Megert CLA 2009-04-08 08:30:01 EDT 
Those tests runs only once and hence aren't stable enough. I've fixed this in HEAD and perf_34x.
Comment 4 Frederic Fusier CLA 2009-04-15 10:18:26 EDT 
I still see a similar regression looking at I20090414-0800 perf results. Note that one could see the changes done on this test in the last baseline...
Comment 5 Dani Megert CLA 2009-04-15 10:26:56 EDT 
Agree. We're still looking into it.
Comment 6 Dani Megert CLA 2009-04-16 11:04:16 EDT 
The interval in which the numbers are is so big that it is almost impossible to find out why there is a regression because many runs to generate a trace are faster on 3.5 than on 3.4. From what I see in my profiler the reason for this nondeterministic behavior is that once we start the refactoring of those 1000 files the Java indexer sometimes blocks files at the OS level which at the same time need to be accessed from the main thread. I suspect that the indexer does a little bit more work (e.g. due to constructor proposals) and maybe even touches more files and hence more potentials for short blocking.

Given this is a corner case (renaming 1000 cus at once) and that the seen regression is not too big we won't invest more time into this, however, I have removed the tests from the fingerprint.
Comment 7 Frederic Fusier CLA 2009-05-06 05:29:06 EDT 
Verified for M7 with I20090430-2300 results