Bug 509922 - Performance tests are giving invalid test results
Summary: Performance tests are giving invalid test results
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 4.7   Edit
Hardware: PC Windows 7
: P1 major (vote)
Target Milestone: 4.8   Edit
Assignee: Sravan Kumar Lakkimsetti CLA
QA Contact:
URL:
Whiteboard:
Keywords: test
Depends on: 517859
Blocks:
  Show dependency tree
 
Reported: 2017-01-04 06:41 EST by Sravan Kumar Lakkimsetti CLA
Modified: 2018-05-30 04:45 EDT (History)
4 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sravan Kumar Lakkimsetti CLA 2017-01-04 06:41:44 EST 
While going through performance tests report I noticed a lot of test failures while running on test baseline. This is causing invalid baselines. 

Did some preliminary analysis on this found we are running latest test suite on baseline eclipse. I looked at jdt core and found many Noclassdef found errors. These classes have been introduced only in 4.7 so they cannot run 4.6.x(baseline). 

Need to tweak performance tests to use baseline test suite on baseline eclipse and latest test suite on latest eclipse. We should not mix these
Comment 1 Dani Megert CLA 2017-05-15 08:52:57 EDT 
Not sure whether it is this bug that causes it, but we do not have any performance tests after M4.

This needs to get fixed for RC1.
Comment 2 Sravan Kumar Lakkimsetti CLA 2017-05-18 06:02:13 EDT 
(In reply to Dani Megert from comment #1)
> Not sure whether it is this bug that causes it, but we do not have any
> performance tests after M4.
> 
> This needs to get fixed for RC1.

The tests are running. During the generation of webpage there is a problem in fetching results from derby database. Still investigating. Moving to RC2
Comment 3 Eclipse Genie CLA 2017-05-24 06:47:58 EDT 
New Gerrit change created: https://git.eclipse.org/r/97850
Comment 5 Eclipse Genie CLA 2017-05-25 04:51:27 EDT 
New Gerrit change created: https://git.eclipse.org/r/97942
Comment 6 Eclipse Genie CLA 2017-05-25 04:51:29 EDT 
New Gerrit change created: https://git.eclipse.org/r/97943
Comment 9 Sravan Kumar Lakkimsetti CLA 2017-05-25 06:39:57 EDT 
(In reply to Eclipse Genie from comment #8)
> Gerrit change https://git.eclipse.org/r/97943 was merged to [master].
> Commit:
> http://git.eclipse.org/c/platform/eclipse.platform.releng.buildtools.git/
> commit/?id=ecc51559e26a9a07372258890f56478f37be6a6c

Performance results are getting published now. I am retargetting this to 4.8 for the correction of baselines.
Comment 10 Eclipse Genie CLA 2017-05-30 07:21:03 EDT 
New Gerrit change created: https://git.eclipse.org/r/98198
Comment 12 Sravan Kumar Lakkimsetti CLA 2017-05-31 07:05:54 EDT 
The baseline is also corrected now. As part of baseline the tests run with 4.6. 

The test results from the current build gets compared with the baseline
Comment 13 Eclipse Genie CLA 2017-06-02 02:39:32 EDT 
New Gerrit change created: https://git.eclipse.org/r/98488
Comment 15 Sravan Kumar Lakkimsetti CLA 2017-06-06 08:38:46 EDT 
We are having trouble with the test machine. Raised bug 517859 to get a new machine.

Moving to 4.8.

I will remove the performance tests line from download pages as it is not good for displaying wrong results.
Comment 17 Sravan Kumar Lakkimsetti CLA 2018-05-30 04:45:48 EDT 
Reenabled performance results from I20180529-0800.