Bug 198957

Summary:	[results] Performance framework average and standard deviation computation is wrong when several commits are done for a same scenario
Product:	[Eclipse Project] Platform	Reporter:	Frederic Fusier <frederic_fusier>
Component:	Releng	Assignee:	Frederic Fusier <frederic_fusier>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	benno.baumgartner, daniel_megert, jerome_lanneluc
Version:	3.3	Keywords:	performance, test
Target Milestone:	3.4 M3
Hardware:	PC
OS:	Windows XP
Whiteboard:
Bug Depends on:	201469
Bug Blocks:

Description Frederic Fusier

2007-08-06 12:40:42 EDT

Looking at StatisticsSession history, it seems to have been introduced in 3.2 with patch of bug 127264.

When several results have been committed for one scenario in a same build, statistics (ie. average, standard deviation and count) are not correctly computed while getting numbers from the releng database.

Note that this problem was already known while writing the code as a comment clearly indicates that current implementation was not correct: see StatisticsSession#computeStatsFromAggregates(Dim) at line 160. This is obvious for standard deviation but also for average: the average of the averages is not equals to the average of all the measures...

See bug 198894 et bug 198895 opened about scenarii currently concerned...

Comment 1 Frederic Fusier

2007-08-28 12:27:34 EDT

See also bug 201320 for some org.eclipse.core tests concerned by this issue...

Comment 2 Frederic Fusier

2007-08-29 06:40:46 EDT

Patch for bug 210469 includes fix for this bug...

Note that even if framework will be able to handle this kind of storage, I would still advice to avoid it if possible as it multiply the numbers stored in the database unnecessarily and so slow down applications which may want to access these numbers.

Comment 3 Frederic Fusier

2007-10-19 11:11:40 EDT

Released for 3.4M3 in HEAD stream (while releasing patch for bug 201469)