Bug 198957 - [results] Performance framework average and standard deviation computation is wrong when several commits are done for a same scenario
Summary: [results] Performance framework average and standard deviation computation is...
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 3.3   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 3.4 M3   Edit
Assignee: Frederic Fusier CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance, test
Depends on: 201469
Blocks:
  Show dependency tree
 
Reported: 2007-08-06 12:40 EDT by Frederic Fusier CLA
Modified: 2008-09-19 04:24 EDT (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Frederic Fusier CLA 2007-08-06 12:40:42 EDT 
Looking at StatisticsSession history, it seems to have been introduced in 3.2 with patch of bug 127264.

When several results have been committed for one scenario in a same build, statistics (ie. average, standard deviation and count) are not correctly computed while getting numbers from the releng database.

Note that this problem was already known while writing the code as a comment clearly indicates that current implementation was not correct: see StatisticsSession#computeStatsFromAggregates(Dim) at line 160. This is obvious for standard deviation but also for average: the average of the averages is not equals to the average of all the measures...

See bug 198894 et bug 198895 opened about scenarii currently concerned...
Comment 1 Frederic Fusier CLA 2007-08-28 12:27:34 EDT 
See also bug 201320 for some org.eclipse.core tests concerned by this issue...
Comment 2 Frederic Fusier CLA 2007-08-29 06:40:46 EDT 
Patch for bug 210469 includes fix for this bug...

Note that even if framework will be able to handle this kind of storage, I would still advice to avoid it if possible as it multiply the numbers stored in the database unnecessarily and so slow down applications which may want to access these numbers.
Comment 3 Frederic Fusier CLA 2007-10-19 11:11:40 EDT 
Released for 3.4M3 in HEAD stream (while releasing patch for bug 201469)