Community
Participate
Working Groups
Using performance results on N20070422-0010 and I20070424-0930. While working on bug 183338, I've extracted performance results from the releng database to understand why plenty of JTD/Core tests were yellow. I computed the Student's t-test on our tests locally using the extracted data and saw differences for some tests. One of this difference is on scenario: org.eclipse.jdt.core.tests.performance.FullSourceWorkspaceModelTests#testPerfReconcileBigFileWithSyntaxError() (Win XP Sun 1.4.2_10 (3 GHz 2 GB)) If you look closely at standard deviation, you'll see that it's 0 for the measures on N20070422-0010. 0 is a strange value but theoretically possible, so I debugged the extraction to see the value stored in the database and see that it was 0x7ff8000000000000L! When converting this extracted value to double, we got a Double.NaN and this explains why the t-test value is OK when you look at method StatisticsUtil#hasSignificantDifference(...). Debugging further, it appears that this wrong stored value comes from a problem while computing the standard deviation from measurements in the performance framework in method StatisticsSession#computeStatsFromMeasurements(Dim, Set) line 210: stats.stddev= Math.sqrt(squaredDeviations / stats.count - 1); With this invalid formula, standard deviation may be negative if stats.count > squaredDeviations and results with a NaN as specified in Math.sqrt javadoc: * @return the positive square root of <code>a</code>. * If the argument is NaN or less than zero, the result is NaN.
The trivial fix is to modify the formula as follow: stats.stddev= Math.sqrt(squaredDeviations / (stats.count - 1));
Created attachment 64855 [details] Proposed patch This patch adds the necessary protection when there's only one measure...
Created attachment 64856 [details] Correct proposed patch Sorry, previous patch was bogus...
Thanks, patch applied for tonight's build.
Patch also applied to maintenance branch for baseline tests.
Reopen to assign
.
