| Re: [platform-releng-dev] Dimensions with unusable statistical properties |
I am to blame.I noticed that some of our performance tests were not robust enough to get any meaningful result from them. That is, some green and red bars on the performance pages really didn't contain any useful information about a test but that it is highly unstable; any result might have occurred by chance, due to sampling fluctuation.
So I included code into the evaluation of performance data that may flag a test as unstable. This is what you are seeing.
A test is flagged:- When the test was just run once, as then there is no measure of the stability of the test (standard deviation) and one cannot infer anything about the real life average of the sampled dimension.
- When the test, given its standard error, would require a sample size > 1000 to be able to measure a 5% deviation from the mean. The standard error is high when there are large differences between the different samples taken. This usually means that a test is very unstable.
Note that also the performance output on the download pages was modified: tests that measure a statistically insignificant deviation from the baseline measurement are grayed out.
Do you think your tests are wrongly flagged? Or is just the output message unclear? Or is the above a bad idea? Please comment (or correct my limited understanding of statistics).
-tom See also:https://bugs.eclipse.org/bugs/show_bug.cgi?id=127264 - performance testing: add significance information to performance graphs https://bugs.eclipse.org/bugs/show_bug.cgi?id=126358 - performance: small update to the performance plug-in
Wassim Melhem wrote:
About two weeks ago, I got the latest org.eclipse.test.performance from HEAD and since then I have not been able to get any numbers when I run the PDE/UI performance tests in my workspace. I keep getting messages like this one spit out to the console: "Dimensions with unusable statistical properties: Used Java Heap, Working Set, Committed, Working Set Peak, Elapsed Process, Kernel time, Page Faults, CPU Time, GDI Objects" What makes the statistical properties of a dimension unusable? The same tests were fine before, so why now? Thanks... Wassim. _______________________________________________ platform-releng-dev mailing list platform-releng-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/platform-releng-dev