Description Frederic Fusier 2007-04-25 10:17:18 EDT

Fup of bug 184015.

Reversely, the second weakness of Student's t-test is when the deviation is big. When this is the case, there's a high chance that the t-test was valid even if the standard error of the test is high.

Of course, it's not a problem with the build where performance has been increased but problem will appear in the next version. We experienced this problem in JDT/Core. We didn't see that we have potential problems of confidence on our tests in 3.2 as we had great performance improvements. But when 3.3 perfs tests started, then lot of our tests become yellow... :-(

So, as for bug 184015, I suggest to temperate the t-test value with the standard error of the difference and signal a problem on test confidence when the error is high...

I use a separate bug for this weakness, as we may want to add the standard error information differently for it than for the weakness exposed in bug 184015. There, we just want not to paint in yellow the tests which have a small standard error (ie. which finally have a good confidence).

In current bug, we want to inform developper, that even if she can have a good confidence in the sign of the difference, the test confidence is not as so good as she can think...

For example we might signal this problem using the test's tooltip which could include the standard error value...