[platform-releng-dev] Status of performance tests?

[Stephan Herrmann <stephan.herrmann@xxxxxxxxx>]

Hi,

We've received some performance related bugs lately,
which let me search for performance test results,
that could help to narrow down potential culprits,
or ideally even actively track problems before we
get bug reports.

What is the status of the releng work in this area?

It seems performance tests are running and reports
are generated, but the selection of enabled tests
still looks like a straw man. In particular clicking
  Detailed performance data grouped by scenario prefix
  - org.eclipse.core*
yields no results - for many of the builds (!).
I found a few builds with results in this group,
but still the selection of test cases looks weak.

Is it time for the individual teams to inspect their
performance tests and re-enable them?

The last communication I could find in this matter
dates Nov. 2014, linking to the umbrella bug
https://bugs.eclipse.org/454921 which, however,
doesn't speak about the actual content of the
performance test suite.

Looking at existing results we also seem to have
a problem with big standard errors. Is this to be
expected according to experience from the old days,
or is it possible that infra structure changes
could better calibrate tests to show lower standard
error? Or just a matter of insufficient amount of
data to average over?

thanks,
Stephan