Re: [platform-releng-dev] Status of performance tests?

["David M Williams" <david_williams@xxxxxxxxxx>]

The status is "very slow progress".
Beside the umbrella bug you mentioned, the most "active" area
of my current work is captured in bug 481272. 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=481272.

It turns out the "analyze and display"
code was written with a great many hard-coded assumptions, most of them
very subtle, not only about the "data and keys" expected, but
even the "procedure" followed! And, from appearances, the code
was written by a number of people over a long period of time, so has ended
up being very fragile. (That is, a small change in one part of the code
has completely unexpected results in another part of the code due to side
effects). 

And, unfortunately, fixing it (or even
making progress) is the sort of task that takes "full time attention"
for a number of days in a row. Something that is hard for me (and I suspect
probably anyone) to come by in these days and times with so many jobs to
juggle. 

But it is not all doom and gloom. I
have made a fair amount of progress, some of which is not apparent yet
because it is not finished for the simple reason that I was distracted
by other "top priority" tasks. 

All of that is to say I do not think
it is the right time for people to *add* tests. If anything, some still
need to be removed, or fixed. Some of the "missing data" you
see is likely a result of the incorrect "analyze and display"
code, not the tests themselves.  As far as I know, very few have been
"disabled" (I know of only one). And, by all means, if anyone
is itching to write new performance tests, feel free to, but you may have
to do your own "comparisons". Pretty "old school",
eh? :) 

I am not sure what to say about the
standard error size. As far as I know, the tests have always had large
standard deviations. And, some of what you see may even be due to the incorrect
"analyze and display" code, so I would not jump to any conclusions
at this point in time about any of the "statistics". 

I would expect more progress early next
year. 

I will emphasize, though, these tests
are purely "regression tests" against the previous reference
release (currently 4.5.1).  Perhaps that is what you meant by "performance
related bugs"? But I just
wanted to remind everyone that the tests do not help "improve performance"
in general. Instead, they do -- as you also imply -- help find regressions
in areas of code that is tested. 

If anyone does write "new tests"
I will also remind everyone that we run our "current tests" against
the current builds and the reference builds. In other words, if you write
a new test, it must be able to run on both, current builds, and the reference
builds without re-compiling, etc. I suspect that's obvious, but I was not
sure, so thought I would mention it. 

I am also not sure if I have answered
your specific questions well enough, Stephan, so feel free to keep asking,
if I have not -- or, even, ask new ones if they come to mind! :) 

Thanks, 
HTH





From:      
 Stephan Herrmann <stephan.herrmann@xxxxxxxxx>
To:      
 platform-releng-dev@xxxxxxxxxxx,
Date:      
 12/15/2015 05:27 AM
Subject:    
   [platform-releng-dev]
Status of performance tests?
Sent by:    
   platform-releng-dev-bounces@xxxxxxxxxxx

---

Hi,

We've received some performance related bugs lately,
which let me search for performance test results,
that could help to narrow down potential culprits,
or ideally even actively track problems before we
get bug reports.

What is the status of the releng work in this area?

It seems performance tests are running and reports
are generated, but the selection of enabled tests
still looks like a straw man. In particular clicking
   Detailed performance data grouped by scenario prefix
   - org.eclipse.core*
yields no results - for many of the builds (!).
I found a few builds with results in this group,
but still the selection of test cases looks weak.

Is it time for the individual teams to inspect their
performance tests and re-enable them?

The last communication I could find in this matter
dates Nov. 2014, linking to the umbrella bug
https://bugs.eclipse.org/454921which, however,
doesn't speak about the actual content of the
performance test suite.

Looking at existing results we also seem to have
a problem with big standard errors. Is this to be
expected according to experience from the old days,
or is it possible that infra structure changes
could better calibrate tests to show lower standard
error? Or just a matter of insufficient amount of
data to average over?

thanks,
Stephan
_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://dev.eclipse.org/mailman/listinfo/platform-releng-dev