Re: [cbi-dev] performance testing

[John Arthorne <John_Arthorne@xxxxxxxxxx>]

Great! If you have never looked at them
before, I suggest as a first step you download the tests and try running
them on your own machine. There are instructions here:

http://wiki.eclipse.org/Platform_UI/Performance_Tests

Once you can do that, you can take a
look at this master bug and its blockers for details on what work is going
on now:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=374441

The big piece that nobody is working
on yet is the infrastructure to gather the performance tests across several
builds into a single place. This will enable comparing performance results
to a baseline (previous release), and graphing results over time so performance
trends can be picked up. We had been using a Derby database for this in
the past. There is some background information on that here:

http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?view=co

That document is most likely out of
date. One concrete thing you can do to help right away is migrate that
document into a wiki page, so it will be easier to correct and update it
as we go. I have created a stub page for this:

http://wiki.eclipse.org/Performance/Automated_Tests

If you are interested in a more research-oriented
area, one problem we always had in the past was automating detection of
real regressions. From build to build there can be noise that makes individual
test runs not very definitive. It is only by seeing a longer term trend
across a few builds that we can be sure there is a real problem. When we
last had running performance tests, this was a manual job of a committer
reviewing the performance graphs for each test and looking for unusual
patterns. We don't really have the committer resources to do this anymore,
so we could really use some more automated tool that can point us at cases
of significant regressions from the mean across multiple runs.

I hope that gives you enough pointers
to get you started!

John

Sukwon Oh <sukwon0709@xxxxxxxxx>
Sent by: cbi-dev-bounces@xxxxxxxxxxx

09/30/2012 09:58 AM

Please respond to
Common-build Developers discussion <cbi-dev@xxxxxxxxxxx>

To	cbi-dev@xxxxxxxxxxx
cc
Subject	[cbi-dev] performance testing

Hi everyone,

I am one of the students from
UCOSP this term and I would like to work on performance testing along with
the Eclipse platform team.

Sukwon_______________________________________________
cbi-dev mailing list
cbi-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/cbi-dev