[platform-releng-dev] running bigger tests, was: Ideas for improving per

[Thomas L Roche <tlroche@xxxxxxxxxx>]

Ed Burnette Mon, 4 Apr 2005 10:37:27 -0400
> I'd also like to see bigger, more representative, tests.
> Micro-benchmarks like the amount of time it takes to draw 1000
> strings on the screen have a place but don't predict performance of
> the system as a whole (which is what the user experiences).

> What I have in mind is something like the way a hardware website
> such as anandtech.com evaluates notebooks, graphics cards or driver
> versions. You have a few big benchmarks (the Eclipse equivalent of
> Winstone, Doom 3, Mobile Mark, and SISoft Sandra), 

FWIW Rational Developer is planning to do something similar for
itself. We're currently developing a long "scenario" test (like
[create a project, open its web diagram, draw some blobs, invoke
wizards using those blobs, create artifacts using those wizards]) to
launch from a workbench (using JUnit Plugin Test), as we're porting
our build process to PDE. Presuming we can finish this scenario (we
currently have it drawing blobs and invoking the first wizard) and run
it under PDE's BVT (aka Automated Testing Framework), we plan to run
it as a performance test case, as described in

http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?rev=HEAD 


> and for each benchmark you have graphs that order the releases by
> the metrics you're interested in (for example, number of seconds,
> disk I/O, and memory usage).