[platform-releng-dev] Ideas for improving performance reports

["Ed Burnette" <Ed.Burnette@xxxxxxx>]

I have a comment and suggestion about the performance results. I don't find the percentage deltas very useful without knowing the absolute value. To get that I have to drill down on each item in the chart. For example, a 10% improvement on something that takes 15 seconds is much better than a 10% improvement on something that takes 85ms.

I'd also like to see bigger, more representative, tests. Micro-benchmarks like the amount of time it takes to draw 1000 strings on the screen have a place but don't predict performance of the system as a whole (which is what the user experiences).

What I have in mind is something like the way a hardware website such as anandtech.com evaluates notebooks, graphics cards or driver versions. You have a few big benchmarks (the Eclipse equivalent of Winstone, Doom 3, Mobile Mark, and SISoft Sandra), and for each benchmark you have graphs that order the releases by the metrics you're interested in (for example, number of seconds, disk I/O, and memory usage). These can be graphed in a number of ways.

Here are some examples: 

http://www.anandtech.com/video/showdoc.aspx?i=2300&p=4
http://www.anandtech.com/video/showdoc.aspx?i=1900&p=18
http://www.anandtech.com/IT/showdoc.aspx?i=2347&p=8
http://www.anandtech.com/mobile/showdoc.aspx?i=2225&p=15
http://www.anandtech.com/mobile/showdoc.aspx?i=2225&p=16
http://www.anandtech.com/mobile/showdoc.aspx?i=2225&p=17