Community
Participate
Working Groups
Something for 3.4 (likely) As an enhancement, it would be good to render the standard error in fingerprints. Currently, beyond a threshold, the graph bar is drawn in yellow. Which means a bad regression may actually be drawn in yellow, rather than in red, which is weakening the perception of the regression. Same for a big improvement. I would propose that instead of rendering all in green/yellow/red, we rather simply render the bar in green/red based on current result. Then the last portion of the bar would be coloured in yellow for the area affected by the standard error. Let's take an example: Result = +70% (+/- 10%) If so, render: [0]--------------------[60]--------[70]--------[80] <--------60%green----->][10%yellow][^^][10% yellow] i.e. the bar is mostly green, even though the standard error is quite high, since the gain is big enough. Result = -10% (+/- 30%) [-40]------[-10]----------[20] [30%yellow][^^][30% yellow] i.e. the bar is entirely yellow, since the error is too high to tell it is wrong for sure.
Probably to make visual effect even better, using transparency would be even better. In latter example, drawing some semi-transparent yellow over underlying red bar would be the best... unclear this can be achieved.
Transparency seems to be reserved for icon, but I didn't work on it a long time. I will simply implement the comment 0 requirement as describe for the former example. The latter will be simply the value entirely yellow (10% in the example). We'll see how to draw fingerprints in an even better way in another bug...
Created attachment 77257 [details] org.eclipse.core results html page with new version Here's an example for build I20070821-0800 for new fingerprints look with the bug 210469 patch (it also includes fix for this bug)... You can compare it with its current version: http://download.eclipse.org/eclipse/downloads/drops/I20070821-0800/performance/org.eclipse.core.php? Full yellow bars are those where abs(value) < error. Otherwise, uncertainty zone is shown in yellow around the value only when the error is over the 3% threshold. It's also not displayed when value is over 100%.
Created attachment 77486 [details] Proposed patch This patch can generate any of the gif files of previous attached zip file just by changing one option value in the code... Note that it will need to be updated when patch for bug 201469 is released.
(In reply to comment #4) > Created an attachment (id=77486) [details] > Proposed patch > > This patch can generate any of the gif files of previous attached zip file just > by changing one option value in the code... Note that it will need to be > updated when patch for bug 201469 is released. > I was talking about bug 201920 attached zip file as this patch also fixes that bug...
Please adjust the target milestone, so it does not point at a closed milestone in the past.
I find the concept of recording standard deviation across different builds is confusing. If a test has a bad regression, and then it is fixed, the test ends up with a very high standard deviation, even if the test is very consistent across multiple runs on the same build. On the other hand measuring standard deviation makes sense for the baselines because it is the same build being tested multiple times, so a high deviation means the test itself is not consistent.
(In reply to comment #7) > I find the concept of recording standard deviation across different builds is > confusing. If a test has a bad regression, and then it is fixed, the test ends > up with a very high standard deviation, even if the test is very consistent > across multiple runs on the same build. On the other hand measuring standard > deviation makes sense for the baselines because it is the same build being > tested multiple times, so a high deviation means the test itself is not > consistent. > The standard deviation shown in status table (and using yellow in fingerprints) is computed on the repeated iterations of the test *during the same run*. It's not the standard deviation computed for all the existing builds. Usually performance test are written using the following template: public void test() { // warm-up // run the code to test once or several times to warm the JIT on it // perform several iteration of the test to get the average for (int i=0; i<MAX; i++) { // start the measure startMeasuring(); // run the code once or several times if the time of the test is too short // (typically less than 100ms) // stop the measure stopMeasuring(); } // Commit the measures (i.e. compute the average and the standard deviation // of the MAX measures and put the numbers in database) commitMeasurements(); // Verify whether the test fails or not (i.e. had a time regression over 10%) assertPerformance(); } So, this standard deviation is meaningful for each test of the current build. It's also computed for baseline, and meaningful as well, but not currently shown in the status table... The standard deviation you're thinking about is computed but shown only in each test data pages (e.g. for one of JDT/Core Search perf test: http://ganymede-mirror2.eclipse.org/eclipse/downloads/drops/R-3.4-200806172000/performance/eclipseperflnx3/org.eclipse.jdt.core.tests.performance.FullSourceWorkspaceSearchTests.testSearchAllTypeNames()_raw.html) You can see the standard deviation computed on all builds at the top of the table in 'STD DEV' line (the 'COEF VAR' line below = 'STD DEV' / MEAN). HTH
Ok, thanks for the info. I didn't realize the standard deviation used to mark the tests yellow was different from the standard deviation in the raw data table. As these graphs get more complicated, it would help to have a legend at the bottom, or a "How do I read these graphs" hyperlink that would take you to a page describing in more detail how to interpret the graphs. I.e., what yellow means, what grey means, etc.
(In reply to comment #9) > Ok, thanks for the info. I didn't realize the standard deviation used to mark > the tests yellow was different from the standard deviation in the raw data > table. > > As these graphs get more complicated, it would help to have a legend at the > bottom, or a "How do I read these graphs" hyperlink that would take you to a > page describing in more detail how to interpret the graphs. I.e., what yellow > means, what grey means, etc. > Please comment bug 202084. I'll use it to improve documentation in generated pages.
Created attachment 114544 [details] Early preview of new fingerprints design Here's an early preview of what could be the new fingerprints. The key points of this new design are: 1) Use time instead of percentage for the results (note that percentage would be still displayed...). This will make easier to figure out the importance of a regression or an improvement: we obviously should first look at a 5% regression of a 10s test before looking at a 30% regression of a 100ms test... 2) Show the reference time. In this preview, the 3.4.0 reference bar is in blue above the current build time. The final goal is also to print all previous references since 3.0! This will also help us to have a better relevance for the importance of a regression: typically if a great improvement has been done between 3.3.0 and 3.4.0, then a flat result in 3.5 would be acceptable... Not yet in this preview: 3) Show the uncertainty zone. Instead of painting the entire bar in yellow when the standard error is over 3%, always paint the uncertainty zone at the end of the bar (as described in comment 0) 4) Display variation regarding previous build(s). This would help us to see how performance evolves for a test during the release cycle. This comparison may be vs. the previous build or/and the previous milestone(s) (previous result(s) would have been smoothed or not). Typically an improvement of 20% may have been done between two milestones and a 10% regression may happen after. Currently, we are not able to detect this regression as the test would still be 10% better than the reference...
Looks nice but could be harder to read for color blind people.
Re: comment 11 >4) Display variation regarding previous build(s (just a thought) Would it help for the duration of a release cycle to render bars for each milestones ? Once the release is over, then the intermediate milestones would be discarded ?
(In reply to comment #12) > Looks nice but could be harder to read for color blind people. > This was just a rough draft preview to have an idea of how the fingerprints look like when using times instead of percentages and with several bars for different builds values... Colors definition and usage will be done after, but I agree that it should be done carefully...
(In reply to comment #13) > Re: comment 11 > >4) Display variation regarding previous build(s > (just a thought) > Would it help for the duration of a release cycle to render bars for each > milestones ? Once the release is over, then the intermediate milestones would > be discarded ? > I like this idea, but that would mean a really big height for each fingerprint at the end of the release as we need a minimal height for each bar... Would it be acceptable that a fingerprint took an entire page height (or more)?
Created attachment 115170 [details] New proposed patch Here a new proposed patch which address points 1) and 3), 2) partially (there's only one basleine) but not 4). Note that a preview of this new fingerprints should be available on M20081001-0800 performance results soon...
(In reply to comment #16) > > Note that a preview of this new fingerprints should be available on > M20081001-0800 performance results soon... > It's now available, thanks Kim :-) Note that it's only a preview, all feedbacks are welcome to improve this new functionality... Tips: 1) a text is in italic in the fingerprint means that there's a warning you can read in the tooltip by flying over it 2) fly over bars to see the build result time
I like the new graphs, good work! It's very good that I can choose between different display types and especially, that there's a logarithmic view. Somme details: - that a bar can be partially yellow should be part of the legend and explained - already the global summary should get a legend - italic not very good visible and confusing because the test name (on the right) can also be italic but has a different meaning - the chosen display type should be preserved (currently I have to set it each time I switch a page) - I get no (or little feedback) when I select another display type
I also like the graphs. And as Dani said, it would be good if the mode I selected (e.g. time (linear)) was remembered when I go to another component page.
Created attachment 117390 [details] Final proposed patch This patch addresses all points of Dani's feedback in comment 18. Note that there was nothing I can do for the last one, as this is the browser which should provide feedback while downloading pages (i.e. status bar and egger mouse pointer at least...). Note that I tested this patch both on Firefox and IE, but only on Windows boxes. I would appreciate to have feedback if it also works on a Mac or not...
Results generated using last patch should be available soon at: http://download.eclipse.org/eclipse/downloads/drops/M20081105-1550/performance/performance.php
Patch released for 3.5M4 in HEAD stream.
