Community
Participate
Working Groups
The recent performance tests I20110419-1004 seem to contain a number of weird jumps in both 3.6.2 baseline and the I-build. For an extreme example, see http://fullmoon.ottawa.ibm.com/downloads/drops/I20110419-1004/performance/epwin3/Scenario376.html where the baseline elapsed process time is reported as "-12.74s". It is fairly important to both address this quickly and remove invalid values from the database as they will break the statistics comparison and render performance tests invalid.
Created attachment 193821 [details] Screenshot
I've started another baseline run. Not sure why this is happening. If it's just a anomaly, in the baseline and the tests, it will just show as a blip in the graph. One thing we did do this week, was to change the performance tests to not run with code coverage enabled. See bug 342785. However, this change didn't impact the baseline test runs.
(In reply to comment #2) > .. it will just show as a blip in the graph. It will cause many graphs to be scaled so that real differences disappear. More importantly, it will throw off calculations of statistics that decide whether there is significant difference between baseline and a current build.
http://fullmoon.ottawa.ibm.com/downloads/drops/I20110424-2000/performance/epwin3/Scenario376.html "Elapsed process" shows 503.8 % improvement. It gets it from comparing with the "-12.74s" (note the minus sign) baseline time. (In reply to comment #2) > I've started another baseline run. There does not seem to be results for this run; the current I-build is compared against bogus results.
Yes, the machines were in a weird state over the weekend. I rebooted them yesterday, and started another baseline run yesterday. The baseline is still running due to the fact that the 8pm Sunday build held onto the performance machines for over a day.
This issue makes pass/fail results invalid in the M7 candidate build (I20110428-0848). For example, see: http://fullmoon.ottawa.ibm.com/downloads/drops/I20110428-0848/performance/epwin2/Scenario355.html The I-build test results are compared with an invalid baseline run.
I ran a new baseline on Monday, not sure why they are running a comparison against and old baseline. They could compare against the R-3.6-201006080911_201104251003 baseline run that is specified in the ref37.properties. Investigating.
(In reply to comment #7) > I ran a new baseline on Monday, not sure why they are running a comparison > against and old baseline. They could compare against the > R-3.6-201006080911_201104251003 baseline run that is specified in the > ref37.properties. Investigating. Baseline R-3.6-201006080911_201104251003 hasn't run on the windows machine. We do have a new baseline R-3.6-201006080911_201104291800 which has run on the windows machines also. However the latest performance run N20110430-2000 still didn't use this baseline for the result generation. We had this problem for a while where sometimes the latest baseline is not being used for comparison - bug 322807. I couldn't find out the real reason for this yet :(.
For some reason, the performance baselines run process continues to stay alive after the tests have completed. The properties file that contains the next baseline doesn't get updated. I have just updated it manually so today's build will run against the latest baseline. I'm investigating why this is happening.
Looks like the baseline and results are fixed in the I20110504-0800 build http://download.eclipse.org/eclipse/downloads/drops/I20110504-0800/performance/epwin2/Scenario355.html
