Community
Participate
Working Groups
Verifying performance results for all 3.6M4 scenarios, it appears that results for OpenTypePerfTest#testWarmException() test are completely unreliable (on all test machines). It seems that something should be done to stabilize this test or remove it from the benchmark.
Created attachment 167118 [details] testfix Actually all tests in OpenTypePerfTest look a bit unreliable to me. The most unreliable ones are testWarmException() (unreliable since 3.3) and testWarmOpenSWT() (unreliable since 3.4). Markus, as discussed the patch does the following - do joinBackgroudActivities() in testColdException to make it a bit more reliable. - assert only absolute performance for testWarmException() and testWarmOpenSWT() to ensure that it doesn't take ages to open the type. This way we avoid affecting the other tests.
Make sure to also update the 'perf_35x' branch.
+1, looks good. Deepak, please commit to HEAD. Furthermore, we need to release this to the perf_35x branch. I think it's best if we do this together via a screensharing session. The basics are: - create a new workspace with Eclipse SDK 3.5 - create the perf_35x branch of org.eclipse.jdt.ui.tests based on R3_5 - apply and commit the patch - check out org.eclipse.releng from the perf_35x branch - release org.eclipse.jdt.ui.tests with tag v20100510-perf_35x
(In reply to comment #3) > +1, looks good. Deepak, please commit to HEAD. Committed to HEAD.
Released this to perf_35x branch as well.
Can't verify this right now. assertPerformanceInAbsoluteBand(..) is not considered in the result page, see bug 89804. And we will have to wait for the perf_35x baseline to compare the changed tests.
(In reply to comment #6) > And we will have to wait for the perf_35x baseline to compare the changed tests. These are still not available.
(In reply to comment #7) > (In reply to comment #6) > > And we will have to wait for the perf_35x baseline to compare the changed tests. > These are still not available. Still not available.
Frédéric, do you have any update on the reference performance results?
There's a new baseline ran on 05/21 and I generated results for I20100523-0800 build including them. Unfortunately, since then there were lot of builds on the releng page (most of them DNF in fact) and this build with valid perf results was pushed out of the page :-( Moreover, there are still perf markers on eclipsebuildserv, hence I still directly access the perf DB to copy it on my local machine and generate perf results for existing builds (I20100527-0800 and I20100527-1700:RC3). However, I have an emergency way to generate results even if the DB is still busy with running perf tests. So, I'll start it right now, but it will take some times to get all the results (around 2 hours for one build). I'll do this for RC3 and I'll inform you when I these results are available...
(In reply to comment #10) > Moreover, there are still perf markers on eclipsebuildserv, hence I still > directly access the perf DB to copy it on my local machine and generate perf > results for existing builds (I20100527-0800 and I20100527-1700:RC3). > Should read: I still *cannot* directly access...
Performance results are now available for RC3 (at least on fullmoon, should be replicated on Eclipse downloads site soon). Note that even if we get a new baseline run on 05/21, I see that results on I-builds still look unstable... It looks a little bit better on Linux machine (although result for RC3 is not good at all for SLED) but the change didn't have any effect on Windows machines results which still look unpredictable :-(
