Re: [platform-releng-dev] Questions/Remarks about 3.1 M6 JDT/Core perfor

[Sonia Dimitrov <Sonia_Dimitrov@xxxxxxxxxx>]

See my comments below denoted by sd>

Sonia

Frederic Fusier <frederic_fusier@xxxxxxxxxx>
Sent by: platform-releng-dev-bounces@xxxxxxxxxxx

04/04/2005 06:24 AM

Please respond to
"Eclipse platform release engineering list."

To	platform-releng-dev@xxxxxxxxxxx
cc
Subject	[platform-releng-dev] Questions/Remarks about 3.1 M6 JDT/Core        performances results

I was just looking at 3.1 M6 performance results
(http://fullmoon.torolab.ibm.com/downloads/drops/S-3.1
M6-200504011645/performance/org.eclipse.jdt.core.php?)
and have several remarks/questions about them.

1) New tests on Scanner and Parser
We added these tests last week. We expect them to reflect a
performance increase around 30% regarding to 3.0 version.
Unfortunately, it seems that last Friday, 3.0 reference was not
rebuilt and results are negative :-(.
Would it be possible to make this reference as soon as possible
and update JDT/Core performance results page to be able to see
the *real* deltas for these tests?

sd> we will rerun the 3.1M6 performance tests on
a fresh baseline on Wednesday.  We will rebuild and rerun the baseline
tests following tomorrow's integration build.  Due to all the build
activity surrounding 3.1M6 (not to mention the team being at 50%), there
was no time to run baseline tests last Friday.

Waiting for this 3.0 rebuild, I wonder why same tests with same code
(no change has been done for Scanner and Parser since tests creation)
produce deltas going from -2.6% to +7.2% (ie. dispersion near 10%).
Does it means that precision for performance tests is not better
than 10%? Or is there any issue with test box during last week?

sd>  good question, we need time to investigate
further to answer with some precision.

2) Full source workspace build without comment
I do not understand why there's no 3.0 reference on Win/XP test box
although there's one on RHEL one?
What does 3.0_200406250800_200412241024 mean?

sd>  this is the "substitute" build
id (this one happens to be a baseline (3.0) we ran on Dec. 24 in 2004,
but could be an I or N build) used by the performance framework to assert
against when the specified reference does not exist.  Andre could
comment best on how the substitute is chosen.  

sd>  as for the missing reference on Windows,
this information is in the consolelog.  I can investigate this further
with a local member of the JDT Core team.

 [java] !ENTRY org.eclipse.test.performance 2
0 2005-04-01 23:24:50.484
     [java] !MESSAGE no reference data named '{config=relengbuildwin2,
build=3.0_200406251208_200503271002, jvm=sun1.4.2_06}' found

     [java] !ENTRY org.eclipse.test.performance 2 0 2005-04-01
23:30:14.937
     [java] !MESSAGE no reference data named '{config=relengbuildwin2,
build=3.0_200406251208_200503271002, jvm=sun1.4.2_06}' found

     [java] !ENTRY org.eclipse.test.performance 2 0 2005-04-01
23:31:51.265
     [java] !MESSAGE no reference data named '{config=relengbuildwin2,
build=3.0_200406251208_200503271002, jvm=sun1.4.2_06}' found

     [java] !ENTRY org.eclipse.test.performance 2 0 2005-04-01
23:32:28.359
     [java] !MESSAGE no reference data named '{config=relengbuildwin2,
build=3.0_200406251208_200503271002, jvm=sun1.4.2_06}' found

     [java] !ENTRY org.eclipse.test.performance 2 0 2005-04-01
23:33:57.546
     [java] !MESSAGE no reference data named '{config=relengbuildwin2,
build=3.0_200406251208_200503271002, jvm=sun1.4.2_06}' found
     [java] -------------------------------------
     [java] DOM/AST creation performance test statistics:
     [java]   - 6,156,412 nodes have been parsed.
     [java]   - 49,713 nodes have extended start position.


3) Difference between 2 test boxes
It seems that Win/XP results have changed a lot since last week and
do not really match reality. I wonder what happens on this machine?
Perhaps is there any change on it or something wrong happens during
reference 3.0 build which makes these results inaccurate?
Do you have any clue on this?


Cordialement/Regards,

Frédéric


_______________________________________________
platform-releng-dev mailing list
platform-releng-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/platform-releng-dev