[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[platform-core-dev] RE: core.resources Performance Test question
|
Hi Frederic,
many thanks for the detailed explanations!
And, thanks for the good news especially :)
Unfortunately I really don't have time at the moment to do
performance runs locally, and given what Boris said in the
Arch call, the chances of getting accurate results on my
Laptop are very small anyways.
So what I did, was run the performance test once -- this does
repeat the same test 5 times already and does some computations
of deviation etc. Doing this, I was not able to see any
measurable difference in performance before my change versus
after it -- which, I think, is news good enough to wait for
final verification by the official performance test machines.
Regarding the heap size, I'm not yet sure whether that is a
usable number. I've been running some local tests and the
number is unexpectedly very unstable -- between unrelated
test runs it varies up and then down again and I have no clue
why that would be. I can only guess that the Hotspot VM might
be to blame. So I guess that's another item to investigate
separately, and I would think that investing into better
infrastructure for displaying heapsize data is not much worth
when the data itself is not yet well understood.
The one thing that I would really suggest for the generated
HTML pages is that obvious outliers in the baseline should
not be taken as baseline number. right now, it looks like
the latest baseline is always taken and if that is an outlier,
it's bad luck. It would be better to just not consider obvous
outliers of baseline (and take a baseline that's 1,2,3...weeks
old instead).
Thanks a lot for your time. For now, I am waiting for another
N-build with performance results [like others as well, I guess].
Cheers,
--
Martin Oberhuber, Senior Member of Technical Staff, Wind River
Target Management Project Lead, DSDP PMC Member
http://www.eclipse.org/dsdp/tm
> -----Original Message-----
> From: Frederic Fusier [mailto:frederic_fusier@xxxxxxxxxx]
> Sent: Friday, October 16, 2009 12:12 PM
> To: Oberhuber, Martin
> Cc: Boris Bokowski; John Arthorne; Eclipse Platform Core
> component developers list.; Szymon Brandys
> Subject: Re: core.resources Performance Test question
>
> Hi Martin,
>
> Here are some clues on your questions:
> I agree that the results of the Elapsed Process Time for this
> scenario does not look really stable. It clearly
> appears that from
> time to time (on each machine), there are some
> outliers. IMO, if you
> need to have a good idea of what is the mean for this
> test, then you
> simply should remove this outliers and compute it... As
> I said in the
> Arch' call, we already do that for the JDT/Core tests
> when we run
> them locally. We perform 20 runs and to smooth the
> results, hence
> remove the potential outliers), we remove the 5 worst
> and the 5 best
> results. Doing this we get a 1% precision on all of our tests...
> except for the compiler which precision felt to 4% since it has
> become multi-threaded...
> As the machine may evolve over time (hardware, OS or core
> applications - e.g. OS update), a baseline is run each
> week. This
> should prevent results to be polluted by other changes
> than the ones
> done in the Eclipse code. One side effect of this is
> that baseline
> results may vary on tests which have some stability issue...
> CPU Time was initially used as reference at the beginning of
> Performance results. They are often more stable than the Elapsed
> Process Time, so logically seemed to be the candidate
> to measure and
> compare performance results. The problem is that this
> time does not
> match what the users see while using the IDE... So, as
> we want to
> measure the performance from a user perspective to be
> sure that the
> user cannot feel any regression (at least unexplained),
> we finally
> decided to use the Elapsed Process Time as reference in
> the Eclipse
> performance tests. We obviously lost precision in our
> measurement but
> became closer to the user perception of the overall IDE
> performance.
> Other numbers than the Elapsed Process and CPU times
> are stored in
> the performance database while running the tests. The
> org.eclipse.test.performance.Dimension gives all
> numbers currently
> stored for each tests. So, the good news is that the memory
> consumption, roughly given by the 'Java Heap Size'
> number should give
> you some clues about the improvement of your change,
> but the bad new
> is that this number is not currently displayed in the HTML pages
> while generating the results. The other good news is
> that I'm working
> on a performance results tool which should allow users
> to generate
> their own results, typically adding any other stored
> Dimension number
> to the generated HTML pages (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=288234)... but the
> other bad news is that this tool is not finalized yet
> and I would
> need some time to implement this specific functionality...
>
> So, to summarize... in your case, to have a good idea of the
> performance
> impact of your change, I would advice you to perform several
> runs of this
> test (at least 10) on a local machine for
> the baseline
> the last released version
> the last released version + your patch
>
> Then for each group of tests (1, 2 & 3):
> remove either the obvious outliers or systematically the
> two worst and
> best results
> compute the means with the remaining results
> compare deltas: (2 - 1) vs (3 vs 1) to know whether your
> change has an
> impact or not on the performance results...
>
> HTH
>
> Cordialement/Regards,
>
> Frédéric
>
>
>
>
> |------------>
> | From: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |"Oberhuber, Martin" <Martin.Oberhuber@xxxxxxxxxxxxx>
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | To: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |"Szymon Brandys" <Szymon.Brandys@xxxxxxxxxx>, "John
> Arthorne" <John_Arthorne@xxxxxxxxxx>, "Boris Bokowski"
> <Boris_Bokowski@xxxxxxxxxx>, Frederic |
> |Fusier/France/IBM@IBMFR, "Eclipse Platform Core component
> developers list." <platform-core-dev@xxxxxxxxxxx>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | Date: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |10/15/2009 08:39 PM
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | Subject: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |core.resources Performance Test question
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
>
>
>
>
>
> Hi all,
>
> as I have mentioned in the Arch call yesterday, I have
> implemented a change
> to core.resources that should greatly reduce memory consumption during
> "Refresh Project", and now I'd like to verify that it doesn't
> take longer
> to run and in fact does reduce memory consumption.
>
> In anticipation of tonight's 3.6 run with performance tests,
> I investigated
> the previous performance run on I20091013-1302:
> http://download.eclipse.org/eclipse/downloads/drops/I20091013-
> 1302/performance/org.eclipse.core.php?fp_type=0
>
> My questions:
> For the "Refresh Project" scenario that I'm interested
> in, the test
> shows 68% improvement (compared to 3.5) on SLED10, but 53%
> degradation on Windows (on no change at 2.3% on RHEL5).
> These numbers
> do not seem reasonable at all. I'm assuming that this
> is due to the
> outlier in elapsed process time (see below) -- right?
> Or how else to
> interpret this?
> Digging deeper:
>
> http://download.eclipse.org/eclipse/downloads/drops/I20091013-
> 1302/performance/eplnx1/Scenario166.html
> the baseline shows a complete outlier in terms of
> elapsed process
> time. How to interpret that? Shouldn't the baseline
> always remain the
> same? And all percentage comparison be against a mean value of
> baseline, rather than an outlier as done here?
> Why do the tests (when looking on the highest level)
> look at Elapsed
> Process time rather than CPU time, which seems stable here?
> I'm interested in memory consumption, but the test data
> does not even
> seem to record that... or where can I find it?
> Thanks,
> Martin,
> --
> Martin Oberhuber, Senior Member of Technical Staff, Wind River
> Target Management Project Lead, DSDP PMC Member
> http://www.eclipse.org/dsdp/tm
>
>
>
>
>
