[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[platform-core-dev] RE: core.resources Performance Test question
|
Hi Frederic,
> BTW, I had a look on my tool and saw that it was not too
> costly to finalize the implementation to display a new number
that is great! - But I don't think I'll need the pages for the
latest I-build since I'll want to compare an upcoming N-build
or I-build on the official performance machines against the
baseline anyways. So it'll be sufficient if that upcoming
N-build or I-build has the new number (heapsize) displayed
for both the new build and the previous baseline.
> Would it be possible to open a bug for this against
> Platform/Releng in order to track the work around this?
You are welcome:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=292545
Cheers,
--
Martin Oberhuber, Senior Member of Technical Staff, Wind River
Target Management Project Lead, DSDP PMC Member
http://www.eclipse.org/dsdp/tm
> -----Original Message-----
> From: Frederic Fusier [mailto:frederic_fusier@xxxxxxxxxx]
> Sent: Friday, October 16, 2009 5:18 PM
> To: Oberhuber, Martin
> Cc: Boris Bokowski; John Arthorne; Eclipse Platform Core
> component developers list.; Szymon Brandys
> Subject: RE: core.resources Performance Test question
>
> Martin,
>
> What you did for the performance test let you know that your
> change didn't
> introduce huge regression. That's a very good first point.
> Now, we'll see the test history to confirm or infirm the
> assumption that it
> didn't either introduce any noticeable regression...
>
> BTW, I had a look on my tool and saw that it was not too
> costly to finalize
> the implementation to display a new number while generating
> performance
> results HTML page. So, now, I have in hand the I20091013-1302
> performance
> results pages with the Java Heap Size number results displayed in all
> scenario data pages. If you are interested in these pages for
> org.eclipse.core component, I can send them to you in a zip
> file (which is
> around 8Mb)...
>
> I agree that the generation could be smarter and filter the
> outlier, but
> it's somehow difficult to clearly identify these kind of number
> statistically speaking. I'll have a think about it and see
> what could be
> done for such scenario results... Would it be possible to
> open a bug for
> this against Platform/Releng in order to track the work
> around this? TIA
>
> Cordialement/Regards,
>
> Frédéric
>
>
>
>
> |------------>
> | From: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |"Oberhuber, Martin" <Martin.Oberhuber@xxxxxxxxxxxxx>
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | To: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |Frederic Fusier/France/IBM@IBMFR
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | Cc: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |"Boris Bokowski" <Boris_Bokowski@xxxxxxxxxx>, "John
> Arthorne" <John_Arthorne@xxxxxxxxxx>, "Eclipse Platform Core
> component developers list." |
> |<platform-core-dev@xxxxxxxxxxx>, "Szymon Brandys"
> <Szymon.Brandys@xxxxxxxxxx>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | Date: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |10/16/2009 02:06 PM
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |------------>
> | Subject: |
> |------------>
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
> |RE: core.resources Performance Test question
>
> |
>
> >-------------------------------------------------------------
> --------------------------------------------------------------
> -----------------------|
>
>
>
>
>
> Hi Frederic,
>
> many thanks for the detailed explanations!
> And, thanks for the good news especially :)
>
> Unfortunately I really don't have time at the moment to do
> performance runs locally, and given what Boris said in the
> Arch call, the chances of getting accurate results on my
> Laptop are very small anyways.
>
> So what I did, was run the performance test once -- this does
> repeat the same test 5 times already and does some computations
> of deviation etc. Doing this, I was not able to see any
> measurable difference in performance before my change versus
> after it -- which, I think, is news good enough to wait for
> final verification by the official performance test machines.
>
> Regarding the heap size, I'm not yet sure whether that is a
> usable number. I've been running some local tests and the
> number is unexpectedly very unstable -- between unrelated
> test runs it varies up and then down again and I have no clue
> why that would be. I can only guess that the Hotspot VM might
> be to blame. So I guess that's another item to investigate
> separately, and I would think that investing into better
> infrastructure for displaying heapsize data is not much worth
> when the data itself is not yet well understood.
>
> The one thing that I would really suggest for the generated
> HTML pages is that obvious outliers in the baseline should
> not be taken as baseline number. right now, it looks like
> the latest baseline is always taken and if that is an outlier,
> it's bad luck. It would be better to just not consider obvous
> outliers of baseline (and take a baseline that's 1,2,3...weeks
> old instead).
>
> Thanks a lot for your time. For now, I am waiting for another
> N-build with performance results [like others as well, I guess].
>
> Cheers,
> --
> Martin Oberhuber, Senior Member of Technical Staff, Wind River
> Target Management Project Lead, DSDP PMC Member
> http://www.eclipse.org/dsdp/tm
>
>
>
> > -----Original Message-----
> > From: Frederic Fusier [mailto:frederic_fusier@xxxxxxxxxx]
> > Sent: Friday, October 16, 2009 12:12 PM
> > To: Oberhuber, Martin
> > Cc: Boris Bokowski; John Arthorne; Eclipse Platform Core
> > component developers list.; Szymon Brandys
> > Subject: Re: core.resources Performance Test question
> >
> > Hi Martin,
> >
> > Here are some clues on your questions:
> > I agree that the results of the Elapsed Process Time for this
> > scenario does not look really stable. It clearly
> > appears that from
> > time to time (on each machine), there are some
> > outliers. IMO, if you
> > need to have a good idea of what is the mean for this
> > test, then you
> > simply should remove this outliers and compute it... As
> > I said in the
> > Arch' call, we already do that for the JDT/Core tests
> > when we run
> > them locally. We perform 20 runs and to smooth the
> > results, hence
> > remove the potential outliers), we remove the 5 worst
> > and the 5 best
> > results. Doing this we get a 1% precision on all of
> our tests...
> > except for the compiler which precision felt to 4%
> since it has
> > become multi-threaded...
> > As the machine may evolve over time (hardware, OS or core
> > applications - e.g. OS update), a baseline is run each
> > week. This
> > should prevent results to be polluted by other changes
> > than the ones
> > done in the Eclipse code. One side effect of this is
> > that baseline
> > results may vary on tests which have some stability issue...
> > CPU Time was initially used as reference at the beginning of
> > Performance results. They are often more stable than
> the Elapsed
> > Process Time, so logically seemed to be the candidate
> > to measure and
> > compare performance results. The problem is that this
> > time does not
> > match what the users see while using the IDE... So, as
> > we want to
> > measure the performance from a user perspective to be
> > sure that the
> > user cannot feel any regression (at least unexplained),
> > we finally
> > decided to use the Elapsed Process Time as reference in
> > the Eclipse
> > performance tests. We obviously lost precision in our
> > measurement but
> > became closer to the user perception of the overall IDE
> > performance.
> > Other numbers than the Elapsed Process and CPU times
> > are stored in
> > the performance database while running the tests. The
> > org.eclipse.test.performance.Dimension gives all
> > numbers currently
> > stored for each tests. So, the good news is that the memory
> > consumption, roughly given by the 'Java Heap Size'
> > number should give
> > you some clues about the improvement of your change,
> > but the bad new
> > is that this number is not currently displayed in the
> HTML pages
> > while generating the results. The other good news is
> > that I'm working
> > on a performance results tool which should allow users
> > to generate
> > their own results, typically adding any other stored
> > Dimension number
> > to the generated HTML pages (see
> >
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=288234)... but the
> > other bad news is that this tool is not finalized yet
> > and I would
> > need some time to implement this specific functionality...
> >
> > So, to summarize... in your case, to have a good idea of the
> > performance
> > impact of your change, I would advice you to perform several
> > runs of this
> > test (at least 10) on a local machine for
> > the baseline
> > the last released version
> > the last released version + your patch
> >
> > Then for each group of tests (1, 2 & 3):
> > remove either the obvious outliers or systematically the
> > two worst and
> > best results
> > compute the means with the remaining results
> > compare deltas: (2 - 1) vs (3 vs 1) to know whether your
> > change has an
> > impact or not on the performance results...
> >
> > HTH
> >
> > Cordialement/Regards,
> >
> > Frédéric
> >
> >
> >
> >
> > |------------>
> > | From: |
> > |------------>
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |"Oberhuber, Martin" <Martin.Oberhuber@xxxxxxxxxxxxx>
> >
> > |
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |------------>
> > | To: |
> > |------------>
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |"Szymon Brandys" <Szymon.Brandys@xxxxxxxxxx>, "John
> > Arthorne" <John_Arthorne@xxxxxxxxxx>, "Boris Bokowski"
> > <Boris_Bokowski@xxxxxxxxxx>, Frederic |
> > |Fusier/France/IBM@IBMFR, "Eclipse Platform Core component
> > developers list." <platform-core-dev@xxxxxxxxxxx>
> > |
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |------------>
> > | Date: |
> > |------------>
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |10/15/2009 08:39 PM
> >
> > |
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |------------>
> > | Subject: |
> > |------------>
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> > |core.resources Performance Test question
> >
> > |
> >
> > >-------------------------------------------------------------
> > --------------------------------------------------------------
> > -----------------------|
> >
> >
> >
> >
> >
> > Hi all,
> >
> > as I have mentioned in the Arch call yesterday, I have
> > implemented a change
> > to core.resources that should greatly reduce memory
> consumption during
> > "Refresh Project", and now I'd like to verify that it doesn't
> > take longer
> > to run and in fact does reduce memory consumption.
> >
> > In anticipation of tonight's 3.6 run with performance tests,
> > I investigated
> > the previous performance run on I20091013-1302:
> > http://download.eclipse.org/eclipse/downloads/drops/I20091013-
> > 1302/performance/org.eclipse.core.php?fp_type=0
> >
> > My questions:
> > For the "Refresh Project" scenario that I'm interested
> > in, the test
> > shows 68% improvement (compared to 3.5) on SLED10, but 53%
> > degradation on Windows (on no change at 2.3% on RHEL5).
> > These numbers
> > do not seem reasonable at all. I'm assuming that this
> > is due to the
> > outlier in elapsed process time (see below) -- right?
> > Or how else to
> > interpret this?
> > Digging deeper:
> >
> > http://download.eclipse.org/eclipse/downloads/drops/I20091013-
> > 1302/performance/eplnx1/Scenario166.html
> > the baseline shows a complete outlier in terms of
> > elapsed process
> > time. How to interpret that? Shouldn't the baseline
> > always remain the
> > same? And all percentage comparison be against a mean value of
> > baseline, rather than an outlier as done here?
> > Why do the tests (when looking on the highest level)
> > look at Elapsed
> > Process time rather than CPU time, which seems stable here?
> > I'm interested in memory consumption, but the test data
> > does not even
> > seem to record that... or where can I find it?
> > Thanks,
> > Martin,
> > --
> > Martin Oberhuber, Senior Member of Technical Staff, Wind River
> > Target Management Project Lead, DSDP PMC Member
> > http://www.eclipse.org/dsdp/tm
> >
> >
> >
> >
> >
>
>
>
>
