Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] hipp9 overloaded 
Could this be a general issue?
The factor of 4 looks similar to what I see in hipp10 (Object Teams).

Some figures:
Two individual test suites A and B as part of the Object Teams build over time
(from some builds which I happened to keep):

up-to 2016-12-08:
	suite A: around 30 min.
	suite B: around 30 min.

2017-01-18:
	suite A at 47 min.
	suite B presumably timed out after 120 min (details no longer available)

2017-01-22:
	suite A at 44 min.
	suite B completed at 110 min

2017-01-27:
	suite A at 60 min.
	suite B presumably timed out after 120 min (details no longer available)

2017-02-04:
	suite A timed out at 120 min
	suite B died at 50 min (not sure if this could possibly be a time out)

Several intermediate builds timed out, providing no useful test results.

To check if this could be a regression in the software being tested, I inspected logs from
local test runs over the given period, which, however, show no performance regression.

When I looked up the various hipps on this hipp10, I couldn't see much info about
build history, in fact only the platform hipp has data in the build history view.

The same suite A is actually part of JDT/Core so we can do some comparison of the
Object Teams HIPP with eclipse.jdt.core-Gerrit on Platform HIPP (same machine hipp10!).
On the latter HIPP we observe alternating time outs but also test runs completing
normally at the typical 30 min.


In this situation I have to run each build several times until I happen to get
complete test results. Of course these unnecessary test runs aggravate the situation...

Short term work around: stop using hipp for testing, test only locally (saves much time)
use hipp only for building / publishing - which will also take some load from that machine.

Maybe we should all do this to retain good response times at least for the actual
production builds?

best,
Stephan


On 02/02/2017 08:08 AM, Markus Knauer wrote:

I see similar problems on hipp8. Its reported load is between 1 and 2 at the moment, sometimes less, i.e. not very high, but the EPP
package builds are extremely slow, much slower than usual. A few months ago in October these builds used to take 1:20h [1], now we
are close to 6 hours [2] which is 4 times slower for the same build. In both cases, builds were running on a day during Simultaneous
Release week which rules out this as a reason for the increased duration.

Maybe its worth to look at this host, too, as the increased build time doesn't seem to be caused by a high load of the CPU.

And I highly appreciate your efforts to distribute CPU resources in a better way!

Much thanks,
Markus

[1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
[2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/



On 2 February 2017 at 01:22, Frederic Gurr <frederic.gurr@xxxxxxxxxxx <mailto:frederic.gurr@xxxxxxxxxxx>> wrote:

    Hi,

    Unfortunately hipp9 has been experiencing a near-constant high load
    during work hours. Due to a few projects hogging resources other
    projects are severely constrained and have problems running their builds
    in a timely manner.

    To fix this, I have taken the following near term steps:

    - Papyrus HIPP
      - disabled execution of concurrent builds for "Papyrus-Gerrit-Oxygen"
      - limited the number of executors to 2

    - EMF Compare HIPP
      - limited the number of executors to 2

    I will also move the EMF Compare HIPP to a different hipp machine
    tomorrow at 9am EST. Downtime will be less than one hour.

    We will also investigate how CPU usage can be limited to ensure a fair
    distribution of resources.

    Regards,

    Fred
    _______________________________________________
    cross-project-issues-dev mailing list
    cross-project-issues-dev@xxxxxxxxxxx <mailto:cross-project-issues-dev@xxxxxxxxxxx>
    To change your delivery options, retrieve your password, or unsubscribe from this list, visit
    https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
    <https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev>






_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

Back to the top