Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] hipp9 overloaded

Hi

On the 'ocl' hipp, I have had to increase the timeout on a 12 minute job from 60 minutes to 180 to get it to run. This at 6:00 GMT on a Sunday!

While observing the progress interactively, it is my impression that the JUnit tests are not dramatically slower, but anything involving repos or GIT, presumably the network, crawls.

    Regards

        Ed Willink

On 05/02/2017 14:24, Stephan Herrmann wrote:
Could this be a general issue?
The factor of 4 looks similar to what I see in hipp10 (Object Teams).

Some figures:
Two individual test suites A and B as part of the Object Teams build over time
(from some builds which I happened to keep):

up-to 2016-12-08:
    suite A: around 30 min.
    suite B: around 30 min.

2017-01-18:
    suite A at 47 min.
suite B presumably timed out after 120 min (details no longer available)

2017-01-22:
    suite A at 44 min.
    suite B completed at 110 min

2017-01-27:
    suite A at 60 min.
suite B presumably timed out after 120 min (details no longer available)

2017-02-04:
    suite A timed out at 120 min
suite B died at 50 min (not sure if this could possibly be a time out)

Several intermediate builds timed out, providing no useful test results.

To check if this could be a regression in the software being tested, I inspected logs from local test runs over the given period, which, however, show no performance regression.

When I looked up the various hipps on this hipp10, I couldn't see much info about build history, in fact only the platform hipp has data in the build history view.

The same suite A is actually part of JDT/Core so we can do some comparison of the Object Teams HIPP with eclipse.jdt.core-Gerrit on Platform HIPP (same machine hipp10!). On the latter HIPP we observe alternating time outs but also test runs completing
normally at the typical 30 min.


In this situation I have to run each build several times until I happen to get complete test results. Of course these unnecessary test runs aggravate the situation...

Short term work around: stop using hipp for testing, test only locally (saves much time) use hipp only for building / publishing - which will also take some load from that machine.

Maybe we should all do this to retain good response times at least for the actual
production builds?

best,
Stephan


On 02/02/2017 08:08 AM, Markus Knauer wrote:
I see similar problems on hipp8. Its reported load is between 1 and 2 at the moment, sometimes less, i.e. not very high, but the EPP package builds are extremely slow, much slower than usual. A few months ago in October these builds used to take 1:20h [1], now we are close to 6 hours [2] which is 4 times slower for the same build. In both cases, builds were running on a day during Simultaneous Release week which rules out this as a reason for the increased duration.

Maybe its worth to look at this host, too, as the increased build time doesn't seem to be caused by a high load of the CPU.

And I highly appreciate your efforts to distribute CPU resources in a better way!

Much thanks,
Markus

[1] https://hudson.eclipse.org/packaging/job/neon.epp-tycho-build/448/
[2] https://hudson.eclipse.org/packaging/job/oxygen.epp-tycho-build/183/



On 2 February 2017 at 01:22, Frederic Gurr <frederic.gurr@xxxxxxxxxxx <mailto:frederic.gurr@xxxxxxxxxxx>> wrote:

    Hi,

    Unfortunately hipp9 has been experiencing a near-constant high load
    during work hours. Due to a few projects hogging resources other
projects are severely constrained and have problems running their builds
    in a timely manner.

    To fix this, I have taken the following near term steps:

    - Papyrus HIPP
- disabled execution of concurrent builds for "Papyrus-Gerrit-Oxygen"
      - limited the number of executors to 2

    - EMF Compare HIPP
      - limited the number of executors to 2

    I will also move the EMF Compare HIPP to a different hipp machine
    tomorrow at 9am EST. Downtime will be less than one hour.

We will also investigate how CPU usage can be limited to ensure a fair
    distribution of resources.

    Regards,

    Fred
    _______________________________________________
    cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx <mailto:cross-project-issues-dev@xxxxxxxxxxx> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
<https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev>






_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev


_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus



Back to the top