| Re: [eclipse.org-committers] Compute cluster instability |
Hi
In addition to the obvious offline problems shown on https://ci.eclipse.org/ I see very variable job start delays. It is almost as if there is a 25% chance that a selected node will work followed by a two minute timeout before another try can occur. I have seen one job take nearly an hour to start.
Please take a moment to Subscribe to the Eclipse Status page at https://eclipsestatus.io/
Greetings everyone,
If you're operating a Jenkins instance at ci.eclipse.org (Jiro), you may have noticed some instability recently. The last software update we've made to the compute cluster is having an ill effect on compute nodes. Although Kubernetes can reschedule a running instance on a different node, this does not always succeed without intervention and it may leave a CI instance in an odd state for hours, depending on EF IT staff availability.
Even when an instance is rescheduled successfully, it does mean a "reboot" of your instance and a potentially failed build.
We apologize for the inconvenience, and the team is planning the next software update to hopefully resolve this recent instability. Our ultimate goal is to test updates on our staging cluster to establish reliability before upgrading the production cluster.
Please take a moment to Subscribe to the Eclipse Status page at https://eclipsestatus.io/
Thanks for reading.
--
Denis Roy
Director, IT Services | Eclipse Foundation
Eclipse Foundation: The Community for Open Innovation and Collaboration
Twitter: @droy_eclipse
_______________________________________________ eclipse.org-committers mailing list eclipse.org-committers@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/eclipse.org-committers