Just a few random tips for Hudson stability based on our
experience…
Build and test should be split into separate jobs. Tests should
never be run on the same nodes as builds. Tests (especially those that are
UI-dependent or launch external processes) are a lot more fragile than the
build. They have higher probability of fubaring a node. It can be very
difficult to tell what fubared a node if you are running many executors per
node. Test and build jobs also place very different demands on the hardware,
which implies that you’d like to configure the node VMs differently.
For running builds, you need as much I/O power as possible and
relatively little CPU power. It works pretty well to create beefier build nodes
and crank up the number of executors. In terms of figuring out the number of
executors to use, I recommend figuring out how many concurrent builds the I/O
system can handle without more than 50% slowdown (as compared to running only
one build process). Then assign one core for every two executors. The remaining
cores and RAM can be used to run test nodes as those will barely touch the
disk.
For running tests, you need very little I/O power and good amount
of CPU power. Since tests are more likely to fubar a node and thereby cause
interference to other jobs, I recommend creating a node/VM per-core as long as
memory allows and using only one Hudson executor per node/VM.
- Konstantin
From:
cross-project-issues-dev-bounces@xxxxxxxxxxx
[mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] On Behalf Of David
Carver
Sent: Thursday, September 30, 2010 1:03 PM
To: cross-project-issues-dev@xxxxxxxxxxx; Denis Roy
Subject: Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is
taking ages
The more important question is why does this particular job
take 7 plus hours to run?
Dave
On 09/30/2010 12:50 PM, Denis Roy wrote:
Trip, in theory that would be a good idea. But many
builds are launched from an SCM trigger, so the load naturally follows the work
days.
Denis
On 09/30/2010 12:32 PM, Trip Gilman wrote:
Do you think it might be helpful to set up
some type of calendar that at least tracks when people have their various
builds scheduled to run? This might cut down on some of the rush hour
effect to the servers.
Trip
On 9/30/10 9:58 AM, "Denis Roy" <denis.roy@xxxxxxxxxxx>
wrote:
Yesterday
the slaves has eight executors each, and we were having SSH issues. So I followed
Apache's lead and reduced the executors per slave.
Even when only one job is running, it seems to get spaced out and no
longer works, even if the slave and master are still in harmony (pun not
intended).
At this point I'm grasping at straws.
On 09/30/2010 10:43 AM, David Carver wrote:
Current
backlog on Hudson doesn't seem to be Master, but the Hudson-Slave1 and
Hudson-Slave2 servers. The number of executors on these
machines is way to low for the number of jobs we have. I noticed that the
number of executors were greatly reduced yesterday.
Dave
On 09/30/2010 07:33 AM, Denis Roy wrote:
On
09/30/2010 10:25 AM, David M Williams wrote:
But,
we can't all run all jobs on master.
Well, why not. Since the master is virtualized, why don't we goose
it up and crank up the executor threads and see how it goes? Maybe then
we'll all be able to get some work done?
Is anyone game?
From: Matthias Sohn <matthias.sohn@xxxxxxxxxxxxxx> <mailto:matthias.sohn@xxxxxxxxxxxxxx>
To: Cross project
issues <cross-project-issues-dev@xxxxxxxxxxx>
<mailto:cross-project-issues-dev@xxxxxxxxxxx>
Date: 09/30/2010 10:11 AM
Subject: [cross-project-issues-dev]
cbi-papyrus-0.7-nightly build is taking
ages
Sent by: cross-project-issues-dev-bounces@xxxxxxxxxxx
This
build seems to take ages, according to Hudson it started Sep
30, 2010 3:11:46 AM
<https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/>
and
now we have Sep 30, 2010 10:08:33 AM, anything wrong here or is this some massive
compile
or test job ?
https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console>
I
am just wondering since I am waiting since hours for my few minutes egit build
job
to
grab a free worker thread.
--
Matthias_______________________________________________
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev