Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages

Title: Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages

Just a few random tips for Hudson stability based on our experience…

 

Build and test should be split into separate jobs. Tests should never be run on the same nodes as builds. Tests (especially those that are UI-dependent or launch external processes) are a lot more fragile than the build. They have higher probability of fubaring a node. It can be very difficult to tell what fubared a node if you are running many executors per node. Test and build jobs also place very different demands on the hardware, which implies that you’d like to configure the node VMs differently.

 

For running builds, you need as much I/O power as possible and relatively little CPU power. It works pretty well to create beefier build nodes and crank up the number of executors. In terms of figuring out the number of executors to use, I recommend figuring out how many concurrent builds the I/O system can handle without more than 50% slowdown (as compared to running only one build process). Then assign one core for every two executors. The remaining cores and RAM can be used to run test nodes as those will barely touch the disk.

 

For running tests, you need very little I/O power and good amount of CPU power. Since tests are more likely to fubar a node and thereby cause interference to other jobs, I recommend creating a node/VM per-core as long as memory allows and using only one Hudson executor per node/VM.

- Konstantin

 

 

From: cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] On Behalf Of David Carver
Sent: Thursday, September 30, 2010 1:03 PM
To: cross-project-issues-dev@xxxxxxxxxxx; Denis Roy
Subject: Re: [cross-project-issues-dev] cbi-papyrus-0.7-nightly build is taking ages

 

The more important question is why does this particular job take 7 plus hours to run?

Dave

On 09/30/2010 12:50 PM, Denis Roy wrote:

Trip, in theory that would be a good idea.  But many builds are launched from an SCM trigger, so the load naturally follows the work days.

Denis



On 09/30/2010 12:32 PM, Trip Gilman wrote:

Do you think it might be helpful to set up some type of calendar that at least tracks when people have their various builds scheduled to run?  This might cut down on some of the rush hour effect to the servers.

Trip


On 9/30/10 9:58 AM, "Denis Roy" <denis.roy@xxxxxxxxxxx> wrote:

  Yesterday the slaves has eight executors each, and we were having SSH issues.  So I followed Apache's lead and reduced the executors per slave.
 
 Even when only one job is running, it seems to get spaced out and no longer works, even if the slave and master are still in harmony (pun not intended).
 
 At this point I'm grasping at straws.
 
 
 
 
 On 09/30/2010 10:43 AM, David Carver wrote:

 Current backlog on Hudson doesn't seem to be Master, but the Hudson-Slave1 and Hudson-Slave2 servers.    The number of executors on these machines is way to low for the number of jobs we have.  I noticed that the number of executors were greatly reduced yesterday.
 
 Dave
 
 On 09/30/2010 07:33 AM, Denis Roy wrote:

  On 09/30/2010 10:25 AM, David M Williams wrote:

But, we can't all run all jobs on master.
 


 Well, why not.  Since the master is virtualized, why don't we goose it up and crank up the executor threads and see how it goes?  Maybe then we'll all be able to get some work done?
 
 Is anyone game?
 
 
 
 
 


 
 
 
 
From:        Matthias Sohn <matthias.sohn@xxxxxxxxxxxxxx> <mailto:matthias.sohn@xxxxxxxxxxxxxx>
 
To:        Cross project issues <cross-project-issues-dev@xxxxxxxxxxx> <mailto:cross-project-issues-dev@xxxxxxxxxxx>
 
Date:        09/30/2010 10:11 AM
 
Subject:        [cross-project-issues-dev] cbi-papyrus-0.7-nightly  build is taking        ages
 
Sent by:        cross-project-issues-dev-bounces@xxxxxxxxxxx
 



 
 
 
This build seems to take ages, according to Hudson it started Sep 30, 2010 3:11:46 AM <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/>  
 
and now we have Sep 30, 2010 10:08:33 AM, anything wrong here or is this some massive
 
compile or test job ?
 
 
https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console <https://hudson.eclipse.org/hudson/job/cbi-papyrus-0.7-nightly/1105/console>  
 
 
I am just wondering since I am waiting since hours for my few minutes egit build job
 
to grab a free worker thread.
 
--
 Matthias
_______________________________________________

 
 
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

 


Back to the top