[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] Hudson shutdown wait from hell

I'd agree with this general idea ... perhaps a general notice go out on a
mailing list "hudson is being shutdown and restarted ... any jobs not done
by +1 hour [or something] will be killed". (I'm proud to say I cancelled
one of my running aggregator jobs when I saw it was about to be shutdown,
saving 50 minutes or so (if I was only job running) .... partially because
I knew it needed to run again anyway.

But, now I am panicking ... seeing 30 jobs stacked up in queue! I suggest
anyone that has a non-essential job running or queued up consider canceling
it until Hudson "stabilizes" and then run your nightlies, etc. I looks like
the number of threads (executors) has been reduced again (understandably).
But, who knows, maybe it will clear itself up in an hour or two? In time
for our RC3 deadline?

In general, I'll say there's been very little communication about the state
of Hudson .... seems several comments made or questions asked on this list
with no response ... Hudson restarted, configuration changes all without
comment. Is there some resistance to that? Everyone too busy to
communicate? Is their a better channel? Hudson bug list? Do we need a
"hudson-and-infrastructure" mailing list? Just asking.

The queue is down to 26 now ... in the 10 minutes I took to write this
note ... if that rate holds, it will be clear in 90 minutes or so? :/

Thanks,






From:	Miles Parker <milesparker@xxxxxxxxx>
To:	Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>,
Date:	02/08/2012 02:22 PM
Subject:	[cross-project-issues-dev] Hudson shutdown wait from hell
Sent by:	cross-project-issues-dev-bounces@xxxxxxxxxxx



Hi all,

I wanted to bring up a Hudson annoyance and see if people had ideas for
improving this. What's happening now is that any time Hudson gets sent a
shutdown, everyone is locked out until the last job in queue finishes.
Which is a) Good news for the people with running builds, b) Bad news for
everyone else. Observing that most of the time most of us are  in category
b) I vote for making things work better for group b). Nothing against
PTP :D, but they happen to be in group a) this time around and the last run
took 22 hours. :O But there are lot's of long builds out there. This means
that snapshots are delayed for everyone.

So I'm wondering if it might be possible to have some kind of policy where
builds are terminated with prejudice under shutdown. I'm not sure if a)
this is even supportable OOTB in Hudson, and b) whether that would have the
possibility of FUBAR'ing anyone project builds. As project builds should
not be relying on previous state, I would say that the answer to b is
probably no. I also imagine allowance should be made for key builds such as
aggregator. IIRC in the past when in release panic mode we were triggering
hard shutdowns from time to time.

thoughts?

Miles
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev