Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] Hudson shutdown wait from hell

I just turned off all the jetty jobs that do CI activities, we have
moved off this infrastructure for the time being and will only use the
handful of enabled jobs for signing release repositories on release
days, at least until things calm down and there is some stability.

took me almost 20 minutes to disable 8 jobs because of the saving
configuration page issues

cheers,
jesse

--
jesse mcconnell
jesse.mcconnell@xxxxxxxxx



On Wed, Feb 8, 2012 at 13:49, David M Williams
<david_williams@xxxxxxxxxx> wrote:
> I'd agree with this general idea ... perhaps a general notice go out on a
> mailing list "hudson is being shutdown and restarted ... any jobs not done
> by +1 hour [or something] will be killed". (I'm proud to say I cancelled
> one of my running aggregator jobs when I saw it was about to be shutdown,
> saving 50 minutes or so (if I was only job running) .... partially because
> I knew it needed to run again anyway.
>
> But, now I am panicking ... seeing 30 jobs stacked up in queue! I suggest
> anyone that has a non-essential job running or queued up consider canceling
> it until Hudson "stabilizes" and then run your nightlies, etc. I looks like
> the number of threads (executors) has been reduced again (understandably).
> But, who knows, maybe it will clear itself up in an hour or two? In time
> for our RC3 deadline?
>
> In general, I'll say there's been very little communication about the state
> of Hudson .... seems several comments made or questions asked on this list
> with no response ... Hudson restarted, configuration changes all without
> comment. Is there some resistance to that? Everyone too busy to
> communicate? Is their a better channel? Hudson bug list? Do we need a
> "hudson-and-infrastructure" mailing list? Just asking.
>
> The queue is down to 26 now ... in the 10 minutes I took to write this
> note ... if that rate holds, it will be clear in 90 minutes or so? :/
>
> Thanks,
>
>
>
>
>
>
> From:   Miles Parker <milesparker@xxxxxxxxx>
> To:     Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>,
> Date:   02/08/2012 02:22 PM
> Subject:        [cross-project-issues-dev] Hudson shutdown wait from hell
> Sent by:        cross-project-issues-dev-bounces@xxxxxxxxxxx
>
>
>
> Hi all,
>
> I wanted to bring up a Hudson annoyance and see if people had ideas for
> improving this. What's happening now is that any time Hudson gets sent a
> shutdown, everyone is locked out until the last job in queue finishes.
> Which is a) Good news for the people with running builds, b) Bad news for
> everyone else. Observing that most of the time most of us are  in category
> b) I vote for making things work better for group b). Nothing against
> PTP :D, but they happen to be in group a) this time around and the last run
> took 22 hours. :O But there are lot's of long builds out there. This means
> that snapshots are delayed for everyone.
>
> So I'm wondering if it might be possible to have some kind of policy where
> builds are terminated with prejudice under shutdown. I'm not sure if a)
> this is even supportable OOTB in Hudson, and b) whether that would have the
> possibility of FUBAR'ing anyone project builds. As project builds should
> not be relying on previous state, I would say that the answer to b is
> probably no. I also imagine allowance should be made for key builds such as
> aggregator. IIRC in the past when in release panic mode we were triggering
> hard shutdowns from time to time.
>
> thoughts?
>
> Miles
> _______________________________________________
> cross-project-issues-dev mailing list
> cross-project-issues-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev
>
>
>
> _______________________________________________
> cross-project-issues-dev mailing list
> cross-project-issues-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev


Back to the top