Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-dev] a bug with old jetty code base (6.1.x)?

Sangjin,

unfortunately we have very little time to look at issues with 6.1.x, as we are really busy getting 9.1.x out the door - ie 3 major versions behind!!!

I'm not sure about your fix as that whole area of code was intended for an old version of the JVM with some very specific bugs in it.   

The real fix is to probably remove the whole _monitorNext block of code and just run on a modern JVM.

You should be able to build your own version of Jetty 6.1.x and try out these changes to see if it helps your issue.      If it does, then lets talk about how we can create a version with such a work around in it.

cheers






On 25 June 2013 05:15, Sangjin Lee <sjlee@xxxxxxxxxx> wrote:
Hi folks,

We're using a patched version of jetty 6.1.26 for hadoop (initially provided by Greg W and updated by Cloudera folks), and we're hitting some serious issue on the NIO acceptor.

Basically under some condition, the jetty acceptor is unable to accept new connections and the acceptor thread is trashed (keeps erroring out on incoming connections).

We traced it down to repeated exceptions of the following:

   at java.lang.Throwable.fillInStackTrace(Native Method)
   - locked <0x00000007ee2e9800> (a java.nio.channels.CancelledKeyException)
   at java.lang.Throwable.<init>(Throwable.java:181)
   at java.lang.Exception.<init>(Exception.java:29)
   at java.lang.RuntimeException.<init>(RuntimeException.java:32)
   at java.lang.IllegalStateException.<init>(IllegalStateException.java:27)
   at java.nio.channels.CancelledKeyException.<init>(CancelledKeyException.java:30)
   at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
   at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
   at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:532)
   at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
   at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
   at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
   at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Once it hits this state, it doesn't recover until the process is restarted. And all incoming connection attempt seem to time out.

I strongly suspect there is a bug in this line: https://github.com/toddlipcon/jetty-hadoop-fix/blob/6.1.26.cloudera/modules/jetty/src/main/java/org/mortbay/io/nio/SelectorManager.java#L532

This line of code is calling interestOps() on a key that is known to be invalid (note '&&'). This always throws a CancelledKeyException, and the rest of the operation in doSelect() is never executed.

Should it be called with '||' instead or not call interestOps() in this case? Thoughts? Any input would be greatly appreciated. Thanks!

Regards,
Sangjin

_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev



Back to the top