Hi folks,
We're using a patched version of jetty 6.1.26 for hadoop (initially provided by Greg W and updated by Cloudera folks), and we're hitting some serious issue on the NIO acceptor.
Basically under some condition, the jetty acceptor is unable to accept new connections and the acceptor thread is trashed (keeps erroring out on incoming connections).
We traced it down to repeated exceptions of the following:
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00000007ee2e9800> (a java.nio.channels.CancelledKeyException)
at java.lang.Throwable.<init>(Throwable.java:181)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.RuntimeException.<init>(RuntimeException.java:32)
at java.lang.IllegalStateException.<init>(IllegalStateException.java:27)
at java.nio.channels.CancelledKeyException.<init>(CancelledKeyException.java:30)
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:532)
at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Once it hits this state, it doesn't recover until the process is restarted. And all incoming connection attempt seem to time out.
I strongly suspect there is a bug in this line:
https://github.com/toddlipcon/jetty-hadoop-fix/blob/6.1.26.cloudera/modules/jetty/src/main/java/org/mortbay/io/nio/SelectorManager.java#L532
This line of code is calling interestOps() on a key that is known to be invalid (note '&&'). This always throws a CancelledKeyException, and the rest of the operation in doSelect() is never executed.
Should it be called with '||' instead or not call interestOps() in this case? Thoughts? Any input would be greatly appreciated. Thanks!
Regards,
Sangjin
