Re: [jetty-users] Jetty completely unresponsive, seems to be caused by C

CentOS 5.5 64-bit

> what operative system?

Jetty version: 7.6.2
Java version: 1.6.0 U30

Every once in a while, Jetty all but completely stops responding to HTTP/HTTPS connections. I got a Java thread dump, and tracing through the locks, it looks like there's a single org.eclipse.jetty.io.nio.ChannelEndPoint.close() which, via locks, seems to block all other close()s, bringing everything to a grinding halt.

There are a ton of threads, all with very similar/identical stack traces, waiting for one lock:
"qtp683347213-21440" prio=10 tid=0x00002aaab57ac000 nid=0x3172 waiting for monitor entry [0x0000000042d8f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.nio.channels.spi.AbstractSelector.cancel(Unknown Source)
        - waiting to lock <0x00000006e659db48> (a java.util.HashSet)
        at java.nio.channels.spi.AbstractSelectionKey.cancel(Unknown Source)
        - locked <0x00000007398808d0> (a sun.nio.ch.SelectionKeyImpl)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:691)
        at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
        at org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Unknown Source)

Which is held by this thread, which is waiting for another lock:
"qtp683347213-90 Selector7" prio=10 tid=0x000000004f5c3000 nid=0x7773 waiting for monitor entry [0x0000000041bdb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
        - waiting to lock <0x00000007398806a0> (a java.lang.Object)
        at sun.nio.ch.EPollSelectorImpl.implDereg(Unknown Source)
        at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
        - locked <0x00000006e659db48> (a java.util.HashSet)
        at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00000006e659e300> (a sun.nio.ch.Util$2)
        - locked <0x00000006e659e2f0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000006e659daf0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.selectNow(Unknown Source)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)
        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Unknown Source)

Which is held by this thread:
"qtp683347213-19755" prio=10 tid=0x000000005005b800 nid=0x7f5e runnable [0x00000000440a3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcher.preClose0(Native Method)
        at sun.nio.ch.SocketDispatcher.preClose(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown Source)
        - locked <0x00000007398806a0> (a java.lang.Object)
        at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown Source)
        at java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown Source)
        - locked <0x000000073987aea0> (a java.lang.Object)
        at org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:205)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:700)
        at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
        at org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Unknown Source)

This happens very sporadically, and so I can't reproduce it on demand. I'd expect that there are certain environmental factors that cause this, but I don't know what those are.

Any thoughts about this?