Re: [jetty-users] Jetty completely unresponsive, seems to be caused by C

[Jesse McConnell <jesse.mcconnell@xxxxxxxxx>]

what operative system?

--
jesse mcconnell
jesse.mcconnell@xxxxxxxxx


On Wed, May 2, 2012 at 2:59 PM, Mennonite _ <mennonite@xxxxxxxxxxx> wrote:
> Jetty version: 7.6.2
> Java version: 1.6.0 U30
>
> Every once in a while, Jetty all but completely stops responding to
> HTTP/HTTPS connections. I got a Java thread dump, and tracing through the
> locks, it looks like there's a single
> org.eclipse.jetty.io.nio.ChannelEndPoint.close() which, via locks, seems to
> block all other close()s, bringing everything to a grinding halt.
>
> There are a ton of threads, all with very similar/identical stack traces,
> waiting for one lock:
> "qtp683347213-21440" prio=10 tid=0x00002aaab57ac000 nid=0x3172 waiting for
> monitor entry [0x0000000042d8f000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.nio.channels.spi.AbstractSelector.cancel(Unknown Source)
>         - waiting to lock <0x00000006e659db48> (a java.util.HashSet)
>         at java.nio.channels.spi.AbstractSelectionKey.cancel(Unknown Source)
>         - locked <0x00000007398808d0> (a sun.nio.ch.SelectionKeyImpl)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:691)
>         at
> org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
>         at
> org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
>         at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
> Which is held by this thread, which is waiting for another lock:
> "qtp683347213-90 Selector7" prio=10 tid=0x000000004f5c3000 nid=0x7773
> waiting for monitor entry [0x0000000041bdb000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>         - waiting to lock <0x00000007398806a0> (a java.lang.Object)
>         at sun.nio.ch.EPollSelectorImpl.implDereg(Unknown Source)
>         at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>         - locked <0x00000006e659db48> (a java.util.HashSet)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>         - locked <0x00000006e659e300> (a sun.nio.ch.Util$2)
>         - locked <0x00000006e659e2f0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000006e659daf0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.selectNow(Unknown Source)
>         at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
> Which is held by this thread:
> "qtp683347213-19755" prio=10 tid=0x000000005005b800 nid=0x7f5e runnable
> [0x00000000440a3000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileDispatcher.preClose0(Native Method)
>         at sun.nio.ch.SocketDispatcher.preClose(Unknown Source)
>         at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown
> Source)
>         - locked <0x00000007398806a0> (a java.lang.Object)
>         at
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown
> Source)
>         at java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown
> Source)
>         - locked <0x000000073987aea0> (a java.lang.Object)
>         at
> org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:205)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:700)
>         at
> org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
>         at
> org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
>         at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
>
> This happens very sporadically, and so I can't reproduce it on demand. I'd
> expect that there are certain environmental factors that cause this, but I
> don't know what those are.
>
> Any thoughts about this?
>
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>