Re: [jetty-users] Jetty completely unresponsive, seems to be caused by C

[Jesse McConnell <jesse.mcconnell@xxxxxxxxx>]

oh, i was sure you were going to say windows :)

I suspect the first thing you should do it update to 7.6.3, there were
a couple of edge case issues addressed can you could be subject to one
of those...if you still get the behavior let us know as it could be
another edge case

jesse

--
jesse mcconnell
jesse.mcconnell@xxxxxxxxx


On Wed, May 2, 2012 at 3:39 PM, Mennonite _ <mennonite@xxxxxxxxxxx> wrote:
> CentOS 5.5 64-bit
>
>
> ________________________________
>
>
>> what operative system?
>
> ________________________________
>
> Jetty version: 7.6.2
> Java version: 1.6.0 U30
>
> Every once in a while, Jetty all but completely stops responding to
> HTTP/HTTPS connections. I got a Java thread dump, and tracing through the
> locks, it looks like there's a single
> org.eclipse.jetty.io.nio.ChannelEndPoint.close() which, via locks, seems to
> block all other close()s, bringing everything to a grinding halt.
>
> There are a ton of threads, all with very similar/identical stack traces,
> waiting for one lock:
> "qtp683347213-21440" prio=10 tid=0x00002aaab57ac000 nid=0x3172 waiting for
> monitor entry [0x0000000042d8f000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.nio.channels.spi.AbstractSelector.cancel(Unknown Source)
>         - waiting to lock <0x00000006e659db48> (a java.util.HashSet)
>         at java.nio.channels.spi.AbstractSelectionKey.cancel(Unknown Source)
>         - locked <0x00000007398808d0> (a sun.nio.ch.SelectionKeyImpl)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:691)
>         at
> org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
>         at
> org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
>         at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
> Which is held by this thread, which is waiting for another lock:
> "qtp683347213-90 Selector7" prio=10 tid=0x000000004f5c3000 nid=0x7773
> waiting for monitor entry [0x0000000041bdb000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at sun.nio.ch.SocketChannelImpl.kill(Unknown Source)
>         - waiting to lock <0x00000007398806a0> (a java.lang.Object)
>         at sun.nio.ch.EPollSelectorImpl.implDereg(Unknown Source)
>         at sun.nio.ch.SelectorImpl.processDeregisterQueue(Unknown Source)
>         - locked <0x00000006e659db48> (a java.util.HashSet)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
>         - locked <0x00000006e659e300> (a sun.nio.ch.Util$2)
>         - locked <0x00000006e659e2f0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00000006e659daf0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.selectNow(Unknown Source)
>         at
> org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)
>         at
> org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
> Which is held by this thread:
> "qtp683347213-19755" prio=10 tid=0x000000005005b800 nid=0x7f5e runnable
> [0x00000000440a3000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.FileDispatcher.preClose0(Native Method)
>         at sun.nio.ch.SocketDispatcher.preClose(Unknown Source)
>         at sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel(Unknown
> Source)
>         - locked <0x00000007398806a0> (a java.lang.Object)
>         at
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(Unknown
> Source)
>         at java.nio.channels.spi.AbstractInterruptibleChannel.close(Unknown
> Source)
>         - locked <0x000000073987aea0> (a java.lang.Object)
>         at
> org.eclipse.jetty.io.nio.ChannelEndPoint.close(ChannelEndPoint.java:205)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:700)
>         at
> org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.close(SslConnection.java:652)
>         at
> org.eclipse.jetty.server.AsyncHttpConnection.onInputShutdown(AsyncHttpConnection.java:194)
>         at
> org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:210)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
>         at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
>         at java.lang.Thread.run(Unknown Source)
>
>
> This happens very sporadically, and so I can't reproduce it on demand. I'd
> expect that there are certain environmental factors that cause this, but I
> don't know what those are.
>
> Any thoughts about this?
>
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>