Dear all,
I found a deadlock recently in my product, and I'm using jetty7.6.8 and jre1.6.0_29.
I was using "atmosphere-framework" to implement a chat tool, and I found the server have no response sometime because of a deadlock.
The link which described the problem:
https://groups.google.com/forum/#!topic/atmosphere-framework/dt0YyzmCojQ
and I've copied and attached the log of deadlock as the end of the mail.
I think it is probably a jetty's bug, and I hope update jetty to a new version may solve it.
The problem is that I searched and found no issues/changelogs which tells which version should I use.
So I wondering is that caused by jetty? if so, can I update jetty solve this problem? and which version should I use?(the lower the better),
And if I have to use jetty8/9, must I update the jre to 1.7? (updating jre may cause a big environment change, I'm afraid of causing other problems in my product)
Any help would be appreciated, thanks so much.
LOG By TDA tool:
Found one Java-level deadlock
"qtp12146666-179":
waiting to lock monitor 0x1812e60c (object 0x08e5c550, a org.atmosphere.cpr.AtmosphereResourceImpl),
which is held by "qtp12146666-155"
"qtp12146666-155":
waiting to lock monitor 0x17df019c (object 0x08e5c5b0, a org.eclipse.jetty.io.nio.SelectChannelEndPoint),
which is held by "qtp12146666-21 Selector1"
"qtp12146666-21 Selector1":
waiting to lock monitor 0x17ec7c3c (object 0x08e5c658, a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455),
which is held by "qtp12146666-155"
Java stack information for the threads listed above
"qtp12146666-179":
at org.atmosphere.util.SimpleBroadcaster.queueWriteIO(SimpleBroadcaster.java:187)
- waiting to lock <0x08e5c550> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.DefaultBroadcaster.deliverPush(DefaultBroadcaster.java:817)
at org.atmosphere.cpr.DefaultBroadcaster.push(DefaultBroadcaster.java:710)
at org.atmosphere.util.SimpleBroadcaster.broadcast(SimpleBroadcaster.java:125)
at xxxxxxxxxxxx.ConnectionManager.deliver(ConnectionManager.java:108)
at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:256)
at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:166)
at org.atmosphere.container.JettyWebSocketUtil.doService(JettyWebSocketUtil.java:78)
at org.atmosphere.container.JettyAsyncSupportWithWebSocket.service(JettyAsyncSupportWithWebSocket.java:69)
at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1342)
at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:219)
at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:183)
at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101)
at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:178)
at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:167)
at org.atmosphere.container.JettyWebSocketHandler.onMessage(JettyWebSocketHandler.java:98)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:845)
at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:359)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:235)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:662)
"qtp12146666-155":
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:212)
- waiting to lock <0x08e5c5b0> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:415)
- locked <0x08e5c600> (a org.eclipse.jetty.io.nio.SslConnection)
at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:673)
at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.flushBuffer(WebSocketGeneratorRFC6455.java:207)
- locked <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455)
at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.addFrame(WebSocketGeneratorRFC6455.java:174)
- locked <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.sendMessage(WebSocketConnectionRFC6455.java:449)
at org.atmosphere.container.version.Jetty8WebSocket.write(Jetty8WebSocket.java:87)
at org.atmosphere.container.version.Jetty8WebSocket.write(Jetty8WebSocket.java:34)
at org.atmosphere.cpr.AtmosphereResponse$4.write(AtmosphereResponse.java:703)
at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:136)
at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1093)
at org.atmosphere.util.SimpleBroadcaster.prepareInvokeOnStateChange(SimpleBroadcaster.java:177)
at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:998)
at org.atmosphere.util.SimpleBroadcaster.queueWriteIO(SimpleBroadcaster.java:187)
- locked <0x08e5c550> (a org.atmosphere.cpr.AtmosphereResourceImpl)
at org.atmosphere.cpr.DefaultBroadcaster.deliverPush(DefaultBroadcaster.java:817)
at org.atmosphere.cpr.DefaultBroadcaster.push(DefaultBroadcaster.java:710)
at org.atmosphere.util.SimpleBroadcaster.broadcast(SimpleBroadcaster.java:125)
at xxxxxxxxxxxx.ConnectionManager.deliver(ConnectionManager.java:108)
at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:256)
at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:166)
at org.atmosphere.container.JettyWebSocketUtil.doService(JettyWebSocketUtil.java:78)
at org.atmosphere.container.JettyAsyncSupportWithWebSocket.service(JettyAsyncSupportWithWebSocket.java:69)
at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1342)
at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:219)
at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:183)
at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101)
at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:178)
at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:167)
at org.atmosphere.container.JettyWebSocketHandler.onMessage(JettyWebSocketHandler.java:98)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:845)
at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:359)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:235)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:662)
"qtp12146666-21 Selector1":
at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.addFrame(WebSocketGeneratorRFC6455.java:79)
- waiting to lock <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.closeOut(WebSocketConnectionRFC6455.java:392)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.close(WebSocketConnectionRFC6455.java:499)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.close(WebSocketConnectionRFC6455.java:626)
at org.atmosphere.container.version.Jetty8WebSocket.close(Jetty8WebSocket.java:154)
at org.atmosphere.cpr.AtmosphereResponse$2.close(AtmosphereResponse.java:597)
at org.atmosphere.cpr.AsynchronousProcessor.cancelled(AsynchronousProcessor.java:584)
- locked <0x08e5c818> (a org.atmosphere.cpr.AtmosphereRequest)
at org.atmosphere.cpr.AsynchronousProcessor$AsynchronousProcessorHook.closed(AsynchronousProcessor.java:652)
at org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:265)
at org.atmosphere.container.JettyWebSocketHandler.onClose(JettyWebSocketHandler.java:114)
at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.onClose(WebSocketConnectionRFC6455.java:312)
at org.eclipse.jetty.websocket.WebSocketServletConnectionRFC6455.onClose(WebSocketServletConnectionRFC6455.java:63)
at org.eclipse.jetty.io.nio.SslConnection.onClose(SslConnection.java:242)
at org.eclipse.jetty.server.AbstractConnector.connectionClosed(AbstractConnector.java:1145)
at org.eclipse.jetty.server.nio.SelectChannelConnector.endPointClosed(SelectChannelConnector.java:274)
at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.endPointClosed(SelectChannelConnector.java:301)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.destroyEndPoint(SelectorManager.java:848)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.doUpdateKey(SelectChannelEndPoint.java:585)
- locked <0x08e5c5b0> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:463)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
at java.lang.Thread.run(Thread.java:662)