| Re: [jetty-dev] A deadlock when use websocket in jetty7.6.8 / jre1.6.0_29 |
Dear all,
I found a deadlock recently in my product, and I'm using jetty7.6.8 and jre1.6.0_29.
I was using "atmosphere-framework" to implement a chat tool, and I found the server have no response sometime because of a deadlock.
The link which described the problem:
https://groups.google.com/forum/#!topic/atmosphere-framework/dt0YyzmCojQ
and I've copied and attached the log of deadlock as the end of the mail.
I think it is probably a jetty's bug, and I hope update jetty to a new version may solve it.
The problem is that I searched and found no issues/changelogs which tells which version should I use.
So I wondering is that caused by jetty? if so, can I update jetty solve this problem? and which version should I use?(the lower the better),
And if I have to use jetty8/9, must I update the jre to 1.7? (updating jre may cause a big environment change, I'm afraid of causing other problems in my product)
Any help would be appreciated, thanks so much.
LOG By TDA tool:Found one Java-level deadlock"qtp12146666-179":waiting to lock monitor 0x1812e60c (object 0x08e5c550, a org.atmosphere.cpr.AtmosphereResourceImpl), which is held by "qtp12146666-155""qtp12146666-155":waiting to lock monitor 0x17df019c (object 0x08e5c5b0, a org.eclipse.jetty.io.nio.SelectChannelEndPoint), which is held by "qtp12146666-21 Selector1""qtp12146666-21 Selector1":waiting to lock monitor 0x17ec7c3c (object 0x08e5c658, a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455), which is held by "qtp12146666-155"
Java stack information for the threads listed above"qtp12146666-179":at org.atmosphere.util.SimpleBroadcaster.queueWriteIO(SimpleBroadcaster.java:187) - waiting to lock <0x08e5c550> (a org.atmosphere.cpr.AtmosphereResourceImpl) at org.atmosphere.cpr.DefaultBroadcaster.deliverPush(DefaultBroadcaster.java:817) at org.atmosphere.cpr.DefaultBroadcaster.push(DefaultBroadcaster.java:710) at org.atmosphere.util.SimpleBroadcaster.broadcast(SimpleBroadcaster.java:125) at xxxxxxxxxxxx.ConnectionManager.deliver(ConnectionManager.java:108) at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:256) at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:166) at org.atmosphere.container.JettyWebSocketUtil.doService(JettyWebSocketUtil.java:78) at org.atmosphere.container.JettyAsyncSupportWithWebSocket.service(JettyAsyncSupportWithWebSocket.java:69) at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1342) at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:219) at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:183) at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101) at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:178) at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:167) at org.atmosphere.container.JettyWebSocketHandler.onMessage(JettyWebSocketHandler.java:98) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:845) at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:359) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:235) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:662)"qtp12146666-155":at org.eclipse.jetty.io.nio.SelectChannelEndPoint.dispatch(SelectChannelEndPoint.java:212) - waiting to lock <0x08e5c5b0> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:415) - locked <0x08e5c600> (a org.eclipse.jetty.io.nio.SslConnection) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:673) at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.flushBuffer(WebSocketGeneratorRFC6455.java:207) - locked <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455) at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.addFrame(WebSocketGeneratorRFC6455.java:174) - locked <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.sendMessage(WebSocketConnectionRFC6455.java:449) at org.atmosphere.container.version.Jetty8WebSocket.write(Jetty8WebSocket.java:87) at org.atmosphere.container.version.Jetty8WebSocket.write(Jetty8WebSocket.java:34) at org.atmosphere.cpr.AtmosphereResponse$4.write(AtmosphereResponse.java:703) at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:136) at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:1093) at org.atmosphere.util.SimpleBroadcaster.prepareInvokeOnStateChange(SimpleBroadcaster.java:177) at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:998) at org.atmosphere.util.SimpleBroadcaster.queueWriteIO(SimpleBroadcaster.java:187) - locked <0x08e5c550> (a org.atmosphere.cpr.AtmosphereResourceImpl) at org.atmosphere.cpr.DefaultBroadcaster.deliverPush(DefaultBroadcaster.java:817) at org.atmosphere.cpr.DefaultBroadcaster.push(DefaultBroadcaster.java:710) at org.atmosphere.util.SimpleBroadcaster.broadcast(SimpleBroadcaster.java:125) at xxxxxxxxxxxx.ConnectionManager.deliver(ConnectionManager.java:108) at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:256) at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:166) at org.atmosphere.container.JettyWebSocketUtil.doService(JettyWebSocketUtil.java:78) at org.atmosphere.container.JettyAsyncSupportWithWebSocket.service(JettyAsyncSupportWithWebSocket.java:69) at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1342) at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:219) at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:183) at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101) at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:178) at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:167) at org.atmosphere.container.JettyWebSocketHandler.onMessage(JettyWebSocketHandler.java:98) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameHandler.onFrame(WebSocketConnectionRFC6455.java:845) at org.eclipse.jetty.websocket.WebSocketParserRFC6455.parseNext(WebSocketParserRFC6455.java:359) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.handle(WebSocketConnectionRFC6455.java:235) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:662)"qtp12146666-21 Selector1":at org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455.addFrame(WebSocketGeneratorRFC6455.java:79) - waiting to lock <0x08e5c658> (a org.eclipse.jetty.websocket.WebSocketGeneratorRFC6455) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.closeOut(WebSocketConnectionRFC6455.java:392) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.close(WebSocketConnectionRFC6455.java:499) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455$WSFrameConnection.close(WebSocketConnectionRFC6455.java:626) at org.atmosphere.container.version.Jetty8WebSocket.close(Jetty8WebSocket.java:154) at org.atmosphere.cpr.AtmosphereResponse$2.close(AtmosphereResponse.java:597) at org.atmosphere.cpr.AsynchronousProcessor.cancelled(AsynchronousProcessor.java:584) - locked <0x08e5c818> (a org.atmosphere.cpr.AtmosphereRequest) at org.atmosphere.cpr.AsynchronousProcessor$AsynchronousProcessorHook.closed(AsynchronousProcessor.java:652) at org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:265) at org.atmosphere.container.JettyWebSocketHandler.onClose(JettyWebSocketHandler.java:114) at org.eclipse.jetty.websocket.WebSocketConnectionRFC6455.onClose(WebSocketConnectionRFC6455.java:312) at org.eclipse.jetty.websocket.WebSocketServletConnectionRFC6455.onClose(WebSocketServletConnectionRFC6455.java:63) at org.eclipse.jetty.io.nio.SslConnection.onClose(SslConnection.java:242) at org.eclipse.jetty.server.AbstractConnector.connectionClosed(AbstractConnector.java:1145) at org.eclipse.jetty.server.nio.SelectChannelConnector.endPointClosed(SelectChannelConnector.java:274) at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.endPointClosed(SelectChannelConnector.java:301) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.destroyEndPoint(SelectorManager.java:848) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.doUpdateKey(SelectChannelEndPoint.java:585) - locked <0x08e5c5b0> (a org.eclipse.jetty.io.nio.SelectChannelEndPoint) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:463) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:662)