[jetty-users] jetty hot/stuck thread

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

[jetty-users] jetty hot/stuck thread

From: Lu Tahmazyan <ltahmazyan@xxxxxxxxxxxxx>
Date: Wed, 1 May 2019 10:30:51 -0700
Delivered-to: jetty-users@xxxxxxxxxxx
List-archive: <https://www.eclipse.org/mailman/private/jetty-users>
List-help: <mailto:jetty-users-request@eclipse.org?subject=help>
List-subscribe: <https://www.eclipse.org/mailman/listinfo/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://www.eclipse.org/mailman/options/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=unsubscribe>

Hello there.

We have been running a reverse proxy based on Jetty's AsyncMiddleManServlet for years without any major issue. However, we started noticing that once in a great while a thread gets stuck in a hot loop. The only way to recover it is to bounce the application.

"Hot" part of the code seems to be around these methods calls (this was verified via JMX CPU sampling)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117)

I've taken heap dumps and thread-dumps but don't seem to be able to figure out what is going on. As of now, this is not a concern for us, given the frequency is extremely low (a stuck thread a week). Just want to reach out to the group. Any suggestions would be appreciated.

Thanks,

-Lu

Details:

Jetty version: 9.4.14.v20181114

JDK version: 10.0.2

Comlete stack traces:

Incident #1

2019-04-30 17:18:55

"default-pool-36369" - Thread t@36369

java.lang.Thread.State: RUNNABLE

at app//org.eclipse.jetty.server.HttpConnection$SendCallback.access$300(HttpConnection.java:678)

at app//org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:536)

at app//org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:841)

at app//org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:891)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor$GzipBufferCB.process(GzipHttpOutputInterceptor.java:398)

at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)

at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:244)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:105)

at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240)

at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:216)

at app//org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:298)

at app//org.eclipse.jetty.server.Response.closeOutput(Response.java:1044)

at app//org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:507)

at app//org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305)

at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:109)

at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

at java.base@10.0.2/java.lang.Thread.run(Thread.java:844)

Locked ownable synchronizers:

- None

Incident #2

2019-04-22 13:41:59

"default-pool-76" - Thread t@76

java.lang.Thread.State: RUNNABLE

at app//org.eclipse.jetty.util.thread.Locker.lock(Locker.java:46)

at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:195)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.gzip(GzipHttpOutputInterceptor.java:148)

at app//org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:117)

at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:240)

at app//org.eclipse.jetty.server.HttpOutput$AsyncWrite.process(HttpOutput.java:1243)

at app//org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)

at app//org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)

at app//org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:471)

at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.write(AsyncMiddleManServlet.java:219)

at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet.writeProxyResponseContent(AsyncMiddleManServlet.java:201)

at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyWriter.onWritePossible(AsyncMiddleManServlet.java:651)

at app//org.eclipse.jetty.proxy.AsyncMiddleManServlet$ProxyResponseListener.onContent(AsyncMiddleManServlet.java:499)

at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:133)

at app//org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:125)

at app//org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:338)

at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:283)

at app//org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1787)

at app//org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1517)

at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:172)

at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:135)

at app//org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73)

at app//org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133)

at app//org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155)

at app//org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)

at app//org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)

at app//org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)

at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)

at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)

at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)

at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)

at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)

at app//com.opentable.metrics.JettyServerMetricsConfiguration$OTQueuedThreadPool.runJob(JettyServerMetricsConfiguration.java:110)

at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)

at java.base@10.0.2/java.lang.Thread.run(Thread.java:844)

Locked ownable synchronizers:

- locked <7ee79395> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

Incident #3

2019-04-17 10:03:03

"default-pool-74" - Thread t@74

java.lang.Thread.State: RUNNABLE

at java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1237)

at java.base@10.0.2/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)