Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-users] Jetty EatWhatYouKill - Deadlock or something!
Hi,

We've been running a java server with Jetty(Dropwizard) and the load of system per node would be ~ 10k QPS as of now. 

We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but couldn't reproduce what's happening on production when we chose to go live

Below is the error that spits out continuosly

[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000} rejected CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
[2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill 
java.util.concurrent.RejectedExecutionException: CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0 of 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
        at org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
        at java.lang.Thread.run(Thread.java:745)

Tried to find an answer, but couldn't. Can anyone help. At this point the server stops accepting new connections and doesn't serve new http requests. 

But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%

Thanks

Back to the top