[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jetty-users] Jetty EatWhatYouKill - Deadlock or something!
|
Hi,
On Fri, Jan 5, 2018 at 6:50 PM, Sarath Prabath Redlapalli Jaya
<harrysarath2010@xxxxxxxxx> wrote:
> Hi,
>
> We've been running a java server with Jetty(Dropwizard) and the load of
> system per node would be ~ 10k QPS as of now.
>
> We ran Load Test, Connection Test and Peak Test(No Stress test yet.), but
> couldn't reproduce what's happening on production when we chose to go live
>
> Below is the error that spits out continuosly
>
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.QueuedThreadPool
> org.eclipse.jetty.util.thread.QueuedThreadPool@dw{STARTED,4000<=4000<=4000,i=0,q=10000}
> rejected
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
> [2018-01-05 16:05:00,874] [] [WARN] MSG - o.e.j.u.t.s.EatWhatYouKill
> java.util.concurrent.RejectedExecutionException:
> CEP:SocketChannelEndPoint@62241285{/127.0.0.1:49625<->/127.0.0.1:7000,OPEN,fill=FI,flush=-,to=0/30000}{io=1/0,kio=1,kro=1}->HttpConnection@23c692fe[p=HttpParser{s=START,0
> of
> 0},g=HttpGenerator@544fccad{s=START}]=>HttpChannelOverHttp@2c630bfe{r=0,c=false,a=IDLE,uri=null}:runFillable:BLOCKING
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.execute(QueuedThreadPool.java:385)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:249)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597)
> at java.lang.Thread.run(Thread.java:745)
>
> Tried to find an answer, but couldn't. Can anyone help. At this point the
> server stops accepting new connections and doesn't serve new http requests.
>
> But Jetty QueuedThreadPool Metrics show that thread Utilization is 100%
What Jetty version ?
Have you tried 9.4.8 ?
Are you using a bounded thread pool ?
If it replicates with 9.4.8, can you take a server dump as explained here:
https://www.eclipse.org/jetty/documentation/9.4.x/jetty-dump-tool.html
--
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.
