| [jetty-users] FW: avoiding earlyEOF |
|
Hey all, Some background: we have an app that consists of webservices that talk to each other. So the pattern of usage is that we have a few components that host a webserver and at the same time have a http client to talk to the
others. We use Jetty webserver and Jetty httpclient for communication (9.4.7.v20170914). The problem that we face is that we regularly see IOExceptions exceptions occurring in the communication between these components. See stacktrace below. These IOExceptions always contain the earlyEOF. A long time ago
I already posted a similar message on this forum (see
https://dev.eclipse.org/mhonarc/lists/jetty-users/msg07965.html). I followed the advice mentioned, upgraded to the latest version and explicitly set a different idleTimeout on both the server and the client. However, that didn’t help. We still see the occasional earlyEOF. What’s worse, we now have another app with more webservices and more rest-communication. In this app, the earlyEOF happens frequently. And this during mild
load tests (not that heavy, so nothing is overloaded), so it’s clear to me that nothing is idle for a second. To get around the problem, as a test, we tried to replace the JettyClient with Apache HTTP client. There the problem is less of an issue, but we still
see that REST calls failing. Our conclusion is that for some reason the server is closing the connection, but this while the client is still reading from the inputstream that contains the response body.
Does anyone have an idea how we can further diagnose and fix this? Here’s a typical stacktrace: Caused by: java.io.EOFException:
HttpConnectionOverHTTP@516b4fbd(l:/10.58.234.140:34446 <-> r:be-pom-node-04.clear2pay.com/10.58.234.175:28080,closed=false)=>HttpChannelOverHTTP@125fa92a(exchange=HttpExchange@71e96cb2
req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@71a5f017(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@7a384f39{s=START}],recv=HttpReceiverOverHTTP@15c1bfad(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0 of -1}]]<-SocketChannelEndPoint@2f77cd5b{be-pom-node-04.clear2pay.com/10.58.234.175:28080<->/10.58.234.140:34446,ISHUT,fill=-,flush=-,to=1/0}{io=1/0,kio=1,kro=1}->HttpConnectionOverHTTP@516b4fbd(l:/10.58.234.140:34446
<-> r:be-pom-node-04.clear2pay.com/10.58.234.175:28080,closed=false)=>HttpChannelOverHTTP@125fa92a(exchange=HttpExchange@71e96cb2 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@71a5f017(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@7a384f39{s=START}],recv=HttpReceiverOverHTTP@15c1bfad(rsp=IDLE,failure=null)[HttpParser{s=CLOSED,0
of -1}]] at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF(HttpReceiverOverHTTP.java:320) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1419) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown(HttpReceiverOverHTTP.java:196) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:143) at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:70) at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:130) at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:116) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:104) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:243) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:679) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:597) thanks, Bert |