I have been trying different Jetty client versions for my http client and I have been hitting some problems with high load (several thousands requests per minutes).
I use 2 client selectors (setSelectors), 16384 as setInputBufferSize. Same number as setRequestBufferSize, setResponseBufferSize.
2024 as setMaxConnectionsPerDestination.
setConnectBlocking set to true.
For response I use BufferingResponseListener with 8388608 as param in constructor (8 MB).
The problems are:
9.3.9.v20160517:
AsynchronousCloseException
2016-11-26 23:20:06,638 [asyncCallback11] WARN http.HttpDataAccess - Exception details:
java.nio.channels. AsynchronousCloseException: null
at org.eclipse.jetty.http2. client.http. HttpConnectionOverHTTP2.close( HttpConnectionOverHTTP2.java: 90) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. client.http. HttpClientTransportOverHTTP2. onClose( HttpClientTransportOverHTTP2. java:160) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. client.http. HttpClientTransportOverHTTP2$ SessionListenerPromise. onClose( HttpClientTransportOverHTTP2. java:216) ~[http2-http-client-transport- 9.3.14.v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Session.notifyClose( HTTP2Session.java:1061) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Session.onGoAway( HTTP2Session.java:424) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.BodyParser. notifyGoAway(BodyParser.java: 187) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.GoAwayBodyParser. onGoAway(GoAwayBodyParser. java:169) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.GoAwayBodyParser.parse( GoAwayBodyParser.java:139) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.Parser.parseBody( Parser.java:182) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. parser.Parser.parse(Parser. java:110) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Connection$HTTP2Producer. produce(HTTP2Connection.java: 199) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.util.thread. strategy.ProduceConsume. execute(ProduceConsume.java: 69) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.http2. HTTP2Connection.onFillable( HTTP2Connection.java:110) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.http2. HTTP2Connection$FillCallback. succeeded(HTTP2Connection. java:251) ~[http2-common-9.3.14. v20161028.jar:9.3.14. v20161028]
at org.eclipse.jetty.io. FillInterest.fillable( FillInterest.java:95) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io.ssl. SslConnection.onFillable( SslConnection.java:202) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. AbstractConnection$ ReadCallback.succeeded( AbstractConnection.java:273) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. FillInterest.fillable( FillInterest.java:95) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.io. SelectChannelEndPoint$2.run( SelectChannelEndPoint.java:93) ~[jetty-io-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume. executeProduceConsume( ExecuteProduceConsume.java: 303) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume. produceConsume( ExecuteProduceConsume.java: 148) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. strategy. ExecuteProduceConsume.run( ExecuteProduceConsume.java: 136) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. QueuedThreadPool.runJob( QueuedThreadPool.java:671) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
at org.eclipse.jetty.util.thread. QueuedThreadPool$2.run( QueuedThreadPool.java:589) ~[jetty-util-9.3.14.v20161028. jar:9.3.14.v20161028]
9.3.10.v20160621 , 9.3.11.v20160721 - same
9.3.12.v20160915 - onComplete callback of BufferingResponseListener is called more than once.
9.3.13.v20161014 -
AsynchronousCloseException, as previous versions.
And in addition something new
2016-11-26 23:21:50,672 [asyncCallback10] WARN http.HttpDataAccess - Exception details:
java.lang. ArrayIndexOutOfBoundsException : 16384
at org.eclipse.jetty.http2.hpack. Huffman.encode(Huffman.java: 466) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. Huffman.encode(Huffman.java: 414) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encodeValue( HpackEncoder.java:370) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encode( HpackEncoder.java:325) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2.hpack. HpackEncoder.encode( HpackEncoder.java:165) ~[http2-hpack-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.HeadersGenerator. generateHeaders( HeadersGenerator.java:72) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.HeadersGenerator. generate(HeadersGenerator. java:56) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. generator.Generator.control( Generator.java:80) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. HTTP2Session$ControlEntry. generate(HTTP2Session.java: 1122) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. HTTP2Flusher.process( HTTP2Flusher.java:231) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.util. IteratingCallback.processing( IteratingCallback.java:241) ~[jetty-util-9.3.13.v20161014. jar:9.3.13.v20161014]
at org.eclipse.jetty.util. IteratingCallback.iterate( IteratingCallback.java:224) ~[jetty-util-9.3.13.v20161014. jar:9.3.13.v20161014]
at org.eclipse.jetty.http2. HTTP2Session.newStream( HTTP2Session.java:496) ~[http2-common-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpSenderOverHTTP2. sendHeaders( HttpSenderOverHTTP2.java:87) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpSender.send(HttpSender. java:204) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpChannelOverHTTP2.send( HttpChannelOverHTTP2.java:87) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpConnection.send( HttpConnection.java:207) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpConnectionOverHTTP2.send( HttpConnectionOverHTTP2.java: 64) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpDestinationOverHTTP2.send( HttpDestinationOverHTTP2.java: 37) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.http2. client.http. HttpDestinationOverHTTP2.send( HttpDestinationOverHTTP2.java: 27) ~[http2-http-client-transport- 9.3.13.v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. MultiplexHttpDestination. process( MultiplexHttpDestination.java: 143) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. MultiplexHttpDestination.send( MultiplexHttpDestination.java: 72) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpDestination.send( HttpDestination.java:202) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpClient.send(HttpClient. java:540) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpRequest.send(HttpRequest. java:693) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
at org.eclipse.jetty.client. HttpRequest.send(HttpRequest. java:677) ~[jetty-client-9.3.13. v20161014.jar:9.3.13. v20161014]
9.3.14.v20161028 - AsynchronousCloseException
Do you have any advice why my clients gets those GO_AWAY message leading to AsynchronousCloseException? This is the main issue. It doesn't happen all the time but still a substantial number of requests
Maxim
