I'm using CXF web services with embedded Jetty over SSL.
I have a CXF client which calls a large number of web services synchronically one after the other.
At some point, the client request isn't served and after a long time it gets "Unexpected end of file from server".
After checking with tcpdump, I've verified that the client request was sent on the socket around 08:02:10,574.
On the server there are the following logs.
Please notice the long time gap between the two blocks (20 seconds).
It seems that although the client sent the request, the server doesn't read it until a timeout event pops on the server and the connection is closed.
If someone has a direction or technique how to investigate such an issue it will be very helpful.
05:02:10,593 DEBUG org.eclipse.jetty.io.nio.ssl.process:347 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@df74a3bc SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=4,h=0,b=0,c=-1},p=HttpParser{s=0,l=7,c=-2},r=6} NOT_HANDSHAKING filled=0/0 flushed=0/0
05:02:10,593 DEBUG org.eclipse.jetty.server.AsyncHttpConnection.reset:211 [qtp2013547526-59] - Enabled read interest SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=4,h=0,b=0,c=-1},p=HttpParser{s=0,l=7,c=-2},r=6}
05:02:10,593 DEBUG org.eclipse.jetty.io.nio.ssl.process:347 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@df74a3bc SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} NOT_HANDSHAKING filled=0/0 flushed=0/0
05:02:10,594 DEBUG org.eclipse.jetty.io.nio.ssl.handle:203 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] handle SslConnection@df74a3bc SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} progress=false
05:05:30,598 DEBUG org.eclipse.jetty.io.nio.ssl.onIdleExpired:256 [qtp2013547526-59] - onIdleExpired 200004ms on SslConnection@df74a3bc SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6}
05:05:30,598 DEBUG org.eclipse.jetty.io.nio.ssl.shutdownOutput:637 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] ssl endp.oshut SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=false {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6}
05:05:30,598 DEBUG org.eclipse.jetty.io.nio.ssl.process:347 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@df74a3bc SSL NEED_WRAP i/o/u=0/0/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} NEED_WRAP filled=0/0 flushed=0/0
05:05:30,599 DEBUG org.eclipse.jetty.io.nio.ssl.wrap:462 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] wrap CLOSED NEED_UNWRAP consumed=0 produced=37
05:05:30,599 DEBUG org.eclipse.jetty.io.nio.ssl.wrap:503 [qtp2013547526-59] - wrap CLOSE SslConnection@df74a3bc SSL NEED_UNWRAP i/o/u=0/37/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} Status = CLOSED HandshakeStatus = NEED_UNWRAP
05:05:30,599 DEBUG org.eclipse.jetty.io.nio.ssl.process:347 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@df74a3bc SSL NEED_UNWRAP i/o/u=0/0/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} NEED_UNWRAP filled=0/0 flushed=37/0
05:05:30,599 DEBUG org.eclipse.jetty.io.nio.ChannelEndPoint.shutdownChannelOutput:157 [qtp2013547526-59] - oshut SCEP@498cfcfd{l(/127.0.0.1:59396)<->r(/127.0.0.1:19009),s=0,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{SslConnection@df74a3bc SSL NEED_UNWRAP i/o/u=0/0/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6}}
05:05:30,599 DEBUG org.eclipse.jetty.io.nio.ssl.process:347 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@df74a3bc SSL NEED_UNWRAP i/o/u=37/0/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} NEED_UNWRAP filled=37/37 flushed=0/0
05:05:30,600 DEBUG org.eclipse.jetty.io.nio.ssl.unwrap:538 [qtp2013547526-59] - [Session-1, SSL_NULL_WITH_NULL_NULL] unwrap CLOSED NOT_HANDSHAKING consumed=37 produced=0
05:05:30,600 DEBUG org.eclipse.jetty.io.nio.ssl.unwrap:582 [qtp2013547526-59] - unwrap CLOSE SslConnection@df74a3bc SSL NOT_HANDSHAKING i/o/u=0/0/12490 ishut=false oshut=true {AsyncHttpConnection@2b68117f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=-3},r=6} Status = CLOSED HandshakeStatus = NOT_HANDSHAKING
