[jetty-users] Jetty 8.1.10 hangs in org.eclipse.jetty.http.HttpGenerator

[Bo Li <bo.li@xxxxxxxxxxxxx>]

Hello

We are using Jetty 8.1.10 with Spring. Jetty is configured for SSL using SslSelectChannelConnector. During load testing, the client is stopped and started multiple times. While this went without a problem most of the time, we occasionally saw the server stop responding all together. Taking a jstack reveals Jetty is stuck trying to send a 100 Continue response back to the client:

"qtp350942644-247" prio=10 tid=0x00007f5ab0315000 nid=0x5470 waiting on condition [0x00007f5ae870e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.eclipse.jetty.http.HttpGenerator.send1xx(HttpGenerator.java:336)
        at org.eclipse.jetty.server.AbstractHttpConnection.getInputStream(AbstractHttpConnection.java:335)
        at org.eclipse.jetty.server.Request.getInputStream(Request.java:622)
        at org.eclipse.jetty.server.Request.extractParameters(Request.java:352)
        at org.eclipse.jetty.server.Request.getParameterValues(Request.java:808)
        at javax.servlet.ServletRequestWrapper.getParameterValues(ServletRequestWrapper.java:207)
        at javax.servlet.ServletRequestWrapper.getParameterValues(ServletRequestWrapper.java:207)
        at org.springframework.web.context.request.ServletWebRequest.getParameterValues(ServletWebRequest.java:117)
        at org.springframework.web.method.annotation.RequestParamMethodArgumentResolver.resolveName(RequestParamMethodArgumentResolver.java:172)
        at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:85)
        at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:77)
        at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:162)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:123)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:746)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:687)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:915)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:822)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:796)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1448)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:144)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)

All clients have already disconnected from the server when this was taken. Netstat reveals 577 connections in the CLOSE_WAIT state waiting for the application to process close the socket. It seems like Jetty never noticed the connection is now closed and to stop trying to send the 100.

Is this a known bug with Jetty or a configuration issue?

Thanks for your help!
Bo