Hi,
We have a custom implementation of Connection and ConnectionFactory for a non-http protocol. (Jetty 10.0.14)
In general, it works great even when many requests are happening concurrently. We've performed extensive stress testing and haven't had any issues.
However, we hit an issue recently where some poorly written client code produced a storm of connection open requests (TLS is enabled). It did this for about an hour or so. After the storm ended, we found that some connections were left stuck in the CLOSE_WAIT state and are seemingly stuck there indefinitely.
netstat -all --program | grep 12345
tcp6 0 0 host:12345
10.0.0.5:20390 CLOSE_WAIT 5727/java
tcp6 0 0 host:12345
10.0.0.5:14840 CLOSE_WAIT 5727/java
tcp6 1 0 host:12345
10.0.0.5:8830 CLOSE_WAIT 5727/java
...
We were also able to see the result of calling ServerConnector.getConnectedEndPoints(). The result similarly showed the stuck connections.
We've been able to reproduce in a development environment once by creating a storm of open/close requests and letting it run for about 45 minutes.
Does anyone know what might cause this to occur so rarely? Could it be a bug in our Connection implementation? Or in Jetty itself?
Thanks,
Josh
