[jetty-users] Growing number of used sockets

[Hanno Wiegard <hwiegard@xxxxxxxxx>]

Hi,

we just migrated from Jetty 7 to Jetty 9.0.3. Now we quite frequently encounter "Too many open files" exceptions (on linux, Ubuntu 12.04, Oracle Java 1.7.0_21 32bit). After looking at the file handle list we see hundreds of sockets for the process as shown here

java    29113 hwiegard  870u  unix 0x00000000      0t0 3521865 socket
java    29113 hwiegard  871u  unix 0x00000000      0t0 3521865 socket
java    29113 hwiegard  872u  unix 0x00000000      0t0 3521865 socket

At the same time we only see one listener

java    29113 hwiegard  155u  IPv4    3521904      0t0     TCP *:8099 (LISTEN)

and there is no traffic on the server. As soon as we start sending requests to the server the number of sockets goes down to 2 (or maybe 3) and stays in a reasonable range (approx. number of concurrent requests). But after stopping all requests the number of sockets goes up stepwise (1-10 per second) till a few hundred are reached. The number seems to be not directly related to the number of requests. The number of sockets stays at the same level, even after waiting for 5-10 minutes and netstat showing no TIME_WAIT, FIN_WAIT1 or FIN_WAIT2 connections anymore.

In the servlet uses the startAsync method, processes the request in a separate thread, sends the result and calls complete on the AsyncContext. The servlet is also marked "async-supported" in the web.xml file. As soon as we remove the startAsync/separate thread/complete code the number of sockets does not grow after stopping the requests. And everything seems to be normal as well if we use startAsync in conjunction with Tomcat 7.0.40 or Jetty 7.6.8.

Any idea what could be wrong here or how to track down the problem?

Thanks,
Hanno