The service is pretty simple and is going to be called once a day for integration purposes so the configuration is fairly modest: 1 acceptor, 2 selectors and 2 threads for request handlers. Omitting irrelevant code it's roughly going like this:
The problem occurs pretty rarely. Here what we found out so far:
1) The service stops closing sockets immediately after two consequent exceptions:
[2014-04-14 12:29:50,552] DEBUG [qtp782189620-16 - /ping] write exception
org.eclipse.jetty.io.EofException
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:189)
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:335)
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125)
at org.eclipse.jetty.server.HttpConnection$CommitCallback.process(HttpConnection.java:555)
at org.eclipse.jetty.util.IteratingCallback.processIterations(IteratingCallback.java:166)
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:126)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:296)
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:715)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:751)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:130)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:124)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:356)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:154)
at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:307)
at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:261)
at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:276)
at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:835)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:411)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:645)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:373)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:711)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1644)
at ru.yandex.bayan2.integration.servlet.LoggingFilter.doFilter(LoggingFilter.java:27)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1615)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1112)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:479)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1046)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:462)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:281)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
at org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
at java.lang.Thread.run(Thread.java:744)
but sockets never get closed.
3) org.eclipse.jetty.io.SelectorManager reports a growing number of selected events:
4) Sockets left in CLOSE_WAIT state.
As a result we have a growing number of sockets and finally "Too many open files" exception.