Re: [jetty-users] Problem with ClosedByInterruptException

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Sat, Nov 5, 2016 at 12:37 AM, John Gardiner Myers
<jgmyers@xxxxxxxxxxxxxx> wrote:
> I've been having problems with some Jetty client (embedded) instances
> encounter frequent ClosedByInterruptException errors attempting to query a
> different service.
>
> I have four instances in our testing environment that have gotten into this
> state. Three are Jetty 9.3.7, one is Jetty 9.3.6. All are either Java 8u72
> or 8u73.
>
> The stack trace is:
>
> java.nio.channels.ClosedByInterruptException
>         at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>         at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:753)
>         at
> org.eclipse.jetty.io.SelectorManager.finishConnect(SelectorManager.java:337)
>         at
> org.eclipse.jetty.io.ManagedSelector.processConnect(ManagedSelector.java:341)
>         at
> org.eclipse.jetty.io.ManagedSelector.access$900(ManagedSelector.java:56)
>         at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:278)
>         at
> org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:170)
>         at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume
> .java:162)
>         at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Thread.java:745)
>
> Looking at AbstractInterruptibleChannel.end(boolean) for 8u92, I see what
> appears to be a bug in NIO. Once an AbstractInterruptibleChannel interrupts
> on a given Thread, all subsequent calls to #end(boolean) from that thread
> will throw ClosedByInterruptException. The line "interrupted = null;" in
> that method has no effect and probably should have been "this.interrupted =
> null;"
>
> I'm not sure what the source of interruptions are, but I do see
> TimeoutException("DNS timeout " + getTimeout() + " ms") exceptions from
> within SocketAddressResolver.Async.resolve() being logged.

The next line after the promise is failed in
SocketAddressResolver.Async.resolve(), Jetty interrupts the thread
that triggered the DNS lookup, in the attempt to return it from what
it was doing back into the pool.
I did not know about this JDK bug.

> So my suspicion is that DNS timeouts might be causing worker threads to get
> into a state where they cannot complete opening connections.
>
> I'd appreciate any suggestions for how to diagnose the problem further or
> otherwise proceed.

I think you have identified the issue.
AFAIK, DNS lookups in Java don't timeout. If your DNS lookup hangs
forever, that thread is forever stuck there.
Jetty adds a DNS timeout, but that is not enough, since the thread
doing the DNS lookup will still be stuck.
So Jetty tries to interrupt it.
However, this hits the JDK bug.

We can make the interruption configurable (or perhaps removing it).

Would you be able to modify the code (or pass your own implementation
of SocketAddressResolver - basically the implementation in Async
without the interruption) and verify that without interruption things
work ?
What would the DNS thread do after the timeout expire ?

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.