Re: [jetty-users] Lockup in jetty-client under low threads / slow destin

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Thu, Jul 28, 2016 at 1:35 AM, John Gardiner Myers
<jgmyers@xxxxxxxxxxxxxx> wrote:
> While investigating reports of requests being starved in the presence of
> other requests to slow destinations, I managed to get embedded jetty-client
> 9.3.10 to lock up pretty hard.
>
> My test case uses two HttpClient instances sharing the same Executor (with 4
> threads), Scheduler, and ByteBufferPool. One HttpClient makes requests to a
> working plain-HTTP destination, the other to a plain-HTTP destination that
> always times out connects.
>
> After a while, all threads block on a Selector.select() with zero keys, out
> of ManagedSelector$SelectorProducer.select(), out of
> ManagedSelector$SelectorProducer.produce(), out of
> ExecuteProduceConsume.produceExecuteConsume(), out of
> ExecuteProduceConsume.produceConsume(). At this point, Jetty will not
> cleanly shut down.
>
> org.eclipse.jetty.client.HttpClient@78dc4696 - STARTED
>  += org.eclipse.jetty.util.thread.Sweeper@1d03459d - STARTED
>  += SslContextFactory@106c710e(null,null) - STARTED
>  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
>  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
>  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
> STARTED
>  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@71aff321 -
> STARTED
>  |   +=
> org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@5ee8e234
> - STARTED
>  |       += org.eclipse.jetty.io.ManagedSelector@59406bd9 id=0 keys=0
> selected=0 id=0
>  |       |   +- sun.nio.ch.KQueueSelectorImpl@65abf03e keys=0
>  |       += org.eclipse.jetty.io.ManagedSelector@2a5795fe id=1 keys=0
> selected=0 id=1
>  |           +- sun.nio.ch.KQueueSelectorImpl@483695b6 keys=0
>  +- org.eclipse.jetty.util.SocketAddressResolver$Async@246f5d6e
>  +=
> HttpDestination[http://localhost:4111]@3d8fe295,queue=2,pool=DuplexConnectionPool[c=2/20,a=0,i=0]
> +- DuplexConnectionPool[c=2/20,a=0,i=0]
>
> org.eclipse.jetty.client.HttpClient@52169758 - STARTED
>  += org.eclipse.jetty.util.thread.Sweeper@17f3461a - STARTED
>  += SslContextFactory@51a5c5e4(null,null) - STARTED
>  +~ http-client-shared{STARTED,4<=4<=4,i=0,q=7} - STARTED
>  +- org.eclipse.jetty.io.MappedByteBufferPool@794e4b46
>  +~ org.eclipse.jetty.util.thread.ScheduledExecutorScheduler@529e10f0 -
> STARTED
>  += org.eclipse.jetty.client.http.HttpClientTransportOverHTTP@5f505f2f -
> STARTED
>  |   +=
> org.eclipse.jetty.client.AbstractHttpClientTransport$ClientSelectorManager@20a52bc4
> - STARTED
>  |       += org.eclipse.jetty.io.ManagedSelector@2b3a50ed id=0 keys=0
> selected=0 id=0
>  |       |   +- sun.nio.ch.KQueueSelectorImpl@70b111fe keys=0
>  |       += org.eclipse.jetty.io.ManagedSelector@24c337f6 id=1 keys=0
> selected=0 id=1
>  |           +- sun.nio.ch.KQueueSelectorImpl@56dd5f46 keys=0
>  +- org.eclipse.jetty.util.SocketAddressResolver$Async@23225908
>  +=
> HttpDestination[http://10.2.45.41:8080]@5db775ed,queue=3,pool=DuplexConnectionPool[c=3/20,a=0,i=0]
>      +- DuplexConnectionPool[c=3/20,a=0,i=0]
>
> Is there anything else I could investigate or produce to make a useful bug
> report?

JVM thread dump ?

Why do you run with only 4 threads ?
It will limit the HttpClient operativity by a lot under load.

You can also try to use different ExecutionStrategies, by setting them
in the transport:
HttpClientTransportOverHTTP.setExecutionStrategyFactory(...).

In summary, running with a thread pool with 4 threads is discouraged,
unless you have very good reasons to do so.
If you really, really need to run with these small number of threads,
you have to configure HttpClient to support this mode of operation:
reduce connection pool size, change the ExecutionStrategy, be prepared
to a major loss of performance.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.