Re: [jetty-users] Help with connect timeout

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Fri, Jun 18, 2021 at 3:00 AM Lu Tahmazyan via jetty-users
<jetty-users@xxxxxxxxxxx> wrote:
>
> Hello, good people of Jetty.
>
> I am looking for some guidance with this exception
>
> java.net.SocketTimeoutException: Connect Timeout
> at org.eclipse.jetty.io.ManagedSelector$Connect.run(ManagedSelector.java:955)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> We have a subclass of AsyncMiddleManServlet which uses Jetty's HttpClient to proxy requests. HttpClient is configured with a connect-timeout of 1000 ms.  What is interesting is that we see socket timeout exception is thrown without utilizing the full 1000 ms allocated for connect timeout, for example, we see calls getting timed out under 25 ms.

That is weird.
Do you have a reproducible case?
What OS and JVM version/vendor?

> Exception is thrwon from here (ManagedSelector:955)
>         public void run()
>         {
>             if (_selectorManager.isConnectionPending(channel))
>             {
>                 if (LOG.isDebugEnabled())
>                     LOG.debug("Channel {} timed out while connecting, closing it", channel);
>                 failed(new SocketTimeoutException("Connect Timeout"));
>             }
>         }
>
> One theory was that the proxied service was not responding fast enough, due to it being under-provisioned.  In order to test that theory, we scaled the backend service 3x, and still, we are seeing these types of exceptions.
>
> Looking at the Jetty code, "SocketTimeoutException: Connect Timeout" is thrown if the connection on the channel is pending.  I don't really understand how the connect timeout is related here.  If it is how would one configure it.  Basically what I am asking for there is there a configuration parameter for addressing the pending connection issue which leads to SocketTimeoutException.

The way it works is that we open the SocketChannel and register it for
OP_CONNECT.
When the connection is complete, the JVM NIO subsystem calls the Jetty
code above where we test whether the connection was actually
completed.
Obviously it is still pending even if the JVM called Jetty with the
OP_CONNECT event, so we throw.

My theory is that you have some bad/under-provisioned configuration on
the server/load balancer where opening many connections leads to them
being refused almost immediately.
What you see is not a "timeout" but the server rejecting the
connection attempt, which generates on the client an OP_CONNECT event
to which we react with the code above.
So perhaps we can improve the exception type and/or message, but I
think you have some other configuration problem on the server or load
balancer.
Can you please open an issue about improving the exception message?

If you can take a network dump, it should be clear what happens.
I would expect the client sending a SYN and the server replying
immediately with a FIN.

Alternatively, configure the client with blocking connects:

HttpClient httpClient = ...;
httpClient.setConnectBlocking(true);

In this way TCP connects are blocking and you should see (if my theory
is correct) "close"-type exceptions being thrown as the server refuses
connect attempts.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
Developer advice, training, services and support
from the Jetty & CometD experts.