Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Leaked EndPoint objects in AbstractConnector._endpoints due to ManagedSelector.destroyEndPoint task rejected by QueuedTheadPool


Julien,

Having the QueuedThreadPool configured so that it may reject jobs is not recommended and is likely to result in problems.  More over, having such a small threadpool (4 threads!) and a small queue (also 4) is going to mean that only moderately low load is required for jobs to be discarded.

Having said that, we should not leak memory.  However in this case, if an task that would have destroyed an endpoint is lost, then the endpoint will not be destroyed.  We can't destroy it on rejection as that kind of defeats the purpose of doing it as a task it in the first place.    The EndPoint will eventually timeout, but that may just queue another destroy event which could be rejected, or maybe even the timeout task itself is being rejected.

In an asynchronous server, if you randomly reject tasks, then some async lifecycles will never complete and a leak will result.   I highly recommend having an infinite job queue and using other means to restrict the size (limite connections or requests rather than tasks).  At the very least, have a limited job queue that blocks when full rather than rejects.

However, please raise an issue on github for this, as we should at least look at a reproduction and see if there is anything more we can do with that rejected execution to be more robust.

regards







On 2 August 2017 at 19:48, Julien Moumne <jmoumne@xxxxxxxxxx> wrote:
Hello,

We are experiencing a memory leak in Jetty 9.4.6.

I am seeking for help in determining if we are not using Jetty properly or if there is indeed a limitation in Jetty.

This behavior is observed under high load in a simulated environment using gatling and bench-rest, where QueuedTheadPool is saturated and rejects jobs.

We are getting the following WARN log from QueuedThreadPool :

org.eclipse.jetty.util.thread.QueuedThreadPool: dw{STARTED,4<=4<=4,i=0,q=4} rejected org.eclipse.jetty.io.ManagedSelector$$Lambda$72/1495355211@413f1bdf

The Lambda is defined in ManagedSelector.destroyEndPoint 

Since the Lambda is not called, AbstractConnector.onEndPointClosed is never called.

Therefore, EndPoint objects remain in AbstractConnector._endpoints and lead to out of memory errors.

A screenshot of a memory dump showing 624159 endpoint objects : https://imagebin.ca/v/3VTm2n97Ntki
A screenshot of how we validated the behavior using a "logging breakpoint" in Intellij : https://imagebin.ca/v/3VTnTSgJDnXx

Please tell me if you need any additional information

Julien

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users



--

Back to the top