| Re: [jetty-users] Leaked EndPoint objects in AbstractConnector._endpoints due to ManagedSelector.destroyEndPoint task rejected by QueuedTheadPool |
------------------------------
Message: 2
Date: Wed, 2 Aug 2017 21:56:43 +1000
From: Greg Wilkins <gregw@xxxxxxxxxxx>
To: JETTY user mailing list <jetty-users@xxxxxxxxxxx>
Subject: Re: [jetty-users] Leaked EndPoint objects in
AbstractConnector._endpoints due to ManagedSelector.destroyEndPoint
task rejected by QueuedTheadPool
Message-ID:
<CAAPGdfFp-q=_9xHd4tXt34UWnyydC7V6Msi4emz8tXN9B3DPzg@mail. >gmail.com
Content-Type: text/plain; charset="utf-8"
Julien,
Having the QueuedThreadPool configured so that it may reject jobs is not
recommended and is likely to result in problems. More over, having such a
small threadpool (4 threads!) and a small queue (also 4) is going to mean
that only moderately low load is required for jobs to be discarded.
Having said that, we should not leak memory. However in this case, if an
task that would have destroyed an endpoint is lost, then the endpoint will
not be destroyed. We can't destroy it on rejection as that kind of defeats
the purpose of doing it as a task it in the first place. The EndPoint
will eventually timeout, but that may just queue another destroy event
which could be rejected, or maybe even the timeout task itself is being
rejected.
In an asynchronous server, if you randomly reject tasks, then some async
lifecycles will never complete and a leak will result. I highly recommend
having an infinite job queue and using other means to restrict the size
(limite connections or requests rather than tasks). At the very least,
have a limited job queue that blocks when full rather than rejects.
However, please raise an issue on github for this, as we should at least
look at a reproduction and see if there is anything more we can do with
that rejected execution to be more robust.
regards
On 2 August 2017 at 19:48, Julien Moumne <jmoumne@xxxxxxxxxx> wrote:
> Hello,
>
> We are experiencing a memory leak in Jetty 9.4.6.
>
> I am seeking for help in determining if we are not using Jetty properly or
> if there is indeed a limitation in Jetty.
>
> This behavior is observed under high load in a simulated environment using
> gatling and bench-rest, where QueuedTheadPool is saturated and rejects
> jobs
> <https://github.com/eclipse/jetty.project/blob/jetty-9.4. >6.v20170531/jetty-util/src/ main/java/org/eclipse/jetty/ util/thread/QueuedThreadPool. java#L384
> .
>
> We are getting the following WARN log from QueuedThreadPool :
>
> org.eclipse.jetty.util.thread.QueuedThreadPool:
> dw{STARTED,4<=4<=4,i=0,q=4} rejected org.eclipse.jetty.io.
> *ManagedSelector$$Lambda*$72/1495355211@413f1bdf
>
> The Lambda is defined in ManagedSelector.destroyEndPoint
> <https://github.com/eclipse/jetty.project/blob/jetty-9.4. >6.v20170531/jetty-io/src/main/ java/org/eclipse/jetty/io/ ManagedSelector.java#L431
>
>
> Since the Lambda is not called, AbstractConnector.onEndPointClosed
> <https://github.com/eclipse/jetty.project/blob/jetty-9.4. >6.v20170531/jetty-server/src/ main/java/org/eclipse/jetty/ server/AbstractConnector.java# L676
> is never called.
>
> Therefore, EndPoint objects remain in AbstractConnector._endpoints
> <https://github.com/eclipse/jetty.project/blob/jetty-9.4. >6.v20170531/jetty-server/src/ main/java/org/eclipse/jetty/ server/AbstractConnector.java# L147
> and lead to out of memory errors.
>
> A screenshot of a memory dump showing 624159 endpoint objects :
> https://imagebin.ca/v/3VTm2n97Ntki
> A screenshot of how we validated the behavior using a "logging breakpoint"
> in Intellij : https://imagebin.ca/v/3VTnTSgJDnXx
>
> Please tell me if you need any additional information
>
> Julien
>
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe
> from this list, visit
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
--
Greg Wilkins <gregw@xxxxxxxxxxx> CTO http://webtide.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/jetty-users/ >attachments/20170802/3dbea6b5/ attachment.html
------------------------------
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users
End of jetty-users Digest, Vol 99, Issue 2
******************************************