Re: [jetty-users] Early EOF and Native Memory Leak

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [jetty-users] Early EOF and Native Memory Leak

From: Jesse McConnell <jesse.mcconnell@xxxxxxxxx>
Date: Wed, 4 Oct 2017 07:34:21 -0500
Delivered-to: jetty-users@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/jetty-users>
List-help: <mailto:jetty-users-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=unsubscribe>

That is an exceedingly old version of Jetty, we are currently up to 9.4.7.v20170914. Unless you are stuck on Java 7 for some reason I would recommend updating and seeing if you can reproduce on a more current version of Jetty, and even if you are stuck on Java 7, you should update to 9.2.22.v20170606. We have continued to push releases of Jetty 9.2 strictly for users that are restricted to Java 7 but will soon be announcing the end to that since Java 9 is out the door now.

cheers,

Jesse

--
jesse mcconnell
jesse.mcconnell@xxxxxxxxx

On Tue, Oct 3, 2017 at 2:38 PM, Jing Yang <jingisgood@xxxxxxxxx> wrote:

Hello jetty-users,

My application is an image/video storage service, which stores/serves up to 100 mb files (traffic is low, under 5 QPS). The uploaded files are stored on Google Cloud Storage. My server runs on jetty-9.2.13.v20150730.

Native Memory Leak:
I noticed a slow native memory leak in my application. It starts with 3G memory, then gradually grows to 6G after 10 days, and the gradually grows to 9G after another 10 days. I took a heap dump but the could not find anything significant, heap size is around 1.5G.

Early EOF:
Besides the native memory leak, another noticeable problem is that I am getting "Early EOF" exceptions from time to time (*full* stacktrace in the end). I was monitoring the TCP traffic, and noticed that this exception is thrown when the client sends Fin to server, after which server tries to push more data to client and client sends back a Reset. (tcpdump snippet in the end)

Question:
So, I am seeking guidance about:
1. Could the Reset package be the cause of the Early EOF? How should I handle it properly?
2. Any suggestions on the direction of investigating the native memory leak?
3. Could these two issues be related to each other?

TCPDump snippet:
client > server: Flags [P.], ... <= client pushes some final data
server > client: Flags [.], ...    <= server acks on the data
client > server: Flags [F.], ...   <= client sends Fin Ack
server > client: Flags [.], ...     <= server Ack the Fin
server > client: Flags [P.], ...    <= server tries to push more data
client > server: Flags [R], ...     <= client sends Reset

Stacktrace:
org.eclipse.jetty.io.EofException: Early EOF
        at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:506)
        at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:124)
        at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:999)
        at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:943)
        at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:922)
        at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:426)
        at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:427)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at MyController.uploadStream(MyController.java:1111)
        at MyController.handlePost(MyController.java:596)
        at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users

References:
- [jetty-users] Early EOF and Native Memory Leak
  - From: Jing Yang

Prev by Date: Re: [jetty-users] proxy AsyncMiddleManServlet: ProxyWriter onWritePossible never called
Next by Date: [jetty-users] (no subject)
Previous by thread: [jetty-users] Early EOF and Native Memory Leak
Next by thread: [jetty-users] Dramatic throughput slow down of custom FileServlet Jetty 9.4.5
Index(es):
- Date
- Thread

Breadcrumbs