| Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1 |
Hi Emmeran,
This is a known bug in the jdk that has been around for quite a while.
The jetty doc on it is here (see subsection "Direct ByteBuffers"):
http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html#jvm-bugs
Sadly, there is still no fix in the jvm for it. If you've got a good
workaround, I'd go with that.
cheers
Jan
On 26 November 2013 21:10, Emmeran Seehuber <rototor@xxxxxxxxxx> wrote:
> Hi,
>
> I already found the cause for this problem: I do big writes.
>
> The user upload wedding photos (the software is a web2print wedding card shop), and I echo them back to the user. To do so I read them fully into a byte[] and then do one big
>
> request.getOutputStream().write(data);
>
> As the images vary in size from 3 - 20+ MB, so do the byte[]…
>
> Jetty internally just wraps the byte[] into a ByteBuffer and passes it to NIO. There it finally lands in
>
> IOUtil.write()
> http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html
>
> If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing is passed to the native write. But if it is not, it allocates a temporary DirectByteBuffer of the full size of the ByteBuffer using
>
> Util.getTemporaryDirectBuffer
> http://www.docjar.com/html/api/sun/nio/ch/Util.java.html
>
> The problem is, this DirectByteBuffer are then after the write cached in a thread local pool of 8 such DirectByteBuffer. The pool keeps the biggest 8 requested buffers.
>
> If you now have about 200+ live threads and any of them can do a 8, 10, … MB write, then you end up using much direct memory. Over time this pool just grows and grows ….
>
> After trying to avoid reading the image data into a byte[] as much as possible and if i have to read the image fully into a byte[] just doing chunked writes of the byte buffer, the problem is gone. I.e. i do now a
>
> ServletOutputStream outputStream = res.getOutputStream();
> int i = 0;
> final int MAX_BYTES_TO_SEND = 64 * 1024;
> while (i < data.length) {
> final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, data.length - i);
> outputStream.write(data, i, maxSendSize);
> i += maxSendSize;
> }
>
> and everything is fine.
>
> Of course, this was no problem with Jetty 7, because there I did not use the NIO connector there.
>
> I am not sure if you can consider this a bug of Jetty, or if this is rather a bug of the JDK. At least it is very unexpected.
>
> Maybe Jetty should workaround this buggy behavior and just always use its internal DirectByteBuffers and do chunked writes with DirectByteBuffers itself?
>
> I´m not sure if the JDK behavior gives any speed advantages if you cache DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple trashes the CPU cache instead ...
>
> Should I still open a bug for this?
>
> Thanks.
>
> cu,
> Emmy
>
> (BTW: The OS is Linux)
>
> Am 25.11.2013 um 02:37 schrieb Jan Bartel <janb@xxxxxxxxxxx>:
>
>> And when you open the bug, please make sure to state which operating
>> system and version you are using.
>>
>> thanks
>> Jan
>>
>> On 24 November 2013 09:55, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:
>>> Hi,
>>>
>>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <rototor@xxxxxxxxxx> wrote:
>>>> Hello everybody,
>>>>
>>>> I´m using jetty as standalone embedded web server behind a nginx and it is
>>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>>>> to 9.1.0.v20131115.
>>>>
>>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>>>> still have the same leaks. Native memory leaks mean, that the process memory
>>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>>>> But to get there the server process has to run for about two weeks.
>>>>
>>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>>>> be 7 GB.
>>>>
>>>> First i thought that the many threads started/stopped in the thread pool are
>>>> the problem. I specified a idle time of -1, and still many threads are
>>>> restarted. Also some answers on the web about DirectByteBuffer OOM
>>>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>>>
>>>> I think the problem may be in ArrayByteBufferPool. It allocates an unbound
>>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>>>> at some peak times many buffers are used and never freed. On the other side
>>>> this doesn’t really fit with the allocated memory. At the moment the server
>>>> has this direct memory allocations (according to the MBeans info):
>>>>
>>>> java.nio
>>>> BufferPool
>>>> name=direct
>>>> - Count 384
>>>> - MemoryUsed 2728821147
>>>> - Name direct
>>>> - ObjectName java.nio:type=BufferPool,name=direct
>>>> - TotalCapacity 2728821147
>>>>
>>>> That would mean an medium buffer size of about 7 MB. But the
>>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>>>> kb sized buffers by default ?!
>>>>
>>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>>>
>>>> This numbers are after the process is running for about 32 hours. I´ve got
>>>> about 320 request/minute at peak times. All this numbers are according to
>>>> the embedded JavaMelody monitoring.
>>>>
>>>> The JVM arguments are:
>>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>>>> JDK 1.7.0_45
>>>>
>>>> The thread pool is configured this way:
>>>>
>>>> tp = new QueuedThreadPool();
>>>> tp.setMaxThreads(256);
>>>> tp.setMinThreads(5);
>>>> tp.setIdleTimeout(-1);
>>>> tp.setName("Http Server Thread Pool");
>>>> tp.setDaemon(true);
>>>>
>>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>>>> hour does not help. Also the i´ve got no other GC problems.
>>>>
>>>> Any ideas what could cause this problems?
>>>> Why are threads restarted in the pool even when i specify a idle timeout of
>>>> -1? I also regulary schedule some own runnables on the thread pool - but
>>>> they should not cause this problems, should they? They do in 99.99% of all
>>>> cases not throw any exceptions.
>>>
>>> Seems like a bug.
>>> I would keep the ThreadPool at default configuration for now, to have
>>> less variables in the system.
>>>
>>> Would you be able to replace usage of ArrayByteBufferPool with
>>> MappedByteBufferPool and see if the problem persist ?
>>>
>>> Also, please file an issue about this, it really looks from JMX that
>>> there is a problem.
>>>
>>> Thanks !
>>>
>>> --
>>> Simone Bordet
>>> ----
>>> http://cometd.org
>>> http://webtide.com
>>> http://intalio.com
>>> Developer advice, training, services and support
>>> from the Jetty & CometD experts.
>>> Intalio, the modern way to build business applications.
>>> _______________________________________________
>>> jetty-users mailing list
>>> jetty-users@xxxxxxxxxxx
>>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>>
>>
>> --
>> Jan Bartel <janb@xxxxxxxxxxx>
>> www.webtide.com
>> 'Expert Jetty/CometD developer,production,operations advice'
>> _______________________________________________
>> jetty-users mailing list
>> jetty-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
>>
>
> Mit freundlichen Grüßen aus Augsburg,
>
> Emmeran Seehuber
> Dipl. Inf. (FH)
> Schrannenstraße 8
> 86150 Augsburg
> USt-IdNr.: DE266070804
>
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
--
Jan Bartel <janb@xxxxxxxxxxx>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users