Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] DirectByteBuffer native memory problems (OOM) with Jetty 9.0/9.1 
Hi,

I already found the cause for this problem: I do big writes. 

The user upload wedding photos (the software is a web2print wedding card shop), and I echo them back to the user. To do so I read them fully into a byte[] and then do one big

request.getOutputStream().write(data);

As the images vary in size from 3 - 20+ MB, so do the byte[]…

Jetty internally just wraps the byte[] into a ByteBuffer and passes it to NIO. There it finally lands in 

IOUtil.write()
http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html

If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing is passed to the native write. But if it is not, it allocates a temporary DirectByteBuffer of the full size of the ByteBuffer using

Util.getTemporaryDirectBuffer
http://www.docjar.com/html/api/sun/nio/ch/Util.java.html

The problem is, this DirectByteBuffer are then after the write cached in a thread local pool of 8 such DirectByteBuffer.  The pool keeps the biggest 8 requested buffers.

If you now have about 200+ live threads and any of them can do a 8, 10, … MB write, then you end up using much direct memory. Over time this pool just grows and grows ….

After trying to avoid reading the image data into a byte[] as much as possible and if i have to read the image fully into a byte[] just doing chunked writes of the byte buffer, the problem is gone. I.e. i do now a 

		ServletOutputStream outputStream = res.getOutputStream();
		int i = 0;
		final int MAX_BYTES_TO_SEND = 64 * 1024;
		while (i < data.length) {
			final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, data.length - i);
			outputStream.write(data, i, maxSendSize);
			i += maxSendSize;
		} 

and everything is fine.

Of course, this was no problem with Jetty 7, because there I did not use the NIO connector there.

I am not sure if you can consider this a bug of Jetty, or if this is rather a bug of the JDK. At least it is very unexpected. 

Maybe Jetty should workaround this buggy behavior and just always use its internal DirectByteBuffers and do chunked writes with DirectByteBuffers itself?

I´m not sure if the JDK behavior gives any speed advantages if you cache DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple  trashes the CPU cache instead ...

Should I still open a bug for this?

Thanks.

cu,
  Emmy

(BTW: The OS is Linux)

Am 25.11.2013 um 02:37 schrieb Jan Bartel <janb@xxxxxxxxxxx>:

> And when you open the bug, please make sure to state which operating
> system and version you are using.
> 
> thanks
> Jan
> 
> On 24 November 2013 09:55, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:
>> Hi,
>> 
>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <rototor@xxxxxxxxxx> wrote:
>>> Hello everybody,
>>> 
>>> I´m using jetty as standalone embedded web server behind a nginx and it is
>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days ago
>>> to 9.1.0.v20131115.
>>> 
>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i
>>> still have the same leaks. Native memory leaks mean, that the process memory
>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers.
>>> But to get there the server process has to run for about two weeks.
>>> 
>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to
>>> be 7 GB.
>>> 
>>> First i thought that the many threads started/stopped in the thread pool are
>>> the problem. I specified a idle time of -1, and still many threads are
>>> restarted. Also some answers on the web about DirectByteBuffer OOM
>>> Exceptions pointed into this direction. But i don’t think thats the problem.
>>> 
>>> I think the problem may be in ArrayByteBufferPool.  It allocates an unbound
>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. So
>>> at some peak times many buffers are used and never freed.  On the other side
>>> this doesn’t really fit with the allocated memory. At the moment the server
>>> has this direct memory allocations (according to the MBeans info):
>>> 
>>> java.nio
>>> BufferPool
>>> name=direct
>>> -  Count 384
>>> -  MemoryUsed 2728821147
>>> -  Name direct
>>> -  ObjectName java.nio:type=BufferPool,name=direct
>>> -  TotalCapacity 2728821147
>>> 
>>> That would mean an medium buffer size of about 7 MB. But the
>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64
>>> kb sized buffers by default ?!
>>> 
>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599
>>> 
>>> This numbers are after the process is running for about 32 hours. I´ve got
>>> about 320 request/minute at peak times. All this numbers are according to
>>> the embedded JavaMelody monitoring.
>>> 
>>> The JVM arguments are:
>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M
>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat
>>> JDK 1.7.0_45
>>> 
>>> The thread pool is configured this way:
>>> 
>>> tp = new QueuedThreadPool();
>>> tp.setMaxThreads(256);
>>> tp.setMinThreads(5);
>>> tp.setIdleTimeout(-1);
>>> tp.setName("Http Server Thread Pool");
>>> tp.setDaemon(true);
>>> 
>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every
>>> hour does not help. Also the i´ve got no other GC problems.
>>> 
>>> Any ideas what could cause this problems?
>>> Why are threads restarted in the pool even when i specify a idle timeout of
>>> -1? I also regulary schedule some own runnables on the thread pool - but
>>> they should not cause this problems, should they? They do in 99.99% of all
>>> cases not throw any exceptions.
>> 
>> Seems like a bug.
>> I would keep the ThreadPool at default configuration for now, to have
>> less variables in the system.
>> 
>> Would you be able to replace usage of ArrayByteBufferPool with
>> MappedByteBufferPool and see if the problem persist ?
>> 
>> Also, please file an issue about this, it really looks from JMX that
>> there is a problem.
>> 
>> Thanks !
>> 
>> --
>> Simone Bordet
>> ----
>> http://cometd.org
>> http://webtide.com
>> http://intalio.com
>> Developer advice, training, services and support
>> from the Jetty & CometD experts.
>> Intalio, the modern way to build business applications.
>> _______________________________________________
>> jetty-users mailing list
>> jetty-users@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/jetty-users
> 
> 
> 
> -- 
> Jan Bartel <janb@xxxxxxxxxxx>
> www.webtide.com
> 'Expert Jetty/CometD developer,production,operations advice'
> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
> 

Mit freundlichen Grüßen aus Augsburg,

Emmeran Seehuber
Dipl. Inf. (FH)
Schrannenstraße 8
86150 Augsburg
USt-IdNr.: DE266070804

Back to the top