| Re: [jetty-users] High CPU Load in Jetty 7.6.2 / JDK 1.7 |
Hi, I just wanted to provide an update on this for the benefit of the list. We went ahead and switched to the BlockingChannelConnector. Unfortunately, not only did that not solve our CPU usage issues, but it introduced some network communication issues. Notably we started seeing ton’s of these in the log files: Caused by: org.eclipse.jetty.io.EofException I tried tuning the tcp driver but that didn’t help at all. Also of note: we’re using IPv6. I did try to force the JVM to use IPv4 but that didn’t have an impact either. There was also a suggestion from the list to try 7.6.3 and we did that too. We didn’t see any improvement in CPU and we still had the EoF errors. For now I’ve gone back to the SelectChannelConnector with 7.6.3. The EoF errors have subsided but our CPU consumption is too high for our liking. We’ve instrumented the code and are looking at some other options internally to understand what the issues might be.
Cheers, // devon From: jetty-users-bounces@xxxxxxxxxxx [mailto:jetty-users-bounces@xxxxxxxxxxx] On Behalf Of Neil Bliss We had seen similar behavior when we used the SelectChannelConnector. In our situation, we found that using the BlockingChannelConnector was a better fit. That might be worth a try in your situation. On Wed, Apr 18, 2012 at 4:41 PM, Devon Lazarus <Devon.Lazarus@xxxxxxxxx> wrote: Hi, I’ve been trying to get to the bottom of what I believe is a performance problem in a Jetty application. Unfortunately, I’ve hit a road block and could use some help. Googling hasn’t shed any light on the issue. Neither has stackoverflow. Basically, I’m seeing very high CPU load without a lot of throughput. To quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, and some GC tuning. We’re using a custom load generation tool to find the point at which the server falls over. However, we’re hitting 90% CPU burn with loads well over 2 while handling ~25 concurrent transactions/second (throughput was 63 tps with ~400ms response times). That just doesn’t seem right (way too low). In profiling the JVM regardless of load (low or under siege), we can see clearly that 99% of our CPU time is spent in one of two places: sun.nio.ch.SelectorImpl.select() at 66% sun.nio.ch.ServerSocketChannelImpl.accept() at 33% with all of that time being consumed by the acceptor/selector threads dumped below in equal parts. In looking at the thread dump, we have many threads that look like this: "qtp665633643-88" - Thread t@88 java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for <2ef1b67e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None And I see the acceptor/selectors like this: "qtp665633643-83 Selector0" - Thread t@83 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <7b18010e> (a sun.nio.ch.Util$2) - locked <23bb2f20> (a java.util.Collections$UnmodifiableSet) - locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None "qtp1586177080-28 Acceptor0 SelectChannelConnector@0.0.0.0:80" - Thread t@28 java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) - locked <58341b55> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None "qtp1586177080-27 Selector0" - Thread t@27 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <be7202b> (a sun.nio.ch.Util$2) - locked <2761d665> (a java.util.Collections$UnmodifiableSet) - locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None Not a single thread in BLOCKED state. I know I only have a single acceptor configured when I have two processors, but when I do configure two acceptors, I see the two acceptor threads blocking each other (one RUNNABLE while the other BLOCKED and switch back and forth after 10s of seconds). Any help is appreciated, and I apologize for the long email. Regards, Devon Lazarus, Internet Services Group Sonos, Inc.
-- |