| [jetty-users] High CPU Load in Jetty 7.6.2 / JDK 1.7 |
Hi, I’ve been trying to get to the bottom of what I believe is a performance problem in a Jetty application. Unfortunately, I’ve hit a road block and could use some help. Googling hasn’t shed any light on the issue. Neither has stackoverflow. Basically, I’m seeing very high CPU load without a lot of throughput. To quantify that, we have a c1.medium (dual core) running Ubuntu 10.04LTS in EC2 with Jetty 7.6.2 configured at 50 min/200 max threads, a single acceptor, and the Oracle JDK 1.7_03 JVM configured with -mx2048m –ms512m, and some GC tuning. We’re using a custom load generation tool to find the point at which the server falls over. However, we’re hitting 90% CPU burn with loads well over 2 while handling ~25 concurrent transactions/second (throughput was 63 tps with ~400ms response times). That just doesn’t seem right (way too low). In profiling the JVM regardless of load (low or under siege), we can see clearly that 99% of our CPU time is spent in one of two places: sun.nio.ch.SelectorImpl.select() at 66% sun.nio.ch.ServerSocketChannelImpl.accept() at 33% with all of that time being consumed by the acceptor/selector threads dumped below in equal parts. In looking at the thread dump, we have many threads that look like this: "qtp665633643-88" - Thread t@88 java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) - parking to wait for <2ef1b67e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None And I see the acceptor/selectors like this: "qtp665633643-83 Selector0" - Thread t@83 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <7b18010e> (a sun.nio.ch.Util$2) - locked <23bb2f20> (a java.util.Collections$UnmodifiableSet) - locked <7557d06a> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None "qtp1586177080-28 Acceptor0 SelectChannelConnector@0.0.0.0:80" - Thread t@28 java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) - locked <58341b55> (a java.lang.Object) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None "qtp1586177080-27 Selector0" - Thread t@27 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <be7202b> (a sun.nio.ch.Util$2) - locked <2761d665> (a java.util.Collections$UnmodifiableSet) - locked <4b49db96> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:722) Locked ownable synchronizers: - None Not a single thread in BLOCKED state. I know I only have a single acceptor configured when I have two processors, but when I do configure two acceptors, I see the two acceptor threads blocking each other (one RUNNABLE while the other BLOCKED and switch back and forth after 10s of seconds). Any help is appreciated, and I apologize for the long email. Regards, Devon Lazarus, Internet Services Group Sonos, Inc. |