Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[jetty-users] Fwd: high cpu load NIO (Jetty7.6.0)
sorry, i meant  to reply this to jetty-users@xxxxxxxxxxx, didnt realize that this points only to thomas

thanx

---------- Forwarded message ----------
From: kim young ill <khiemu@xxxxxxxxxxxxxx>
Date: Tue, Apr 17, 2012 at 8:09 PM
Subject: Re: [jetty-users] high cpu load NIO (Jetty7.6.0)
To: Thomas Becker <tbecker@xxxxxxxxxxx>


hi there, no one seems to have the same issue ?

i switched back to blocking IO & dont have this problem anymore.

thanx


On Mon, Apr 2, 2012 at 1:15 PM, kim young ill <khiemu@xxxxxxxxxxxxxx> wrote:
hi there,
update to 7.6.2 doesnt seem to solve our problem, cpu gets up stays there after some days again,
with the tool from here

http://weblogs.java.net/blog/brucechapman/archive/2008/03/hot_threads.html

this is what i got (same problem with Selector thread):

47.0% CPU Usage by Thread 'qtp650519471-16 Selector0'
  3/10 snapshots sharing following 10 elements

    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
    org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    java.lang.Thread.run(Thread.java:662)
  3/10 snapshots sharing following 11 elements
    sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
    sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:246)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:209)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:88)
    org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)

    org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    java.lang.Thread.run(Thread.java:662)
  4/10 snapshots sharing following 4 elements

    org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
    java.lang.Thread.run(Thread.java:662)
 
12.0% CPU Usage by Thread 'qtp650519471-27914'
  10/10 snapshots sharing following 8 elements
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337)
    org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517)
    org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39)
    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563)
    java.lang.Thread.run(Thread.java:662)
 
12.0% CPU Usage by Thread 'qtp650519471-27909'
  9/10 snapshots sharing following 8 elements
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:337)
    org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:517)
    org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:39)
    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:563)
    java.lang.Thread.run(Thread.java:662)


///
i suspected that we got hit by this bug:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6525190

but testing on our jvm (1.6 update 31) shows that it's not the issue.

the difference on 7.6.2 is that we dont see this kind of message anymore even when cpu get high:

Selector  org.eclipse.jetty.io.nio.SelectorManager$SelectSet@7413a356 keys=8
selected=0 is too busy, pausing!

thanx






On Sat, Mar 10, 2012 at 12:06 PM, kim young ill <khiemu@xxxxxxxxxxxxxx> wrote:
the servers are still running ( slow thou) & we cannot restart it yet, but i'll test it on a test server & let you know in any case.

by the way, on one of the machine which spins which has fine-log, this message fill the logs endless, dont know if it helps to narrow the issue:


2012-03-10 10:51:19,740 AsyncHttpConnection-qtp986381142-19351 [INFO ] EndPoint making no progress: -1807095600 SSL NEED_WRAP i/o/u=0/0/714 ishut=false oshut=true {AsyncHttpConnection@7816f7f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=-3},r=0} AsyncHttpConnection@7816f7f,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=0,l=0,c=-3},r=0



thanx




On Mon, Mar 5, 2012 at 5:25 PM, Thomas Becker <tbecker@xxxxxxxxxxx> wrote:
Hi,

could you please upgrade to the most recent jetty version and report back if that has fixed the problem? Some bugs causing cpu spins like this have been fixed since 7.6.0.
http://download.eclipse.org/jetty/

Cheers,
Thomas




On 3/5/12 5:16 PM, kim young ill wrote:
Hi there,

after running for 2 days, we have suddenly high cpu (permanent 100%), even there's no traffic,  cpu usage wouldnt go down even there's no traffic.

env:
jetty 7.6.0 (embedded)
linux 64bit (opensuse, kernel 2.6.37.1-1.2
oracle-jdk 1.6.0.31


from jconsole i got these 2 threads, each use 50-60% cpu:


sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
java.lang.Thread.run(Thread.java:662)
 


sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:256)
sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:175)
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.wakeup(SelectorManager.java:817)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:520)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.close(SelectChannelEndPoint.java:698)
org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:377)
org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43)
org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.flush(SslConnection.java:718)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:85)
org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
java.lang.Thread.run(Thread.java:662)


any hints ?

thanx



_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users


-- 
thomas becker
tbecker@xxxxxxxxxxx

http://webtide.com / http://intalio.com
(the folks behind jetty and cometd)




Back to the top