Hello,
I am seeing a periodic CPU spike when the Jetty server is handling requests. The spike occurs regularly every 5 minutes and uses up 90-100% CPU for a short duration.
The CPU utilization falls back to normal after that short spike. This happens only when the server is receiving requests.
I have taken thread dumps during several spikes and I have not been able to conclude anything concrete. In the dumps I observed the following:
1. There were hundreds of threads in BLOCKED state waiting for a lock held by a thread.
2. The thread that was holding the lock was in turn BLOCKED. But it was not waiting for any other lock. Example shown below.
"qtp732533575-307956" prio=10 tid=0x00002aad980c4800 nid=0x3e1f waiting for monitor entry [0x0000000067f69000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.hibernate.util.SoftLimitMRUCache.get(SoftLimitMRUCache.java:74)
- locked <0x00002aabefafa198> (a org.hibernate.util.SoftLimitMRUCache)
at org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:88)
at org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:156)
at org.hibernate.impl.AbstractSessionImpl.getNamedQuery(AbstractSessionImpl.java:82)
at org.hibernate.impl.SessionImpl.getNamedQuery(SessionImpl.java:1287)
at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source)
3. The actual lock that the threads were waiting on, varied. It was not the same in the different spikes every 5 minutes.
Environment Details:
* Embedded Jetty Version 8.0.4
* Java 1.6.0_17
* Red Hat Enterprise Linux Server release 5.2 (Tikanga)
JVM Parameters:
-server -Xmx11g -Xms11g -XX:MaxPermSize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:NewSize=5g -XX:MaxNewSize=5g -XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
I have also attached the CPU usage pattern. Any pointers would be greatly appreciated.
Thanks!
Vinay
