We're using Jetty 8.1.3.v20120416 w/ JDBCSessionManager as our application server with Tomcat 7's JDBC Connection pool 7.0.28. We've run this particular combination in production since at least Sept 2012. (Database is Amazon RDS/MySQL.) (java 6 64-bit ubuntu)
This morning one Jetty instance stopped accepting requests and the problem appears to be a deadlock btw a jetty connection handler thread and the conn pool's "PoolCleaner" thread. From jstack trace:
Deadlock Detection:
Found one Java-level deadlock:
=============================
"qtp1840392480-3740":
waiting to lock Monitor@0x00007f4350001fd0 (Object@0x00000006c01a0e88, a com/mysql/jdbc/JDBC4Connection),
which is held by "PoolCleaner[2009981184:1363034108768]"
"PoolCleaner[2009981184:1363034108768]":
waiting to lock Monitor@0x00007f4350001f28 (Object@0x00000006c1ed5738, a com/mysql/jdbc/JDBC4ResultSet),
which is held by "qtp1840392480-3740"
Once this happened all other worker threads blocked at various points within JDBCSession manager. Obviously restarting the instance resolved the issue for now.
How does one go about diagnosing / resolving such an issue? It's not clear to me if either component is "at fault" here...I have the full jstack output as well as a full heap dump of the java process, if that would help. I hate to switch connection pools at this point unless there is a well-known "best" pool to use with Jetty?
Regards,
Colin
