Hello,
We have a critical issue where our app running in Tomcat (8.x) attempts to make a request to our other app running in Jetty (9.2.7) with Java 8 occasionally gets one or two ConnectExceptions which results in that app being marked unavailable - because our previous understanding was that we'd only receive that in cases where the app/server was physically down (or some other permanent issue like firewall, etc).
We can't determine why this happens (in one of the two cases) or when it might happen (no pattern yet) and are at a loss on how to troubleshoot this further. Is there logging for the OS or Jetty that might help here? Or any other ideas why we might have this?
It's happened a few times at one customer site running Windows 2012 - but there it's always happened under extremely heavy load. So my guess is that it's a thread or OS or port limit that's being encountered. (But we still can't say for sure yet.)
The other case is on CentOS release 6.9 (Final) on a popular cloud server provider, and it's happened two separate times, and in neither was the server or applications under any kind of substantial load. There's nothing in the (minimal) system logs or the app logs to indicate an issue. Just a request from Tomcat that ends up getting a ConnectException and so we mark the server as unavailable.
We use something along these lines for our Embedded Jetty server...
WebAppContext context = new WebAppContext();
QueuedThreadPool threadPool = new QueuedThreadPool(maxThreads); // usually 256
Server server = new Server(threadPool);
HttpConfiguration http_config = new HttpConfiguration();
ServerConnector http = new ServerConnector(server, new HttpConnectionFactory(http_config));
http.setPort(port);
server.setConnectors(new Connector[]
{
http
});
server.setHandler(context);
server.start();
server.join();
The app works quite well most of the time but we're baffled under what circumstances Jetty would briefly refuse connections (when not under heavy load). Nothing on the server indicates any issues at the time.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1031937
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 8192
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 8192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Memory of the server or app is not low at the time this occurs. And I've tried to reproduce in-house by forcing long gc cycles (or overloading the app) but we just end up getting timeouts rather than ConnectExceptions. Any help appreciated.
