[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jetty-users] How to get Jetty to degrade gracefully under load
|
On 6/8/2012 11:47 AM, Simone Bordet wrote:
Hi,
On Fri, Jun 8, 2012 at 6:21 PM, ccleve<ccleve.tech@xxxxxxxxx> wrote:
We're seeing an odd problem when load testing our site. Something's probably
misconfigured, but I don't know where to look.
As we ramp up the number of users with our load testing software, the number
of requests Jetty can handle tops out at about 100/second. (Our app is doing
some work to process each request, so the low rate isn't Jetty's fault). As
more and more requests come in, average page load time creeps up to 6 or 7
seconds, max load time becomes very high, and then all at once the system
starts sending 503s. Any new requests get a 503, and the load testing
software reports that it starts getting about 500 503s per second,
presumably from earlier requests that were queued up. This continues for
about a minute, and then the system starts accepting requests again.
So when a lot of users come in, some get serviced, some have to wait a long
time, and then whole system shuts down for a minute while it recovers.
I'd prefer not to have a catastrophic failure like this. Perhaps the system
should send 503s earlier so the queue never gets too big. Or maybe it should
never send a 503, and just make the wait times uniformly longer for all
users.
We're on Amazon, so we can use monitoring software to detect when page load
time is> X seconds and spin up more instances.
What parameters can we twiddle to control how Jetty degrades? Or have we
misconfigured things?
Are you sure that's Jetty sending the 503 and not the application or
some other network component (e.g. Apache) in front of Jetty ?
Are you using the QoSFilter ?
I ask because Jetty does not send, to my knowledge, 503s when the
system is overloaded.
Simon
Thank you for the clue. Turns out that there was a load balancer between
the loadtester and our site that I didn't know about. That was throwing
the 503s. I will check out QoSFilter.
