Thanks Simone.
Unfortunately, I still don't have a test case. As an experiment we swapped jetty component with HTTP4 and this seems to avoid the problem. In that light we've narrowed the issue to either a problem in Jetty or in the new OS patch that was recently applied (or some combination of the two).
The system was running fine until the following OS patch was installed:
As we can only reproduce this in production, removing the patch was not a popular option as affected too many other systems. The threads I mentioned below seem to be a moment in time for that particular lock up. Subsequent threads didn't show the same traces - just that things were blocked. I'm afraid it was a red herring.
This particular route talks to an IIS server via F5 Load Balancer. The Load Balancer has been reconfigured to use a pass through type connection so it should be direct communications channel now. We've also taken the load balancer out of the equation as well. We still saw the same issue. When the problem arises, the number of connections grow - all stuck in an ESTABLISHED state. TCP traces show that data is sent back but they are ignored and the request times out. Restarting or suspending the process clears the issue.
Would you be able to shed any light on what may be in the above patch that would affect the jetty in any way? Something isn't playing nicely there and we haven't been able to isolate it.
Any information greatly appreciated,
Susan
Hi,
On Wed, Nov 27, 2013 at 12:01 AM, Susan Javurek <sjavurek@xxxxxxxxxx> wrote:
Hi Simone,
Yes, update 65 is apparently available. However, it seems my customer has it installed but is not using it. There's no change to java in this scenario. The versions are:
* java 1.6.0_33 * jetty-client-7.6.7.v20120910.jar (so sorry for not including that!).
The customer is in production so I'm afraid an upgrade to latest version of Jetty isn't possible. The only change we are aware of is an update to the Kernel:
https://getupdates.oracle.com/readme/150401-03
There has been no change in the load running through the route. It's un-reproducable in any other environment. Updating the bundle or touching the process in some way, i.e jstack which suspends the process briefly seems to have a positive effect.
Have you all seen this type of issue before? Does the stack trace lend any clues?
No, never seen this issue before, sorry.
-- Simone Bordet ---- http://cometd.org http://webtide.com http://intalio.com Developer advice, training, services and support from the Jetty & CometD experts. Intalio, the modern way to build business applications. _______________________________________________ jetty-dev mailing list jetty-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/jetty-dev
|