Re: [jetty-dev] long response time lead to high CPU in jetty when using

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting

From: Jan Bartel <janb@xxxxxxxxxxx>
Date: Fri, 16 Jan 2015 11:53:24 +0100
Delivered-to: jetty-dev@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/jetty-dev>
List-help: <mailto:jetty-dev-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/jetty-dev>, <mailto:jetty-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/jetty-dev>, <mailto:jetty-dev-request@eclipse.org?subject=unsubscribe>

Hi Vicky,

Actually, Jetty-8 is end-of-life, so it will be difficult for any of
the jetty committers to spend much time on this issue unless you could
provide a small reproduction test harness. Even then, any work on it
would have to be a lower priority than jetty-9 work.  If you can
produce a small test case, then please also include details of your
execution platform (os and version) and jvm version and we'll do our
best to try and get some time to take a look at it, but no promises.

If its possible for you, then an upgrade to jetty-9 would be
recommended as there have been a lot of modifications to the io layer,
including handling of ssl. If you absolutely cannot upgrade, or you
need help with upgrading, then please contact Webtide
(info@xxxxxxxxxxx) who can provide you some commercial support to meet
your requirements.

cheers
Jan

On 16 January 2015 at 03:34, Vicky Wu <xiwu@xxxxxxxxxx> wrote:
> Joakim,
>
> Thanks a lot for the response!  I am testing on the latest jetty
> 8.1.16.V20140903.
> I invoked the service for 3 times.
> curl -k --user admin:admin https://localhost:443/demo/aaa
> And collected the thread dumps 6 times.
> I can see that these three threads 19057,19052, 19055 are consuming high
> cpu.
> And the method HttpGenerator.flushBuffer() went to an infinite loop.
>
> eg:
> "qtp1224044906-21" prio=10 tid=0x00007f22a88d5000 nid=0x4a71 runnable
> [0x00007f22c9d68000]
>    java.lang.Thread.State: RUNNABLE
>     at
> org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:811)
>     at
> org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
>     at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
>     at
> org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1123)
>     at
> org.eclipse.jetty.server.AbstractHttpConnection$Output.sendContent(AbstractHttpConnection.java:1237)
>     at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination$JettyOutputStream.copyFrom(JettyHTTPDestination.java:316)
>     at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
>     at
> org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.copyFrom(AbstractHTTPDestination.java:752)
>     at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:105)
>     at org.apache.cxf.helpers.IOUtils.copyAndCloseInput(IOUtils.java:113)
>     at
> org.apache.cxf.jaxrs.provider.BinaryDataProvider.copyInputToOutput(BinaryDataProvider.java:186)
>     at
> org.apache.cxf.jaxrs.provider.BinaryDataProvider.writeTo(BinaryDataProvider.java:151)
>     at
> org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1360)
>     at
> org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.serializeMessage(JAXRSOutInterceptor.java:244)
>     at
> org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.processResponse(JAXRSOutInterceptor.java:117)
>     at
> org.apache.cxf.jaxrs.interceptor.JAXRSOutInterceptor.handleMessage(JAXRSOutInterceptor.java:80)
>     at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
>     - locked <0x00000000c6dc6c20> (a
> org.apache.cxf.phase.PhaseInterceptorChain)
>     at
> org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83)
>     at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
>     - locked <0x00000000c6667ae8> (a
> org.apache.cxf.phase.PhaseInterceptorChain)
>     at
> org.apache.cxf.phase.PhaseInterceptorChain.resume(PhaseInterceptorChain.java:277)
>     - locked <0x00000000c6667ae8> (a
> org.apache.cxf.phase.PhaseInterceptorChain)
>     at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:78)
>     - locked <0x00000000c6667ae8> (a
> org.apache.cxf.phase.PhaseInterceptorChain)
>     at
> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:243)
>     at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:261)
>     at
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
>     at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
>     at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1024)
>     at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>     at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:193)
>     at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>     at org.eclipse.jetty.server.Server.handleAsync(Server.java:410)
>     at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:519)
>     at
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
>     at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
>     at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
>     at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
>     at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>     at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
>     at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
>     at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
>     at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>     at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>     at java.lang.Thread.run(Thread.java:744)
>
>    Locked ownable synchronizers:
>     - None
>
>
>
>
> On 01/15/2015 09:31 PM, Joakim Erdfelt wrote:
>
> What version of Jetty?
> And have you taken a few timed thread dumps during the high CPU situation to
> capture what specifically is looping?
>
> --
> Joakim Erdfelt <joakim@xxxxxxxxxxx>
> webtide.com - intalio.com/jetty
> Expert advice, services and support from from the Jetty & CometD experts
> eclipse.org/jetty - cometd.org
>
> On Thu, Jan 15, 2015 at 3:05 AM, Vicky Wu <xiwu@xxxxxxxxxx> wrote:
>>
>> All,
>>
>> I have a long response time CXF restful web service and I am using camel
>> to expose this CXF endpoint. The response time is 2 mins.
>> At the same time I have the httpd as the front proxy while it has the
>> setting timeout=60
>> <VirtualHost *:443>
>> ...
>>       <Proxy balancer://cxfrs>
>>            BalancerMemberhttps://localhost:9292/demo/aaa  retry=10
>> timeout=60
>>       </Proxy>
>>       ProxyPass /demo/aaa balancer://cxfrs
>> ...
>> </VirtualHost>
>>
>> When I visit the service through the below command several times(3~5
>> times):
>>
>> curl -k --user admin:adminhttps://localhost:443/demo/aaa
>>
>> after several mins, I can see the high cpu in the jetty through "top"
>> command and the jetty code is running in an infinite loop.
>>
>> Would you please help on this issue?
>>
>>
>> --
>> Best Regards,
>> Xiaohui(Vicky) Wu
>>
>> _______________________________________________
>> jetty-dev mailing list
>> jetty-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/jetty-dev
>
>
>
>
> _______________________________________________
> jetty-dev mailing list
> jetty-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/jetty-dev
>
>
> --
> Best Regards,
> Xiaohui(Vicky) Wu
>
>
> _______________________________________________
> jetty-dev mailing list
> jetty-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/jetty-dev



-- 
Jan Bartel <janb@xxxxxxxxxxx>
www.webtide.com
'Expert Jetty/CometD developer,production,operations advice'

Follow-Ups:
- Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
  - From: Vicky Wu

References:
- [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
  - From: Vicky Wu
- Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
  - From: Joakim Erdfelt
- Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
  - From: Vicky Wu

Prev by Date: [jetty-dev] Sequential pushing with HTTP/2
Next by Date: Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
Previous by thread: Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
Next by thread: Re: [jetty-dev] long response time lead to high CPU in jetty when using httpd server in front and a short timeout setting
Index(es):
- Date
- Thread

Breadcrumbs