Re: [jetty-users] Response > Buffer + Continuations = Deadlock?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [jetty-users] Response > Buffer + Continuations = Deadlock?

From: Matthew Painter <matthew.painter@xxxxxxxxxx>
Date: Fri, 11 Nov 2011 22:13:36 +0000
Delivered-to: jetty-users@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/jetty-users>
List-help: <mailto:jetty-users-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/jetty-users>, <mailto:jetty-users-request@eclipse.org?subject=unsubscribe>

Ah! I tell a lie, there is! Apologies for not spotting it earlier - sleep deprived with a month old baby ;)

*grovel*

Thanks for your help!

On Fri, Nov 11, 2011 at 9:57 PM, Matthew Painter <matthew.painter@xxxxxxxxxx> wrote:

Hi Jan,

No, there's no synchronization on the continuation object in my code.

I have been able to make it work by increasing the response buffer size, so it is definitely linked - however this is clearly not a production fix :)

Let me know if I can provide any further information.

Thanks

Matt

On Fri, Nov 11, 2011 at 9:44 PM, Jan Bartel <janb@xxxxxxxxxxx> wrote:
Matthew,

Does your code synchronize on the Continuation object at all? If so,
don't use it
as the gate for concurrent access, but rather an application object
(see a thread
on this list entitled "Asynchronous servlets - thread
safety/synchronization & reusing objects").

If your code does not synchronize on the Continuation, then I'll dig
further into the
code, but on a pretty quick inspection (admittedly of trunk, not 7.5.1), I
couldn't see where Thread-111 would lock the AsyncContinuation, hence
my question.

cheers
Jan

On 12 November 2011 05:10, Matthew Painter <matthew.painter@xxxxxxxxxx> wrote:
> Sorry for the multiple emails :)
>
> It is a deadlock between the Selector thread, which owns SSLChannelEndpoint
> and is waiting on AsyncContinuation, and the thread writing to the output
> stream, which is waiting on SSLChannelEndpoint and owns AsyncContination.
> Details below. All help appreciated - even if just to tell me to open a bug
> report :)
> Thanks
> Matt
> Thread [pool-1-thread-5 Selector0] (Suspended)
> owns: SslSelectChannelEndPoint (id=338)
> waiting for: AsyncContinuation (id=8895)
> AsyncContinuation.isSuspended() line: 173
> SelectChannelConnector$SelectChannelHttpConnection(HttpConnection).isSuspended()
> line: 653
> SslSelectChannelEndPoint(SelectChannelEndPoint).isReadyForDispatch() line:
> 316
> SslSelectChannelEndPoint(SelectChannelEndPoint).schedule() line: 164
> SelectorManager$SelectSet.doSelect() line: 607
> SelectorManager$1.run() line: 283
> ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1110
> ThreadPoolExecutor$Worker.run() line: 603
> Thread.run() line: 722
>
> Daemon Thread [Thread-111] (Suspended)
> owns: AsyncContinuation (id=8895)
> waiting for: SslSelectChannelEndPoint (id=338)
> owned by: Thread [pool-1-thread-5 Selector0] (Suspended)
> SslSelectChannelEndPoint(SelectChannelEndPoint).blockWritable(long) line:
> 371
> HttpGenerator(AbstractGenerator).blockForOutput(long) line: 530
> HttpConnection$Output(HttpOutput).write(Buffer) line: 154
> HttpConnection$Output(HttpOutput).write(byte[], int, int) line: 96
> Utf8Generator._flushBuffer() line: 1437
> Utf8Generator.writeString(String) line: 465
> StdSerializers$StringSerializer.serialize(String, JsonGenerator,
> SerializerProvider) line: 123
> StdSerializers$StringSerializer.serialize(Object, JsonGenerator,
> SerializerProvider) line: 113
> MapSerializer.serializeFields(Map<?,?>, JsonGenerator, SerializerProvider)
> line: 257
> MapSerializer.serialize(Map<?,?>, JsonGenerator, SerializerProvider) line:
> 177
> MapSerializer.serialize(Object, JsonGenerator, SerializerProvider) line: 22
> BeanPropertyWriter.serializeAsField(Object, JsonGenerator,
> SerializerProvider) line: 428
> BeanSerializer.serializeFields(Object, JsonGenerator, SerializerProvider)
> line: 245
> BeanSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
> 212
> ContainerSerializers$IndexedListSerializer.serializeContents(List<?>,
> JsonGenerator, SerializerProvider) line: 291
> ContainerSerializers$IndexedListSerializer.serializeContents(Object,
> JsonGenerator, SerializerProvider) line: 242
> ContainerSerializers$IndexedListSerializer(ContainerSerializers$AsArraySerializer<T>).serialize(T,
> JsonGenerator, SerializerProvider) line: 130
> BeanPropertyWriter.serializeAsField(Object, JsonGenerator,
> SerializerProvider) line: 428
> BeanSerializer.serializeFields(Object, JsonGenerator, SerializerProvider)
> line: 245
> BeanSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
> 212
> StdSerializerProvider._serializeValue(JsonGenerator, Object) line: 587
> StdSerializerProvider.serializeValue(SerializationConfig, JsonGenerator,
> Object, SerializerFactory) line: 245
> JsonUtils$MyObjectMapper(ObjectMapper)._configAndWriteValue(JsonGenerator,
> Object) line: 1993
> JsonUtils$MyObjectMapper(ObjectMapper).writeValue(OutputStream, Object)
> line: 1563
> JsonUtils.toJson(Object, OutputStream) line: 340
> SearchServlet.writeResponse(ServletResponse, Object) line: 601
> SearchServlet.access$2(SearchServlet, ServletResponse, Object) line: 598
> SearchServlet$3.hasFinished(PartialSearchContinuation) line: 392
> SearchServlet$3.run() line: 414
> ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1110
> ThreadPoolExecutor$Worker.run() line: 603
> Thread.run() line: 722
>
> On Fri, Nov 11, 2011 at 5:37 PM, Matthew Painter
> <matthew.painter@xxxxxxxxxx> wrote:
>>
>> It would seem similar to this:
>>
>> http://www.servlets.com/archive/servlet/ReadMsg?msgId=559312&listName=jetty-support
>> Although this was said to be fixed.
>>
>>
>> On Fri, Nov 11, 2011 at 5:23 PM, Matthew Painter
>> <matthew.painter@xxxxxxxxxx> wrote:
>>>
>>> Hi all,
>>> I wonder if any of you bright sparks can shed some light on this problem
>>> with Jetty 7.5.1?
>>> I have a response that is bigger than the standard servlet buffer size
>>> (32,768).
>>> I am using the continuation suspend/complete pattern.
>>> When writing to the response outputstream, 32768 bytes are written to the
>>> client, and then the thread hangs forever on
>>> SelectChannelEndpoint.blockWritable() - maybe deadlocked?
>>> This is a huge issue for us, as it brings down the server as threads get
>>> deadlocked.
>>> Any information would be great - I can see a few similar bugs have been
>>> raised, but none quite the same.
>>> Thanks :)
>>> Daemon Thread [Thread-111] (Suspended)
>>> Object.wait(long) line: not available [native method]
>>> SelectChannelEndPoint.blockWritable(long) line: 384
>>> HttpGenerator(AbstractGenerator).blockForOutput(long) line: 530
>>> HttpGenerator(AbstractGenerator).flush(long) line: 456
>>> HttpConnection$Output(HttpOutput).flush() line: 89
>>> HttpConnection$Output.flush() line: 995
>>> HttpConnection$Output(HttpOutput).write(Buffer) line: 172
>>> HttpConnection$Output(HttpOutput).write(byte[], int, int) line: 96
>>> Utf8Generator._flushBuffer() line: 1437
>>> Utf8Generator.writeString(String) line: 465
>>> StdSerializers$StringSerializer.serialize(String, JsonGenerator,
>>> SerializerProvider) line: 123
>>> StdSerializers$StringSerializer.serialize(Object, JsonGenerator,
>>> SerializerProvider) line: 113
>>> MapSerializer.serializeFields(Map<?,?>, JsonGenerator,
>>> SerializerProvider) line: 257
>>> MapSerializer.serialize(Map<?,?>, JsonGenerator, SerializerProvider)
>>> line: 177
>>> MapSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
>>> 22
>>> ContainerSerializers$IndexedListSerializer.serializeContents(List<?>,
>>> JsonGenerator, SerializerProvider) line: 291
>>> ContainerSerializers$IndexedListSerializer.serializeContents(Object,
>>> JsonGenerator, SerializerProvider) line: 242
>>>
>>> ContainerSerializers$IndexedListSerializer(ContainerSerializers$AsArraySerializer<T>).serialize(T,
>>> JsonGenerator, SerializerProvider) line: 130
>>> MapSerializer.serializeFields(Map<?,?>, JsonGenerator,
>>> SerializerProvider) line: 257
>>> MapSerializer.serialize(Map<?,?>, JsonGenerator, SerializerProvider)
>>> line: 177
>>> MapSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
>>> 22
>>> MapSerializer.serializeFields(Map<?,?>, JsonGenerator,
>>> SerializerProvider) line: 257
>>> MapSerializer.serialize(Map<?,?>, JsonGenerator, SerializerProvider)
>>> line: 177
>>> MapSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
>>> 22
>>> MapSerializer.serializeFields(Map<?,?>, JsonGenerator,
>>> SerializerProvider) line: 257
>>> MapSerializer.serialize(Map<?,?>, JsonGenerator, SerializerProvider)
>>> line: 177
>>> MapSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
>>> 22
>>> ContainerSerializers$IndexedListSerializer.serializeContents(List<?>,
>>> JsonGenerator, SerializerProvider) line: 291
>>> ContainerSerializers$IndexedListSerializer.serializeContents(Object,
>>> JsonGenerator, SerializerProvider) line: 242
>>>
>>> ContainerSerializers$IndexedListSerializer(ContainerSerializers$AsArraySerializer<T>).serialize(T,
>>> JsonGenerator, SerializerProvider) line: 130
>>> BeanPropertyWriter.serializeAsField(Object, JsonGenerator,
>>> SerializerProvider) line: 428
>>> BeanSerializer.serializeFields(Object, JsonGenerator, SerializerProvider)
>>> line: 245
>>> BeanSerializer.serialize(Object, JsonGenerator, SerializerProvider) line:
>>> 212
>>> StdSerializerProvider._serializeValue(JsonGenerator, Object) line: 587
>>> StdSerializerProvider.serializeValue(SerializationConfig, JsonGenerator,
>>> Object, SerializerFactory) line: 245
>>> ObjectMapper(ObjectMapper)._configAndWriteValue(JsonGenerator, Object)
>>> line: 1993
>>> ObjectMapper(ObjectMapper).writeValue(OutputStream, Object) line: 1563
>>> ...
>>>
>>>
>>
>
>

> _______________________________________________
> jetty-users mailing list
> jetty-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/jetty-users
>
>
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users

References:
- [jetty-users] Response > Buffer + Continuations = Deadlock?
  - From: Matthew Painter
- Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
  - From: Matthew Painter
- Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
  - From: Matthew Painter
- Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
  - From: Jan Bartel
- Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
  - From: Matthew Painter

Prev by Date: Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
Next by Date: [jetty-users] Sealed jar file in both main program (start.jar) and web application
Previous by thread: Re: [jetty-users] Response > Buffer + Continuations = Deadlock?
Next by thread: [jetty-users] Sealed jar file in both main program (start.jar) and web application
Index(es):
- Date
- Thread

Breadcrumbs