Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-dev] 8.1.2.v20120308 - rare mixup of requests 
Thomas Becker skrev:

Hi Per,

Hi
First of all, thanks for your quick response! Nice that someone really seem to care and want to actively participate in doing something about it. Have to say that this is not the case on all OSS projects.
this is not a known bug that has been fixed, but looks like a severe issue to investigate. 
Agree!
It doesn't look like a jvm issue as according to your description stuff doesn't seem to get mixed up on the wire.
That was also my conclusion, but Im not a Jetty expert, and havnt really dived very deep into it yet, so I wasnt sure. Just quickly wanted to mention my observations and interpretation in its current state, so that I could get reactions from Jetty community. Didnt want do go very deep into it, if you already knew all about the problem and had a fix. 

Is there a specific reason that you use a blocking connector?
Wasnt really aware that I had taken such a decision :-) Im just using Solr "out of the box". It ships with Jetty including a default/easy way of starting Solr running on Jetty. I basically just use that, so havnt done anything to configure Jetty og make any other specific decissions on how it should run.
Could you please try a SelectChannelConnector and see if the problem still occurs?
Yes certainly! But, in order to do it quicly, I probably would like you to tell me what I should do in order to acheive this. As I mentioned, I havnet really done any configuring or anything to Jetty before. I basically just looked a little at the code and decided to modify StreamEndPoint to do a little logging for me in order to see what it receives from the wire. This way I saw that "received from wire" seems to be fine, but "handed over to Filter/Servlet" seems to be mixed up with content from other requests. 
That information would give us some hints about where to look first.
Can you try to create a small webapp (maybe + simple loadtest) that reproduces the issue?
I can say that I definitely want to participate actively in narrowing in on the "bug" to have it fixed. Creating a small webapp that reproduces the problem "on your side" will certainly be a nice way forward, but if it is the best Im a little bit in doubt. We have several "unit-test"-sized tests that do a fair amount on concurrent requesting and we never came across the problem it such setups. So maybe this problem requires "bigger" setups or something - Im not really sure about the "properties" that need to hold in order for this to occur. Maybe another way forward is that you provide we with additional logging (or something) that I could insert into the code, run the "performance/endurance" test that (sometimes) produces this problem again, and provide the logging to you to evaluate in corrporation with me. Is that a path that could fit you? If we get closer to the cause of the problem following this path, we might be able to reproduce in a smaller ad-hoc test for quicker "turnaround" on potential fixes. Right now I just very much in doubt if I will be able to reproduce this problem in a fairly small "test" to hand over to you. 

Cheers,
Thomas

Regards, Per Steffensen


On 10/23/12 5:55 PM, Per Steffensen wrote:

Hi

I have found a bug in Jetty 8.1.2.v20120308.
Running with many concurrent clients sending requests to Solr (web-app running on top of Jetty) sometimes (very rarely) the content of the HTTP requests are mixed up when handed over by Jetty as HttpServletRequest-objects to the doFilter method of a javax.servlet.Filter configured in web.xml. When "hacking" org.eclipse.jetty.io.bio.StreamEndPoint to log something about what is actually received over the wire, requests does not seem to be mixed up "yet". So it is something that goes wrong in Jetty somewhere between receiving the data over sockets and handing the (parsed) request over to a filter (or servlet, probably) as a HttpServletRequest. 

A little about what I mean with "mixed up":

* Data in request A: abcdefghijklmnopqrstuvwxyz
* Data in request B: 1234567890987654321
Well the requests are longer in my concrete cases, and thats probably important, but not when just trying to illustrate
Data read by consecutive "fill"-calls in org.eclipse.jetty.io.bio.StreamEndPoint - e.g: 
Regarding request A:
* abcdefghij
* klmnopqrstu
* vwxyz
Regarding request B:
* 12345678909876
* 54321
Seems ok

Data "behind" HttpServletRequest-object handed over to filter/servlet:
Request A: abcdefghijklmnopqrstuvw345678909876

WUT!!!!
Closer investigations show that it is "always" (well I do not have much statistical material) the last part of the data read in the last "fill"-call by the "first request" (request A in this example) that is "magically" replaced by something from a "later request" (request B in this example) when the data reaches the filter/servlet. And that the size of the correct part of the data read in the last "fill"-call by request A is equal to the size of data read in the "violating" "fill"-call (request B) which is before the data the gets mixed from request B into request A. Never mind if you didnt understand (I almost did not :-) ) - let me illustrate by example. E.g. lets say (as the example above also suggests) that it is the last "fill"-call regarding request A that gets mixed up with data from the first "fill"-call regarding request B. The "end result" seen in request A can be: 
* abcdefghijklmnopqrstuv2345678909876
* abcdefghijklmnopqrstuvw345678909876
* abcdefghijklmnopqrstuvwx45678909876
* abcdefghijklmnopqrstuvwxy5678909876
But not e.g.
* abcdefghijklmnopqrstuvw5678909876
To me it really seems like org.eclipse.jetty.io.bio.StreamEndPoint reuse (/share) byte-arrays among requests, and that a byte-array used by request A will be "released" for usage by request B before request A has finished reading from it - maybe org.eclipse.jetty.io.bio.StreamEndPoint is finished using the byte-array for request A stuff, but layers higher in Jetty might not have ready from the byte-array yet to get the data for the request-A-HttpServletRequest-object. But what do I know?
I can deepen the explanation and I can provide more "evidence" or details about exactly how I revealed this bug. But first of all I just want to know: 
* Is this a known bug?
* Has it been fixed?

Regards, Per Steffensen
_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev


_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-dev

Back to the top