Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-dev] Solr unable to upgrade to Jetty 9.3 from Jetty 9.2.
Mark,

no probs - we are very please that Jetty is used by Solr, so if you have issues then err on the side of contacting us sooner rather than later - we are here to help!

cheers


On 8 January 2016 at 10:59, Mark Miller <markrmiller@xxxxxxxxx> wrote:
Great, thanks for the attention Greg.

- Mark

On Thu, Jan 7, 2016 at 6:58 PM Greg Wilkins <gregw@xxxxxxxxxxx> wrote:
Mark,

I think both Jetty and Apache are acting reasonably.   Jetty saw a servlet terminate without consuming all of it's content, so it is entitled to think this is an error case and be prudent and terminate the connection.   Apache sees a terminated connection before all the content is sent, so it too is correct to throw an exception saying the request did not terminate correctly.

I've posted on the SOLR issues a suggestion that you modify SOLR to always consume the content.  This can be done with a simple filter (or not so simple filter if you are async, but it still can be done).

Otherwise it is too difficult for Jetty to determine if an abnormally terminated request handler is actually just a sub-optimal termination and the app would prefer if we consumed all the content and continued.   That case is very hard for Jetty to distinguish from an app that failed really abnormally and there is lots of content being sent bit by bit by a DOS attacker.

cheers






On 8 January 2016 at 10:53, Mark Miller <markrmiller@xxxxxxxxx> wrote:


On Thu, Jan 7, 2016 at 5:44 PM Greg Wilkins <gregw@xxxxxxxxxxx> wrote:

Mark,

This is a change in behaviour in jetty 9.3, but not one that should affect any application.

The problem with consumeAll previously is that it was an unlimited commitment - a client could send chunks forever and consume a thread forever.

The only reason we need to consume all is to maintain a persistent connection.  If we don't consumeAll, we can close that connection and the client needs to start a new one.

So the approach that 9.3 takes is that if the application has not read all it's content (which is normally an error condition), then jetty will make a best effort attempt to read the content, but if it has to block - it gives up and the connection should be closed.

So I don't see how this can break an application - unless the client is trying to do pipelining and not correctly retrying queued requests if the connection closes  OR perhaps jetty is not correctly closing the connection (this I will check shortly).

Can you tell us more about how this is causing a problem for SOLR?     Why is the content not consumed?  Why does it cause a problem if Jetty does not consume it?

Basically, an exception on the server (which can happen when an update fails), ends consuming the request on the server. If the client takes a moment to end the request, the server will reset the connection. We would prefer to reuse the connection in these cases, because it's possible to have many update fails in a group and we don't want that to spin up a storm of connections. We would prefer the client to manage the connection. In this case, it seems we have 10-12ms to get the request close chunk sent instead of the previous 30 seconds before other timeouts hit.

But really, part of issue for us is that Apache HttpClient also gets messed up by this. It will fail trying to finish the request and instead of reporting the error from the server, will report an internal error where it tries to write to the reset connection on outputstream close, flush. That exception comes out of HttpClient and you don't even have access to the exception the server hit.

- Mark

 


cheers










On 8 January 2016 at 03:48, Mark Miller <markrmiller@xxxxxxxxx> wrote:


On Thu, Jan 7, 2016 at 10:52 AM Joakim Erdfelt <joakim@xxxxxxxxxxx> wrote:
This would be news.


Yes, and SOLR-8453.
 
Have you verified that the client sends the correct data via wireshark (or tcpdump) yet?

Yes, it looks correct to me. It's also the same code that has worked against many other Jetty versions.

I'm pretty confident the behavior has changed, and if the whole request stream has not been consumed, if the client pauses too long before finishing the request, the server will reset the connection. Previously it would wait and consume for the client until it was done, or unrelated timeouts were hit. That would leave the connection around for reuse.

- Mark
 

We test regularly with various http clients, under load testing and in real world scenarios (browsers, command line, other http client libs, our http client libs, rest libs, etc...).
While we have an occasional report of a quirk or bug, we investigate those.
Of the ones reported against Jetty 9.3, we have identified a few bugs related to improper handling of multipart data parsing on our side, and fixed those.
Others have either been bugs in the http client usage, or bad http client assumptions / expectations (such as thinking that once a response starts, the request is complete).

AFAIK, there's only 1 outstanding bug, and its a report of strange Request object handling with Handlers under load when the server is on Windows (which we are currently investigating).
 

Joakim Erdfelt / joakim@xxxxxxxxxxx

On Thu, Jan 7, 2016 at 8:12 AM, Mark Miller <markrmiller@xxxxxxxxx> wrote:
After upgrading Solr to Jetty 9.3, we noticed a lot of Solr tests started to randomly fail, and a couple would consistently fail.

After some investigation, I tracked down two issues that I think are involved.

One issue seems to be based on Locale. I think there is a default Locale bug or something, but I still have to dig into that one. More to come.


Specifically, it seems to be an HttpInput#consumeAll change.

The code went from:

T item = getNextContent();
if (item == null)
 _contentState.waitForContent(this);

To:

Content item = nextContent();
if (item == null)
break; // Let's not bother blocking

Is that legal? And even if it is, isn't it at least a pretty big back compat break?

We can hit this case quite easily - and in 9.2 Jetty seems to properly wait on the server until the client is done sending and consumes all it should. In 9.3, if there is even a short pause in the wrong place on the client side, the server resets the connection on the client as it consumeAll bails early now. That messes with the client (broken pipe, connection reset) and prevents connection reuse.

Comments?

- Mark
--

_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev

_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev
--

_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev



--
--



--
--



--

Back to the top