Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-users] Upgrading to Jetty 9.1
While others digest that (seems more than fine to me except for the hang),  will try to replicate in the next couple of days. 

Wasn't going to be doing our upgrade for a month or so, but seems worth dipping toe in water to see if this will be an issue for us also.

We also use "selective" chunking but _do_ use one connect.   Any reason why you have one connect off ?

Oh, the request header X-Forwarded-SSL.    I thought Jetty looks for X-Forwarded-Proto with value "https" to enable secure session cookies.

Oh, and totally irrelevant, why are your session cookies so huge and containing a repeat of the first part.  Feel free to ignore this :)


On 29 January 2014 00:30, Stefan Magnus Landrø <stefan.landro@xxxxxxxxx> wrote:
Hi there,

I've been looking further into the issue, and it seems clear what is going on.

This is what happens when running requests through our big ip (not using one connect): 

jetty 9.1:

whenever a request contains a connection:keep-alive header (the default for all browsers I believe, but not for curl), curl hangs until the keep alive timeout is reached. Also, jetty replies with a connection: keep-alive header in the response.

whenever a request does not contain a connection:keep-alive header, curl does not hang, and jetty doesn't add a connection:keep-alive to the response

jetty 8:

it doesn't matter if the request contains a connection:keep-alive header or not, curl does not hang under any circumstance and jetty will never add a connection:keep-alive header in the response.

In addition, it turns out that big ip, removes the transfer-encoding:chunked header that jetty generated (see local debug block) from the response (we're using the recommended selective response chunking mode in the big ip http profile). In my opinion that is wrong, since it doesn't add the Content-Length header, something which would require the connection to be closed in order for the client to know when there is no more content.

BTW, the servlet is spitting out the request headers as-is.


Comments, anyone?
 

*************
  Jetty 9.1 (through big ip)
*************


➜  tmp  curl -i https://whereever.no
HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
X-MiniProfiler-Ids: ["c2600ed9-5e37-4d86-bfd0-f6330c818961"]
Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098g74r8cd0loxh65ry63fq5mle.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
X-Content-Type-Options: nosniff
Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/
Cache-Control: no-cache, no-store, no-transform
Content-Type: text/plain
Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/
Vary: Accept-Encoding

Header name X-Forwarded-SSL
 - Header true
Header name X-Forwarded-For
 - Header 139.112.144.209
Header name User-Agent
 - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
Header name Accept
 - Header */*
Header name Host
 - Header whereever.no

 Does not hang


+++++++++++++++++++

curl -i  -H "Connection: Keep-Alive" https://whereever.no
HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
X-MiniProfiler-Ids: ["7549c3f2-c396-4b26-802d-40350aa64d4c"]
Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee0981q2s047k9wh6aspppnrzwaxbe.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
X-Content-Type-Options: nosniff
Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/
Cache-Control: no-cache, no-store, no-transform
Content-Type: text/plain
Connection: keep-alive
Set-Cookie: BIGipServerpool_sticky=111834251.3879.0000; path=/
Vary: Accept-Encoding

Header name X-Forwarded-SSL
 - Header true
Header name X-Forwarded-For
 - Header 139.112.144.209
Header name User-Agent
 - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5
Header name Connection
 - Header keep-alive
Header name Accept
 - Header */*
Header name Host
 - Header whereever.no

 This one hangs


*************
  Jetty 8 (through big ip)
*************

 ➜  tmp  curl -i https://whereever.no
HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
X-MiniProfiler-Ids: ["edba831e-bee6-46e2-bb05-74ecf4dd211f"]
Set-Cookie: SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pqsvn1ha8o6ve3zjba0wq42s.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
X-Content-Type-Options: nosniff
Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/
Cache-Control: no-cache, no-store, no-transform
Content-Type: text/plain
Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/
Vary: Accept-Encoding

Header name Host
 - Header whereever.no
Header name Accept
 - Header */*
Header name X-Forwarded-For
 - Header 139.112.144.209
Header name X-Forwarded-SSL
 - Header true
Header name User-Agent
 - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5

 Does not hang

+++++++++++++++++++


 tmp  curl -i  -H "Connection: Keep-Alive" https://whereever.no
HTTP/1.0 200 Connection established

HTTP/1.1 200 OK
X-MiniProfiler-Ids: ["509b4ffb-cd28-4dfc-9f29-6d05571162c5"]
Set-Cookie: SESSION_COOKIE=8e9b06084ab5cf8d683e98c2cc3aece3pul9pgxgmzmt14loahxebxjab.8e9b06084ab5cf8d683e98c2cc3aece3;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
X-Content-Type-Options: nosniff
Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/
Cache-Control: no-cache, no-store, no-transform
Content-Type: text/plain
Set-Cookie: BIGipServerpool_sticky=95057035.3879.0000; path=/
Vary: Accept-Encoding

Header name Host
 - Header whereever.no
Header name Accept
 - Header */*
Header name X-Forwarded-For
 - Header 139.112.144.209
Header name X-Forwarded-SSL
 - Header true
Header name Connection
 - Header keep-alive
Header name User-Agent
 - Header curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8y zlib/1.2.5

  Does not hang



*************
  Local test (not through big ip)
*************

this is jetty 8, but it looks similar for jetty 9 with regards to the cunked transfer-encoding

$ curl -i -H "Connection: keep-alive" http://localhost:12345/whereever
HTTP/1.1 200 OK
X-MiniProfiler-Ids: ["c9172169-755c-4bc2-a03d-9049e2b77e95"]
Set-Cookie: SESSION_COOKIE=f71e3e84ab741db2c5a8d20df07ee098r6navchnwslj1xn8vwlkrqoq2.f71e3e84ab741db2c5a8d20df07ee098;Path=/;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Permitted-Cross-Domain-Policies: master-only
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Content-Security-Policy-Report-Only: default-src 'none'; report-uri /post/cspreport/
Cache-Control: no-cache, no-store, no-transform
Content-Type: text/plain
Transfer-Encoding: chunked

Header name Host
 - Header localhost:12345
Header name Accept
 - Header */*
Header name Connection
 - Header keep-alive
Header name User-Agent
 - Header curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5





2014-01-28 Stefan Magnus Landrø <stefan.landro@xxxxxxxxx>

Now new infrastructure, so this is definitely triggered by upgrading jetty. However, the actual issue might be caused by new behaviour in jetty triggering a bug in big ip. 

Stefan  


2014-01-28 "" <lord.buddha@xxxxxxxxx>

Stefan, 

You said this was an 8.x to 9.x upgrade, so if the F5 was existing, it shouldn't be an issue unless there are infrastructure changes at the same time for upgrade.  We use F5 and the only time we have seen something similar was where a perimeter f/w was dropping an established connection (due to inactivity) within the keep alive time.   When the f/w does that the browser does not know the connection it has is stuffed.

tcpdump is the best approach.




On 28 January 2014 08:45, Stefan Magnus Landrø <stefan.landro@xxxxxxxxx> wrote:
Thanks, Simone! I'm afraid I have to agree with you - this probably won't fix my issue. I'll do some tcpdumping in the next few days - I'll keep you posted. This might as well turn out being a f5 big ip issue.

Stefan


2014-01-27 Simone Bordet <sbordet@xxxxxxxxxxx>

Hi,

On Fri, Jan 24, 2014 at 9:36 PM, "" <lord.buddha@xxxxxxxxx> wrote:
> Good find and a _critical_ bug in my view.
>
> A change that came with 9.0 as it was this in 8.x :-
>
>                      _header.put(CONNECTION_KEEP_ALIVE);
>                      if (connection!=null)
>                      {
>                          _header.setPutIndex(_header.putIndex()-2);
>                          _header.put((byte)',');
>                          _header.put(connection.toString().getBytes());
>                          _header.put(CRLF);
>                      }
>
> which had less logic, shorter lines... Yes, it copied 2 more bytes than it
> needed to but I would prefer less logic/shorter code over saving 2 bytes.
>
> To fix this, you could _try_ turning off one connect on the f5 if you don't
> need it for other reasons.
>
> We are also behind an F5, and are just about to start looking at moving from
> 8.x to 9.x.   ... so thanks ... will raise on support.

I filed https://bugs.eclipse.org/bugs/show_bug.cgi?id=426739 and fixed it.

Note that this is a corner case that should happen very rarely, only
when the "Connection" header gets some custom value that is not
"close" or "upgrade" or "keep-alive".

Therefore, I am dubious that the original problem (curl hanging for
chunked responses) is fixed by this fix.

Let us know if that is the case, though.

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users



--
BEKK Open
http://open.bekk.no

TesTcl - a unit test framework for iRules

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users



_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users








--
BEKK Open
http://open.bekk.no

TesTcl - a unit test framework for iRules

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users


Back to the top