[jetty-dev] Content Encoding Error with Kibana running behind Jetty 9.3

[Sanjay Bhat <Sanjay.Bhat@xxxxxxxxxxxxxx>]

Hi all,

 

Environment:

• In my environment, Kibana 4.5.2 is running behind a "reverse proxy"
  which is created using Jetty's ProxyServlet class.
• This is done so that the Kibana web interface can be accessed using the URL https://Jetty_Server_IP:8443/visual-analytics/proxy/... Requests for this URL are intercepted by the ProxyServlet running in the Jetty Server and redirected to https://localhost:5601/... i.e., to the Kibana Server that is running in the same machine as the Jetty Server.
• The Kibana Server then processes the request forwarded by the Jetty Server and returns the response back to the web browser.
• NOTE: Jetty is running in "embedded mode" within my application.

 

Issue:

• The above setup was working fine with Jetty version 8.1.15.v20140411 i.e., the Kibana web interface could be launched successfully in the web browser using the URL https://Jetty_Server_IP:8443/visual-analytics/proxy/...
• I upgraded the Jetty version from 8.1.15.v20140411 to 9.3.14.v20161028 and did the required code changes to make the embedded Jetty and ProxyServlet work with the upgraded Jetty version.
• However, after the Jetty upgrade, I am not able to launch the Kibana web interface using the URL https://Jetty_Server_IP:8443/visual-analytics/proxy/... in any of the web browsers - Firefox, Chrome and IE. The following "content encoding/decoding" related error is shown in the web browser:
  • Firefox: Content Encoding Error - The page you are trying to view cannot be shown because it uses an invalid or unsupported form of compression. Please contact the website owners to inform them of this problem.
  • Chrome: This site can’t be reached - The web page at https://164.99.175.139:8443/visual-analytics/proxy/app/kibana... might be temporarily down or it may have moved permanently to a new web address. ERR_CONTENT_DECODING_FAILED
  • IE: This page can’t be displayed. Make sure the web address is correct. Look for the page with your search engine. Refresh the page in a few minutes.

 

Workaround:

• In Firefox, I accessed the advanced settings using "about:config" and changed the value of the "network.http.accept-encoding.secure" setting from the default value of "gzip, deflate, br" to "true".
• After this, the Kibana web interface could be launched successfully in Firefox.

 

Request and response headers:

Headers for working scenario with Jetty 8.1.15.v20140411:

General

·        Request URL:https://10.204.103.81:8443/visual-analytics/proxy/app/kibana

·        Request Method:GET

·        Status Code:200 OK

·        Remote Address:10.204.103.81:8443

Response Headers

·        Cache-Control:no-cache

·        Content-Encoding:gzip

·        Content-Length:12506

·        Content-Type:text/html

·        Date:Mon, 19 Dec 2016 10:57:16 GMT

·        Date:Mon, 19 Dec 2016 10:57:16 GMT

·        kbn-name:kibana

·        kbn-version:4.5.2-snapshot

·        Vary:accept-encoding

·        X-Frame-Options:SAMEORIGIN

Request Headers

·        Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8

·        Accept-Encoding:gzip, deflate, sdch, br

·        Accept-Language:en-GB,en-US;q=0.8,en;q=0.6

·        Cache-Control:max-age=0

·        Connection:keep-alive

·        Cookie:Spiffy_Session="X-SAML%2Ccbe9ca8dc1b8c5548010698626a3d463fc098c58c2f08d5c8f74c78d345d86b1%2Chttps%3A%2F%2Flocalhost%3A8443%2FSentinelAuthServices%2Fauth%2Ftokens%2Chttps%3A%2F%2Flocalhost%3A8443%2FSentinelAuthServices%2Fauth%2Ftokens%2Cadmin"

·        Host:10.204.103.81:8443

·        Referer:https://10.204.103.81:8443/sentinel/views/main.html

·        Upgrade-Insecure-Requests:1

·        User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

 

Headers for non-working scenario with Jetty 9.3.14.v20161028:

General

·        Request URL:https://164.99.175.139:8443/visual-analytics/proxy/app/kibana

·        Request Method:GET

·        Status Code:200 OK

·        Remote Address:164.99.175.139:8443

Response Headers

·        Cache-Control:no-cache

·        Content-Encoding:gzip

·        Content-Type:text/html

·        Date:Mon, 19 Dec 2016 10:51:18 GMT

·        Date:Mon, 19 Dec 2016 10:51:18 GMT

·        kbn-name:kibana

·        kbn-version:4.5.2-snapshot

·        Transfer-Encoding:chunked

·        Vary:accept-encoding

·        X-Frame-Options:SAMEORIGIN

Request Headers

·        Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8

·        Accept-Encoding:gzip, deflate, sdch, br

·        Accept-Language:en-GB,en-US;q=0.8,en;q=0.6

·        Cache-Control:max-age=0

·        Connection:keep-alive

·        Cookie:Spiffy_Session=X-SAML%2Cf9a479e40a0f79ee94ca3177d2ec1e493e0d5e00ed4dd5e7221f5ff448ed6325%2Chttps%3A%2F%2Flocalhost%3A8443%2FSentinelAuthServices%2Fauth%2Ftokens%2Chttps%3A%2F%2Flocalhost%3A8443%2FSentinelAuthServices%2Fauth%2Ftokens%2Cadmin

·        Host:164.99.175.139:8443

·        Upgrade-Insecure-Requests:1

·        User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36

 

Need help:

• The error message shown in the web browser and the workaround that I tried indicate that there is some issue with the "encoding/decoding" of the response that is received by the web browser after the Jetty upgrade from version 8.1.15.v20140411 to 9.3.14.v20161028.
• Difference in the request and response headers:
  • In the working scenario, the response has the "Content-Length" header. In the non-working scenario, the response doesn't have the "Content-Length" header but has the "Transfer-Encoding" header set to "chunked" instead.
  • In the working scenario, the request has the "Referer" header but this request header is missing in the non-working scenario.
• Can anyone please guide me on how to debug this issue further and fix it?

 

Looking forward for any response…

 

Regards,

Sanjay