Re: [jetty-users] Broken pipe after jetty client upgrade

[Silvio Bierman <sbierman@xxxxxxxxxxxxxxxxxx>]

Hallo Mark,

I encountered the same as I posted earlier but the list of dropped clients is a bit longer including android 4.0-4.3, IE8-10 on Win7, Java7 and Safari6.0.4.

Thanks for the fixing code, that was what I was looking for.

I am all for being aggressive on excluding weak ciphers/protocols but having something to offer the reference browsers on Qualys SSL-Labs is a must for us (customer demand an thus in our SLA).

Cheers,

Silvio

On 04/09/2016 07:47 AM, Mark Mielke wrote:

On Thu, Apr 7, 2016 at 6:13 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:

The whole issue of TLS/SSL protocols and ciphers is fraught and it is very difficult to make changes without having unexpected consequences.

We make  efforts to make sure that the jetty default configuration is up to date with the latest recommendations for both protocols and ciphers, as do the JVM vendors that provide the base set of protocols/ciphers available.

So if you are using a recent release of Jetty running on a recent JVM, then the default exclusion list is a reasonable approach.

It will not always be perfect, as we appear to live in an era of many discoveries of crypto vulnerabilities, so sometimes it may be necessary to manually configure the exclusions.  But as of today, we do not know of any such changes that we would recommend to deploy.

I agree with the sentiment above. It is hard getting the recipe right for everyone.

I want to share a very recent (this week) experience, to add context to this discussion. I am not saying a mistake was made, but this is more along the lines of better understanding the line between "security" and "function"

Recently, I upgraded a small but important embedded server that we have from Jetty 9.3.6 to Jetty 9.3.8. It worked fine throughout testing. However, once deployed to production, certain users reported failures. I tracked this down to users of Java 7u79 which is one of the last public releases of Java 7, and still widely used. In our company, we have access to the Java 7 non-public updates, and I had tested with 7u91, which was the most recent version pushed out to most of our users, and Java 7u91 worked fine. But the users on Java 7u79 missed this push somehow, and were encountering failures that looked like this:

Apr 05, 2016 9:44:24 AM org.apache.http.impl.execchain.RetryExec execute

INFO: I/O exception (javax.net.ssl.SSLHandshakeException) caught when processing request to

{s}->https://prism-cmti.ciena.com:443: Remote host closed connection during handshake

Apr 05, 2016 9:44:24 AM org.apache.http.impl.execchain.RetryExec execute

INFO: Retrying request to {s} 

The client is using Apache HttpClient.

I was aware of the TLS/SSL vulnerabilities and attempts to close them, so it didn't take me long to realize what had probably happened. I confirmed that Java 7u79 definitely failed, and Java 7u91 definitely worked. I then went back to Java 6 and I found that all versions of Java 6, including the most recent non-public patch Java 6u113 all failed. We still use Java 6 on Solaris 8, as Java 7 is not available for Solaris 8.

After narrowing down the cause of the problem, I found it was this commit in Jetty 9.3.7 that introduce the problem:

commit 0a1b0b2bc69ea7e7f5f44992f47a84f926cdeebb

Author: Joakim Erdfelt <joakim.erdfelt@xxxxxxxxx>

Date:   Wed Jan 13 15:00:16 2016 -0700

    485714 - Update SSL configuration to mitigate SLOTH vulnerability

In particular, this change:

--- a/jetty-util/src/main/java/org/eclipse/jetty/util/ssl/SslContextFactory.java

+++ b/jetty-util/src/main/java/org/eclipse/jetty/util/ssl/SslContextFactory.java

@@ -250,14 +250,10 @@ public class SslContextFactory extends AbstractLifeCycle

         setTrustAll(trustAll);

         addExcludeProtocols("SSL", "SSLv2", "SSLv2Hello", "SSLv3");

         setExcludeCipherSuites(

-                "SSL_RSA_WITH_DES_CBC_SHA",

-                "SSL_DHE_RSA_WITH_DES_CBC_SHA",

+                "^.*_RSA_.*_(MD5|SHA|SHA1)$",

                 "SSL_DHE_DSS_WITH_DES_CBC_SHA",

-                "SSL_RSA_EXPORT_WITH_RC4_40_MD5",

-                "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",

-                "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",

                 "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA");

-}

+    }

I did not fully debug the TLS/SSL protocol in use here, but I found that it was this particularly aggressive exclude that caused the grief:

+                "^.*_RSA_.*_(MD5|SHA|SHA1)$",

I believe this disables all RSA / SHA(1) ciphers, and that Java 7u79 and before, and all versions of Java 6, are unable to find a usable cipher to negotiate. (I did test with only MD5 and SHA1 excluded, and it still worked... so it seemed it was definitely the "SHA$" case that was the issue)

From a Java 7 perspective, I have asked that the clients be updated to Java 7u91 or later (and we will be moving to Java 8 where possible).

From a Java 6 perspective, I do not have any options here. As long as we need Solaris 8 (another separate discussion started), we need Java 6, and Java 6 needs to work with our server based upon Jetty. To address this temporarily, I have added these lines to our server code:

+        /*

+         * Jetty 9.3.7 changed the excluded cipher suites to reduce exposure to the SLOTH vulnerability. We

+         * still have clients on Solaris 8 running Java 6u113 or earlier, and clients on RHEL 6.x running Java

+         * 7u79 or earlier, that do not work with these changes. The following code restores the excluded

+         * cipher suites to match the default behaviour of Jetty 9.3.6 which was known to work for our

+         * clients. Java 7u91 or later, and Java 8u66 or later are not affected either way.

+         */

+        sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA", "SSL_DHE_RSA_WITH_DES_CBC_SHA",

+                                                 "SSL_DHE_DSS_WITH_DES_CBC_SHA",

+                                                 "SSL_RSA_EXPORT_WITH_RC4_40_MD5",

+                                                 "SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",

+                                                 "SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",

+                                                 "SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA");

Effectively, I am reverting the effects of the Jetty 9.3.7 commit

I do recognize that most people should not have a need to use Java 6 clients. I think it may be surprising to people that Java 7u79 clients may also not work (at least with our particular set of components, including Apache HttpClient).

I think my main point here is that there can be far-reaching consequences for these TLS/SSL lockdowns, and:

1. If you have a server application that uses Jetty (or any software that tries to implement a "secure" TLS/SSL implementation by default), you need to be aware of the potential issues introduced as these TLS/SSL lockdowns are put into force. Really, these changes are purposeful breaks in compatibility and they need to be recognized as such.
2. If you have a client that needs to run on very old Java 6, or only a year old Java 7u79, you need to be aware that this is a fragile condition to be in, and special precautions (extra testing, custom changes, ...) may need to be employed.

I am not asking for the above commit to be reverted In Jetty 9.3.x. Just looking to add some awareness here. We are reaching a point where the list of acceptable modern ciphers, has almost on overlap with the list of acceptable ciphers from a few years ago, and this means that any software that is stuck in time (for whatever reason), will become broken as a result.

Also not Jetty's fault, but I will point out that in many of these cases, one could draw the conclusion that TLS/SSL isn't worth it. If the ciphers are considered vulnerable to the point that upstream providers such as JSSE, Jetty, Apache HttpClient, OpenSSL, and so on, are going to purposefully break compatibility, and we have no choice to use more modern ciphers, then it may be simpler to disable https:// for these requirements. This verges on the "cure is worse than the disease" concern in that in order to prevent the risk of exposure of confidential information, we introduce the likelihood of actual breakage. I'm 50/50 joking/serious here. I intend to continue to use TLS/SSL wherever it makes sense, which is quite often (and especially for Internet use). But, there are cases where we have been burned several times by trying our best to use TLS/SSL, and the answer may be to minimize the risk in some other way, such as reducing the need for authentication and privacy and then not using these technologies for this use case any more. For example, if we have two servers in the same data center, managed by the same IT staff, on the same limited access VLAN... maybe TLS/SSL is over kill for requirements. NFS v3 is frequently used in these cases, and it is much less secure. :-)

--

Mark Mielke <mark.mielke@xxxxxxxxx>

_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-users