Inline ...
Hello Joakim,
Firstly, this is the setup (Scala)
code for the SslContextFactory etc.:
/***** start code *****/
val sslContextFactory = new
SslContextFactory
val ciphers = Array(
"TLS_AES_128_GCM_SHA256",
/* TLS 1.3 */
"TLS_AES_256_GCM_SHA384",
/* TLS 1.3 */
"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA",
/* TLS 1.0 / 1.1 - IE7/8/9 - Android
<= 4.3 */
"TLS_DHE_DSS_WITH_AES_256_GCM_SHA384",
"TLS_DHE_DSS_WITH_AES_128_GCM_SHA256",
"TLS_DHE_DSS_WITH_AES_256_CBC_SHA256")
sslContextFactory.setCipherComparator(HTTP2Cipher.COMPARATOR)
sslContextFactory.setUseCipherSuitesOrder(true)
Not relevant, anywhere anymore.
You can get an A+ rating on ssllabs
without attempting to control cipher
suite order.
(Note: A+ rating is for all HTTP/1.x
and HTTP/2 behaviors)
sslContextFactory.setIncludeCipherSuites(ciphers:_*)
Don't include, exclude only.
You eliminate a ton of valid Cipher
Suites doing it this way.
sslContextFactory.setExcludeCipherSuites("SSL_RSA_WITH_DES_CBC_SHA",
"SSL_DHE_RSA_WITH_DES_CBC_SHA",
"SSL_DHE_DSS_WITH_DES_CBC_SHA",
"SSL_RSA_EXPORT_WITH_RC4_40_MD5",
"SSL_RSA_EXPORT_WITH_DES40_CBC_SHA",
"SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA",
"SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA")
Use defaults, none of those Cipher
suites would be selected anyway due to
default Cipher Suite exclusions in
SslContextFactory AND the JVM disabled
cipher suites.
/** Default Excluded Cipher Suite
List */
private static final String[]
DEFAULT_EXCLUDED_CIPHER_SUITES = {
// Exclude weak / insecure
ciphers
"^.*_(MD5|SHA|SHA1)$",
// Exclude ciphers that
don't support forward secrecy
"^TLS_RSA_.*$",
// The following
exclusions are present to cleanup
known bad cipher
// suites that may be
accidentally included via include
patterns.
// The default enabled
cipher list in Java will not include
these
// (but they are available
in the supported list).
"^SSL_.*$",
"^.*_NULL_.*$",
"^.*_anon_.*$"
};
sslContextFactory.addExcludeProtocols("SSLv3")
This is the in the default excluded
protocols values, and also
excluded/disabled at the JVM level,
don't set it.
/** Default Excluded Protocols
List */
private static final String[]
DEFAULT_EXCLUDED_PROTOCOLS = {"SSL",
"SSLv2", "SSLv2Hello", "SSLv3"};
sslContextFactory.setRenegotiationAllowed(false)
sslContextFactory.setEndpointIdentificationAlgorithm("HTTPS")
This is only relevant for a client
initiating a connection to a destination
machine with TLS.
Comment out all of the above.
It screams old-school behaviors and
techniques.
Use defaults for SslContextFactory
when it comes to Cipher Suites and
Protocols, and don't attempt to support
TLS/1.0 or TLS/1.1 Those protocols are
disabled in so many places (User-Agents,
Browsers, OS level, Java VM level,
Server level, various hardware
intermediates) now its pointless to
attempt to support them. If this is
still important to you, make a different
connector for legacy behaviors.
sslContextFactory.setKeyStorePath(keyStore)
sslContextFactory.setKeyStorePassword(keyPassword)
sslContextFactory.setKeyManagerPassword(keyPassword)
sslContextFactory.setKeyStoreType(keyStoreType)
val httpsConfig = new
HttpConfiguration
httpsConfig.setSecureScheme("https")
httpsConfig.setSecurePort(port)
httpsConfig.setSendXPoweredBy(false)
httpsConfig.setSendServerVersion(false)
httpsConfig.setOutputBufferSize(32768)
httpsConfig.addCustomizer(new
SecureRequestCustomizer)
val http1 = new
HttpConnectionFactory(httpsConfig)
val connector =
{
if (httpVersion !=
"http/2")
{
val ssl = new
SslConnectionFactory(sslContextFactory,"http/1.1")
val http = new
HttpConnectionFactory(httpsConfig)
new
ServerConnector(server,ssl,http)
}
else
{
val http2 = new
HTTP2ServerConnectionFactory(httpsConfig)
val alpn = new
ALPNServerConnectionFactory
alpn.setDefaultProtocol(http1.getProtocol)
Use
alpn.setDefaultProtocol(http1.getDefaultProtocol())
val ssl = new
SslConnectionFactory(sslContextFactory,alpn.getProtocol)
new
ServerConnector(server,ssl,alpn,http2,http1)
}
}
if (address != null)
connector.setHost(address)
connector.setPort(port)
connector.setIdleTimeout(1000L
* idleTimeout)
server.addConnector(connector)
/***** end code *****/
Then for the ThreadPool part:
/***** start code *****/
val pool = new
QueuedThreadPool(maxThreads,minThreads,idleTimeout,new
ArrayBlockingQueue(queueSize))
val server = new Server(pool)
/***** end code *****/
HTTP/2 will seriously stress out your
thread pool.
Don't specify the queue, use
defaults.
Don't use less then defaults for
min/max threads with HTTP/2.
It's good that you are using the
QueuedThreadPool, it behaves much better
then a raw Executor with regards to
HTTP/2.
Threads are 10-100, idletimeouts I
have tried are from 1000 to 20000,
queueSize is now 500 but I have tried
both much smaller and larger values.
I have no idea what values would be
sensible. The number of concurrent
users is relatively low: up to a
couple of hundred, rarely more. Users
can do requests that take longish to
complete (sometimes up to 30 or even
60 seconds) but those are very rare.
The majority is relatively quick (<
50ms). Current load on the problematic
servers is very low (perhaps 10 users)
and as said: low memory and CPU close
to zero.
No problems with HTTP/1.1 with this
code/settings, problematic with
HTTP/2. Strangely enough some servers
seem fine (or users just did not
report any problems yet).
Start by using a default
QueuedThreadPool, no configuration, no
min thread, no max threads, all strictly
defaults.
Then monitor your usages and
behaviors with HTTP/2 enabled over at
least a week.
Then you can tune your maxThreads.
As to thread idle timeouts, don't
bother setting them yet.
Your connector.setIdleTimeout() is
far more useful to you.
Any help would be welcome.
Kind regards,
Silvio
Many folks have gotten HTTP/2 working
just fine in embedded-jetty.
The two most common issues:
1. Not having alpn support setup
properly
2. Too restrictive of an
SslContextFactory that is incompatible
with HTTP/2
When it comes to the ALPN suppport,
you have to worry about what JVM runtime
you are on first.
If you are using Java 8 then you have
to align your alpn-boot version to the
specific Java 8 JVM you are using.
Under Java 8, you MUST use the
-Xbootclasspath on JVM startup to have
alpn work properly.
However, if you are using Java 11
then you can use the
jetty-alpn-java-server artifact instead
(no bootclasspath requirement) to get
ALPN behaviors required for HTTP/2.
You also have a conscrypt option for
running ALPN + HTTP/2 on Jetty, which
will work the same regardless of Java 8
or Java 11.
When it comes to too restrictive of
an SslContextFactory we have the
following advice:
* If you are specifying Include
Cipher Suites you are too restrictive.
* Your certificates need to be RSA
(not DES) and at a higher bit size (like
2048+)
* If you are attempting to get old
Protocols (TLS/1.0 TLS/1.1) working,
then you are introducing incompatible
Cipher Suites too.
* If you are resetting the Exclude
Cipher Suites you are introducing
incompatible Cipher Suites.
* Avoid using Include Cipher Suites
(this exists for people stuck in the
past and not ever wanting to use HTTP/2)
* Avoid using Include Protocols (this
exists for people stuck supporting old
clients and not wanting to use HTTP/2)
* Add to the existing Exclude Cipher
Suites (if needed)
* Add to the existing Exclude
Protocols (if needed)
* Always use an Up to date
(not-expired) JVM
* Learn how to use the
SslContextFactory.dump() methods at
runtime to know what the current state
of your SslContextFactory is (including
all selected cipher suites and
protocols)
We have an example codebase that
shows how to start an HTTP/2 server
connector.
You can check the Jenkins project for
their usage of HTTP/2 on Embedded Jetty
as well. (their self executing WAR file
does just this)
You could also run h2spec against
your server and see if its behaving
properly.
- Joakim
On
20-11-18 16:06, Joakim Erdfelt
wrote:
How do you enable
HTTP/2 in your embedded
application?
Also, If you have configured
your SslContextFactory, what
does this configuration look
like?
Lastly, what does your
ThreadPool / Executor
configuration look like?
Hello
all,
We are running Jetty 9.4.14
embedded in an application that
runs on a
number of Ubuntu 18.10 servers.
Last night we switched from
HTTP1.1 only
to HTTP2. Today we are
experiencing serious problems.
Users can not
access the application or are
experiencing extremely long
latencies. At
the same time the server is
almost idle.
We have configured the Ubuntu
servers as almost default
setups.
OpenJDK11 with Jetty using pure
Java HTTPS. We have been doing
this for
years now and each time we
switched from HTTP1.1 to HTTP2
things started
falling apart. In the past we
had reports of people not being
able to
connect at all receiving
spurious messages from their
browsers (some of
which are undoubtedly outdated).
We expected things to have
improved by
now and decided to try again,
with the above result.
Are there additional things we
need to take care of when using
HTTP2
when compared to HTTP1.1? More
threads, different queue size,
more file
handles etc?
Any pointers would be very much
appreciated.
Kind regards,
Silvio
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options,
retrieve your password, or
unsubscribe from this list,
visit
https://www.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jetty-users
