Re: [jetty-users] Upgrading to Jetty 9.1

[Stefan Magnus Landrø <stefan.landro@xxxxxxxxx>]

Hi Jan,

2014-01-30 Jan Bartel <janb@xxxxxxxxxxx>:

Is this referring to jetty-8 or jetty-9?

 

Jetty 8 and Jetty 9.1 (both latest and greatest)

Actually, Jetty 9.1 performs slightly better (in our setup, it can handle 10% more concurrent logins (actually scenario below) - 50 logins/sec instead of 45 logins/sec) 

 

Recently in jetty-9 there was
some unnecessary synchronization removed. However, I'm not sure that
the synchronization is necessarily the limiting factor here ....
database access tends to be the culprit speedwise, so it would be
interesting to know about your environment and what you are observing.

 

Setup is as follows in our performance test environment:

3 linux boxes - 2 CPUs (4 cores per CPU + hyperthreading). Sticky sessions bigip load balancer.

Dedicated DB server - Linux -  2 CPUs (4 cores per CPU + hyperthreading) - Oracle 11g

None of the servers are suffering cpu- or / load-wise at any time during the tests.

Scenario:

authenticate

get home page

wait 60 secs

get home page

wait 60 secs

logout

When requests start failing/timing out, a thread dump from one of the app servers typically looks like this:

378 occurrences: 

"qtp994485315-105" prio=10 tid=0x0000000010efb800 nid=0x7061 waiting for monitor entry [0x00000000479c1000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.eclipse.jetty.server.session.AbstractSessionIdManager.newSessionId(AbstractSessionIdManager.java:121)

- waiting to lock <0x00000000c5ea0290> (a no.customer.project.jetty.config.ProjectJDBCSessionIdManager)

at org.eclipse.jetty.server.session.AbstractSession.<init>(AbstractSession.java:77)

     

93 occurrences:

  "qtp994485315-327" prio=10 tid=0x00002aaac4105000 nid=0x7308 waiting for monitor entry [0x0000000054187000]

   java.lang.Thread.State: BLOCKED (on object monitor)

at org.eclipse.jetty.server.session.AbstractSessionManager.addSession(AbstractSessionManager.java:717)

- waiting to lock <0x00000000c5ea0290> (a no.customer.project.jetty.config.ProjectJDBCSessionIdManager)

Given that a remote SQL-statement execution takes +/- 10 ms (which i believe is quite decent), it is not very surprising that we see all these blocked threads. 

You may want to try the jetty mongodb session clustering solution?

Introducing a new db technology is not at option at this time, and we're using hazelcast and jdbc databases already.

It would be great to have a Hazelcast-based session implementation -
might be a good opportunity to collaborate between your company and
Webtide?

 

Indeed. I'll get back to you concerning this. 

BTW, we've tried the hazelcast-based - session(id)manager found at https://github.com/jalpedersen/jetty-hazelcast. It performs a little better than the jdbc manager (since reading and writing to hazelcast is a bit faster than sql), but due to all the synchronized methods in AbstractSessionManager etc, the threads still get blocked when load is high.

To increase performance significantly, we're trying to remove all synchronized code - basicaylly rewriting the session management code from scratch.