Hi Jetty community. We have an application (Jenkins) that was running on a Tomcat web server, with a couple of remote apps making thousands of REST api calls without issue.
We have switch to using Jetty but are having CLOSE_WAIT issues on the clients doing the REST api's.
I ran TCP/IP traces against Tomcat 7.0.52, Jetty 9.1.2, Jetty 9.1.3, and stand-alone Jenkins (with some version of embedded Jetty). Using a single 'wget' to the server I found that Tomcat and the stand alone Jenkins work without causing a CLOSE_WAIT on the wget socket. But the Jetty tests cause the wget socket to enter CLOSE_WAIT for a period of time. In the failing instances, it appears that the server side does not complete the close sequence and send a final ACK. See below traces. the 'adcd-006' or 'scm-021' are the server side hosts running the various combos (Tomcat, Jetty, or standalone Jenkins with embedded Jetty). I highlighted the last line showing the server sending final ACK in the
working case, and the client being the last to send on the socket in the failing cases. It also appears that in the working cases, the server is the side to send the first FIN, and on the failing cases, it is the client to send the first FIN.
Clearly Jetty is behavior different than Tomcat and even the older embedded Jetty in Jenkins. These test were performed on RedHat EL 6.4. It can be seen from client to do the wget, then
netstat-an | grep CLOSE_WAIT. In the working cases, no socket is held in CLOSE_WAIT. On the cases, you will find a CLOSE_WAIT to the target server.
Can anyone help eliminate the CLOSE_WAIT issue? I've used sysctl -a net.ipv4.tcp_tw_recycle = 1 with some success on one client, not so much on the other.
Any help, insight would be appreciated.
Daryl Spartz
**************************************************************************************************************************************************************
Working cases:
***** Jenkins/Tomcat
[dspartz@scm-021 ~]$ sudo tcpdump host scm-025.corp.gq1.yahoo.com
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet),
capture size 65535 bytes
***Start TCP/IP Connection setup
14:45:08.700656 IP scm-025.corp.gq1.yahoo.com.39936 > scm-021.corp.gq1.yahoo.com.distinct: Flags [S], seq 2771746985, win 14600, options [mss 1460,sackOK,TS val 3241642192 ecr 0,nop,wscale 7], length 0
14:45:08.700673 IP scm-021.corp.gq1.yahoo.com.distinct > scm-025.corp.gq1.yahoo.com.39936: Flags [S.], seq 4266270921, ack 2771746986, win 14480, options [mss 1460,sackOK,TS val 732014087 ecr 3241642192,nop,wscale 7], length 0
14:45:08.700757 IP scm-025.corp.gq1.yahoo.com.39936 > scm-021.corp.gq1.yahoo.com.distinct: Flags [.], ack 1, win 115, options [nop,nop,TS val 3241642192 ecr 732014087], length 0
***End TCP/IP Connection setup
14:45:08.700806 IP scm-025.corp.gq1.yahoo.com.39936 > scm-021.corp.gq1.yahoo.com.distinct: Flags [P.], seq 1:183, ack 1, win 115, options [nop,nop,TS val 3241642192 ecr 732014087], length 182
14:45:08.700813 IP scm-021.corp.gq1.yahoo.com.distinct > scm-025.corp.gq1.yahoo.com.39936: Flags [.], ack 183, win 122, options [nop,nop,TS val 732014087 ecr 3241642192], length 0
14:45:08.701515 IP scm-021.corp.gq1.yahoo.com.distinct > scm-025.corp.gq1.yahoo.com.39936: Flags [P.], seq 1:415, ack 183, win 122, options [nop,nop,TS val 732014088 ecr 3241642192], length 414
14:45:08.701579 IP scm-021.corp.gq1.yahoo.com.distinct > scm-025.corp.gq1.yahoo.com.39936: Flags [F.], seq 415, ack 183, win 122, options [nop,nop,TS val 732014088 ecr 3241642192], length 0
14:45:08.701603 IP
scm-025.corp.gq1.yahoo.com.39936 > scm-021.corp.gq1.yahoo.com.distinct: Flags [.], ack 415, win 123, options [nop,nop,TS val 3241642193 ecr 732014088], length 0
14:45:08.701863 IP scm-025.corp.gq1.yahoo.com.39936 > scm-021.corp.gq1.yahoo.com.distinct: Flags [F.], seq 183, ack 416, win 123, options [nop,nop,TS val 3241642193 ecr 732014088], length 0
14:45:08.701876 IP scm-021.corp.gq1.yahoo.com.distinct > scm-025.corp.gq1.yahoo.com.39936: Flags [.], ack 184, win 122, options [nop,nop,TS val 732014088 ecr 3241642193], length 0
***** jenkins.war (standalone, imbedded jetty, what version?)
[dspartz@adcd-006 ~]$ sudo tcpdump host scm-025.corp.gq1.yahoo.com
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:22:43.379833 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [S], seq 2641922706,
win 14600, options [mss 1460,sackOK,TS val 3337499944 ecr 0,nop,wscale 7], length 0
17:22:43.379865 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [S.], seq 713466498, ack 2641922707, win 14480, options [mss 1460,sackOK,TS val 2784499880 ecr 3337499944,nop,wscale 7], length 0
17:22:43.415042 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1, win 115, options [nop,nop,TS val 3337499979 ecr 2784499880], length 0
17:22:43.415065 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [P.], seq 1:149, ack 1, win 115, options [nop,nop,TS val 3337499979 ecr 2784499880], length 148
17:22:43.415075 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [.], ack 149, win 122,
options [nop,nop,TS val 2784499915 ecr 3337499979], length 0
17:22:43.485889 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [P.], seq 1:1014, ack 149, win 122, options [nop,nop,TS val 2784499986 ecr 3337499979], length 1013
17:22:43.486004 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [.], seq 1014:8254, ack 149, win 122, options [nop,nop,TS val 2784499986 ecr 3337499979], length 7240
17:22:43.520237 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1014, win 130, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.520256 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [P.], seq 8254:9387, ack 149, win 122, options [nop,nop,TS val
2784500020 ecr 3337500087], length 1133
17:22:43.520376 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 2462, win 153, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.520388 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 3910, win 176, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.520393 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 5358, win 198, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.520400 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 6806, win 221, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.520409 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 8254, win 244, options [nop,nop,TS val 3337500087 ecr 2784499986], length 0
17:22:43.554658 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 9387, win 266, options [nop,nop,TS val 3337500122 ecr 2784500020], length 0
17:22:43.701234 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [.], seq 9387:12283, ack 149, win 122, options [nop,nop,TS val 2784500201 ecr 3337500122], length 2896
17:22:43.701258 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [P.], seq 12283:13388, ack 149, win 122, options [nop,nop,TS val 2784500201 ecr 3337500122], length 1105
17:22:43.701457 IP
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [F.], seq 13388, ack 149, win 122, options [nop,nop,TS val 2784500201 ecr 3337500122], length 0
17:22:43.736525 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 10835, win 289, options [nop,nop,TS val 3337500307 ecr 2784500201], length 0
17:22:43.736553 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 12283, win 311, options [nop,nop,TS val 3337500307 ecr 2784500201], length 0
17:22:43.736560 IP scm-025.corp.gq1.yahoo.com.43820 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 13388, win 334, options [nop,nop,TS val 3337500307 ecr 2784500201], length 0
17:22:43.736574 IP scm-025.corp.gq1.yahoo.com.43820 >
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [F.], seq 149, ack 13389, win 334, options [nop,nop,TS val 3337500307 ecr 2784500201], length 0
17:22:43.736585 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.43820: Flags [.], ack 150, win 122, options [nop,nop,TS val 2784500237 ecr 3337500307], length 0
Failing
cases:
******** jetty 9.1.2/jenkins
[dspartz@adcd-006 ~]$ sudo tcpdump host scm-025.corp.gq1.yahoo.com
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:10:46.570170 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [S], seq 1293833853, win 14600, options [mss 1460,sackOK,TS val 3559980982 ecr 0,nop,wscale 7], length 0
08:10:46.570216 IP
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [S.], seq 915925482, ack 1293833854, win 14480, options [mss 1460,sackOK,TS val 3006932742 ecr 3559980982,nop,wscale 7], length 0
08:10:46.605591 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1, win 115, options [nop,nop,TS val 3559981018 ecr 3006932742], length 0
08:10:46.605626 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [P.], seq 1:157, ack 1, win 115, options [nop,nop,TS val 3559981018 ecr 3006932742], length 156
08:10:46.605638 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [.], ack 157, win 122, options [nop,nop,TS val 3006932778 ecr 3559981018], length 0
08:10:46.656553 IP
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [.], seq 1:7241, ack 157, win 122, options [nop,nop,TS val 3006932828 ecr 3559981018], length 7240
08:10:46.656682 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [.], seq 7241:13033, ack 157, win 122, options [nop,nop,TS val 3006932828 ecr 3559981018], length 5792
08:10:46.656710 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [P.], seq 13033:14034, ack 157, win 122, options [nop,nop,TS val 3006932828 ecr 3559981018], length 1001
08:10:46.692055 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1449, win 137, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692085 IP
scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 2897, win 160, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692093 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 4345, win 182, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692100 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 5793, win 205, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692106 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 7241, win 228, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692117 IP scm-025.corp.gq1.yahoo.com.49033 >
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 8689, win 250, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692184 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 10137, win 273, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692194 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 11585, win 296, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692200 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 13033, win 318, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692207 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 14034, win
341, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.692809 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [F.], seq 157, ack 14034, win 341, options [nop,nop,TS val 3559981105 ecr 3006932828], length 0
08:10:46.693133 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49033: Flags [F.], seq 14034, ack 158, win 122, options [nop,nop,TS val 3006932865 ecr 3559981105], length 0
08:10:46.728524 IP scm-025.corp.gq1.yahoo.com.49033 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 14035, win 341, options [nop,nop,TS val 3559981141 ecr 3006932865], length 0
****** jetty 9.1.3/jenkins
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:23:46.258402 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [S], seq 1749209732, win 14600, options [mss 1460,sackOK,TS val 3560760127 ecr 0,nop,wscale 7], length 0
08:23:46.258456 IP
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [S.], seq 3480511074, ack 1749209733, win 14480, options [mss 1460,sackOK,TS val 3007711149 ecr 3560760127,nop,wscale 7], length 0
08:23:46.293844 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1, win 115, options [nop,nop,TS val 3560760163 ecr 3007711149], length 0
08:23:46.293869 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [P.], seq 1:157, ack 1, win 115, options [nop,nop,TS val 3560760163 ecr 3007711149], length 156
08:23:46.293878 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [.], ack 157, win 122, options [nop,nop,TS val 3007711185 ecr 3560760163], length 0
08:23:46.589361 IP
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [.], seq 1:7241, ack 157, win 122, options [nop,nop,TS val 3007711480 ecr 3560760163], length 7240
08:23:46.589480 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [.], seq 7241:13033, ack 157, win 122, options [nop,nop,TS val 3007711480 ecr 3560760163], length 5792
08:23:46.589494 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [P.], seq 13033:14035, ack 157, win 122, options [nop,nop,TS val 3007711480 ecr 3560760163], length 1002
08:23:46.624935 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 1449, win 137, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.624962 IP
scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 2897, win 160, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.624972 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 4345, win 182, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.624985 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 5793, win 205, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.624991 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 7241, win 228, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625004 IP scm-025.corp.gq1.yahoo.com.49117 >
adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 8689, win 250, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625010 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 10137, win 273, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625016 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 11585, win 296, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625024 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 13033, win 318, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625064 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 14035, win
341, options [nop,nop,TS val 3560760498 ecr 3007711480], length 0
08:23:46.625605 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [F.], seq 157, ack 14035, win 341, options [nop,nop,TS val 3560760499 ecr 3007711480], length 0
08:23:46.626284 IP adcd-006.adx.pool.corp.ne1.yahoo.com.webcache > scm-025.corp.gq1.yahoo.com.49117: Flags [F.], seq 14035, ack 158, win 122, options [nop,nop,TS val 3007711517 ecr 3560760499], length 0
08:23:46.661626 IP scm-025.corp.gq1.yahoo.com.49117 > adcd-006.adx.pool.corp.ne1.yahoo.com.webcache: Flags [.], ack 14036, win 341, options [nop,nop,TS val 3560760535 ecr 3007711517], length 0
