Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jetty-dev] Jetty 9.3.15 stopped accepting incoming connections (too many open files?)

The line ...

WARN |       o.e.j.h.HttpParser||bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@6d081ff2

Smells like a connection using normal HTTP was made to a connector on your server specified as HTTPS.

Would suggest you also check the netstat output.

On Jan 13, 2017 7:10 AM, "Michele Rossi" <michele.rossi@xxxxxxxxx> wrote:
hi all,

our production system running Jetty 9.3.15 this morning stopped accepting incoming connection producing the following suspicious log lines:


10:26:52.121|WARN |        o.e.j.s.HttpInput||
java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
        at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
        at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[ ........................... removed application log lines ......................................]

10:43:30.148|WARN |       o.e.j.h.HttpParser||Illegal character 0x16 in state=START for buffer HeapByteBuffer@23931b7a[p=1,l=265,c=8192,r=264]={\x16<<<\x03\x01\x01\x04\x01\x00\x01\x00\x03\x03Xx\xA1B\xF3h\x9a...\x03\x02\x03\x03\x02\x01\x02\x02\x02\x03\x00\x0f\x00\x01\x01>>>,application/xml;...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
10:43:30.148|WARN |       o.e.j.h.HttpParser||bad HTTP parsed: 400 Illegal character 0x16 for HttpChannelOverHttp@6d081ff2{r=0,c=false,a=IDLE,uri=null}
10:43:30.974|WARN |o.e.j.s.AbstractConnector||
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:373)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)



We have only just recently upgraded the system to Jetty 9.3.15 and with the previous build we never saw anything like this happening.

And the load on the system wasn't high at all.

I am letting you guys know in case some of you is aware of changes made that could cause this kind a problem.

A simple bounce of the system resolved the problem.


I am now considering a roll-back to the previous Jetty release while I try to gather more data on what happened.


If the problem re-occurs I am planning to attempt to diagnose it by dumping the threads and also by running unix's "lsof"on the jetty process to determine the number of open files.

thanks,
Michele




_______________________________________________
jetty-dev mailing list
jetty-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jetty-dev


Back to the top