Hello,
We have found as the number of clients has increased on our Mosquitto production server, more and more clients are getting dropped. We don't use any plugins, authentication is by client prefix, and we are using SSL/TLS in our connections to the Mosquitto broker.
To try to understand the problem better I parsed through a day's worth of the verbose log of the broker (size 1 Gig), isolating the lines with errors on them, removing the timestamps, and finally finding duplicates and sorting. The first 3 lines (the most common lines appearing in the logs) are below. The first column shows the number of times that line appeared in the 24 hour log:
3498 OpenSSL Error: error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
1616 Socket error on client <unknown>, disconnecting.
1197 OpenSSL Error: error:140F3042:SSL routines:SSL_UNDEFINED_CONST_FUNCTION:called a function you should not call
Mosquitto seems to average 20% CPU but the hills and valleys are huge: it seems to spend 4/5 of its time at almost 0% CPU then peaks at 100% CPU for the other 1/5 time. Looking at tcpdump it seems the messages are coming in scattered, not all at once, so I'm not really understanding these hills and valleys and don't know if it is related . There's nothing in the syslog for clues on what is going on.
I found this:
which makes me wonder if the problem might be at least partially solved by using haproxy for my SSL termination? (I'm thinking haproxy would work as well as Nginx and is something we were looking at doing for a future Mosquitto cluster.)
Any clues, help, thoughts would be appreciated.
John
