Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mosquitto-dev] client drops

If you have 3800 clients connected without doing serious kernel
trickery, you're way in the deep end of poll() breakdown range.
something like htop would probably show that you're cpu time is
in "system", not in "user" land. I've not done any in depth
testing for a few years now. Here's where I broke it all down
back in 2013
https://lists.launchpad.net/mosquitto-users/msg00335.html

There's more out there now, but I believe that at 3800, you're
just about at the end.

Cheers,
Karl P

John Harrison <john@xxxxxxxxxxx> wrote:
> Hi Karl. Thanks for the questions.
> 
> The persistence write interval, as I understand it, is the same
> as the autosave interval which defaults to 30 minutes. The
> surges of 99.9% CPU happen for a few seconds every 10 seconds
> or so.
> 
> Here's some data that I believe answers your other questions:
> 
> $SYS/broker/clients/connected: 3827
> $SYS/broker/clients/expired: 0
> $SYS/broker/clients/disconnected: -1
> $SYS/broker/clients/total: 3825
> $SYS/broker/load/connections/+: 73.78, 68.74, 67.17
> $SYS/broker/load/bytes/received/+: 20925.18, 22602.58, 22880.39
> $SYS/broker/load/bytes/sent/+: 141812.25, 151984.54, 160541.45
> $SYS/broker/load/messages/sent/+': 2883.43, 3147.90, 3278.97
> $SYS/broker/load/messages/received/+: 1621.39, 1658.80, 1668.28
> $SYS/broker/load/publish/received/+: 120.27, 136.55, 142.08
> $SYS/broker/load/publish/sent/+: 1498.97, 1680.24, 1824.82
> $SYS/broker/messages/received: 351311395
> $SYS/broker/messages/sent: 591194360
> $SYS/broker/messages/stored: 2591
> $SYS/broker/publish/messages/received: 23615914
> $SYS/broker/publish/messages/sent: 280424529
> 
> John
> 
> On Thu, Jun 1, 2017 at 5:32 AM, Karl Palsson
> <karlp@xxxxxxxxxxxx> wrote:
> 
> >
> > John Harrison <john@xxxxxxxxxxx> wrote:
> > > Hello,
> > >
> > > We have found as the number of clients has increased on our
> > > Mosquitto production server, more and more clients are getting
> > > dropped. We don't use any plugins, authentication is by client
> > > prefix, and we are using SSL/TLS in our connections to the
> > > Mosquitto broker.
> > >
> > > To try to understand the problem better I parsed through a
> > > day's worth of the verbose log of the broker (size 1 Gig),
> > > isolating the lines with errors on them, removing the
> > > timestamps, and finally finding duplicates and sorting. The
> > > first 3 lines (the most common lines appearing in the logs) are
> > > below. The first column shows the number of times that line
> > > appeared in the 24 hour log:
> > >
> > > >    3498 OpenSSL Error: error:1408F119:SSL
> > > > routines:SSL3_GET_RECORD:decryption failed or bad record mac
> > > >    1616 Socket error on client <unknown>, disconnecting.
> > > >    1197 OpenSSL Error: error:140F3042:SSL
> > > > routines:SSL_UNDEFINED_CONST_FUNCTION:called a function you should
> > not call
> > >
> > >
> > > Mosquitto seems to average 20% CPU but the hills and valleys
> > > are huge: it seems to spend 4/5 of its time at almost 0% CPU
> > > then peaks at 100% CPU for the other 1/5 time. Looking at
> > > tcpdump it seems the messages are coming in scattered, not all
> > > at once, so I'm not really understanding these hills and
> > > valleys and don't know if it is related . There's nothing in
> > > the syslog for clues on what is going on.
> >
> > Are those spikes corresponding to your persistence write
> > interval? Do you have lots of offline queued clients, or large
> > amounts of retained messages? How many simultaneous clients?
> >
> > Cheers,
> > Karl P
> >
> > >
> > > I found this:
> > > https://github.com/eclipse/mosquitto/issues/383
> > >
> > > which makes me wonder if the problem might be at least
> > > partially solved by using haproxy for my SSL termination? (I'm
> > > thinking haproxy would work as well as Nginx and is something
> > > we were looking at doing for a future Mosquitto cluster.)
> > >
> > > Any clues, help, thoughts would be appreciated.
> > >
> > > John

Attachment: signature.html
Description: OpenPGP Digital Signature


Back to the top