| Re: [mosquitto-dev] Mosquitto increasing maximum connection limit |
Hi Jiji,
I’m not sure if I can attach a file in this mailing list, so I copy/paste inline. We did a lot of changes, mainly in Linux, to:
-
Increase
the number of potential concurrent connections.
- Decrease the use of CPU to take the maximum advantage of a single core.
Content of /etc/security/limits.conf
* soft nofile 999999
Content of /etc/rc.local
defrt=`ip route | grep "^default" | head -1`
ip route change $defrt initcwnd 10
ifconfig eth0 txqueuelen 2000
ifconfig eth1 txqueuelen 2000
Content of /etc/pam.d/sshd
@include common-authaccount required pam_nologin.so
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
session required pam_loginuid.so
@include common-session
session optional pam_motd.so motd=/run/motd.dynamic noupdate
session optional pam_motd.so
session optional pam_mail.so standard noenv
session required pam_limits.so
session required pam_env.so
session required pam_env.so user_readenv=1 envfile=/etc/default/locale
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
@include common-password
Content of /etc/ssh/sshd_config
Port 22Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
UsePrivilegeSeparation yes
KeyRegenerationInterval 3600
ServerKeyBits 768
SyslogFacility AUTH
LogLevel INFO
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
X11Forwarding no
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
UsePAM yes
Content of /etc/sysctl.conf
fs.file-max = 99999999vm.swappiness = 10
vm.min_free_kbytes = 65536
net.ipv4.ip_local_port_range = 1024 65535
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 90000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
net.ipv4.conf.all.log_martians = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_sack = 0
Content of /etc/init/mosquito.conf
description "Mosquitto MQTTv3.1 broker"author "Roger Light <roger@xxxxxxxxxx"
limit nofile 999999 999999
start on net-device-up
respawn
exec /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
Relevant parameters configuration in /etc/mosquito/mosquito.conf
allow_duplicate_messages true
autosave_interval 1800
autosave_on_changes false
connection_messages true
log_dest syslog
log_timestamp true
log_type all
max_inflight_messages 1
max_queued_messages 1000
message_size_limit 10240
persistence true
persistence_file mosquitto.db
retry_interval 5
store_clean_interval 10
sys_interval 15
listener 8883
tls_version tlsv1
We was testing Mosquitto + Java clients + Nodejs clients + TLS + mysql-auth-plug + MySQL, using a physical server to host the mosquito and several (8-10, don’t remember) clients. And everything in our LAN (we tested AWS and others, but not for 100K connections).
We needed more than one VM because of ephemeral ports (one VM = Max 64K connections to the broker) and because virtual machines we configured were not too large.
We wanted to try whether we could support one million users that are not connected concurrently. And we do not know the number of users that would be connected simultaneously if we have a million users. So… we started to try to take advantage of Mosquitto as much as possible. Just for fun!
With the configuration I’ve copied/pasted in this email we achieved more than 100K connections (as a peak, sometimes more, sometimes less). Let’s say… Mosquitto is able to establish the connection. However from 50K connection (more or less) we started to have lots of timeouts because of keepalive messages did not reach the clients on time.
In our experience, the problem is not the memory usage. You can buy several GB of RAM at a very low price now. The memory of our broker remained under 10GB (of 48 GB). Also, 23 of our 24 Xeon cores were underused while one was 100% used after a few dozen thousands connections. I could not tell you the specific numbers, sorry. So, we think that the main problem is that Mosquitto is single threaded and not able to take advantage of multi-core/multi-processor systems. Our cores run at 2.5GHz and, definitely are not enough to support more than 25-30K connection without having disconnections. Mosquito does not scale horizontally too well.
Jiji, you can view old mails at http://dev.eclipse.org/mhonarc/lists/mosquitto-dev/.
Hope this information is helpful for you.
Regards.
Hi Manuel,I want to know more information about it. The points behind our calculations are these :1. The capacity of Mosquitto broker is depending on the underlying OS and which in turn depends on what is maximum number of files (physical files or Network Sockets ) that OS can open concurrently. The maximum number of files that can be opened depends on what is the memory available for each process.2. In linux the default value of max number of file open is 1024 and MQTT broker takes this value as its default. In order to set the max value beyond the 1024 , we need to change the linux settings for increasing the upper file limit.3. The maximum value we can set on process level depending on how much memory is available on system for each process.4. For each keep alive request memory conception is 16 bytes. Hence to keep 10k concurrent connection memory requirement is 160 MB and for 100k 1.6 GBNow another important points are1. we are opening the connection only based on the user activity. If user got notification, we are closing the connection.2. So connection will not be open always.3. Based on our understandings we can make more scalablility by introducing bridging of Mosquitto brokersPlease give me the answer for following questions :1. At what point you couldn't able to support concurrent connections ? (Maximum limit of connection based on a specific configurations). ?2. Is it correct that you are trying to support more than 100k connection using one mosquitto broker with the mentioned configuration ?3. If possible please provide or tell me how will i get old archives of your mail, since i joined to the group recently.Thanks & RegardsJijiOn Tue, Dec 13, 2016 at 3:29 AM, Manuel Domínguez Dorado <manolodd@xxxxxxxxx> wrote:Jiji,I achieved this number of concurrent connections but everything goes wrong when they started to send messages. So, in practice, it is not feasible this number of connections using Mosquitto.I started as usual and after that I continued tuning linux kernel, reducing the number of bits used in TLS certificates, increasing the keepalive time... look for my messages in the mailist archives if you need more info.The main problem was that Mosquitto uses only one core that become exhausted. I've 48 GB RAM and 24 Xeon cores. Everything ok but one core (the one running Mosquitto, reached 100% quickly. Nothing to do :-(Regards.PS: Karl, again, no I did not documented the overall process in an empirical way. He, heEl 12/12/2016 22:50, "Roger Light" <roger@xxxxxxxxxx> escribió:Hi Jiji,
> 1. I am using service mosquitto restart , is this upstart command. what is
> this limit nofile 4096 4096 ?
If you install mosquitto from the ppa provided by the project, or
through the normal ubuntu repository you will be using upstart. The
line I gave you will need to be added to /etc/init/mosquitto.conf to
allow mosquitto to use more than 1024 files - you should change the
4096 to the value of your choice.
> 2. Why you'r saying we can't able to establish a lakh o connection in Ubuntu
> 14.04 medium server with 4 GB memory. Please give me more details.
I'm saying that based on the experience of other people on this list
it will not be possible.
Regards,
Roger
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev