[mosquitto-dev] mosquitto oom issues

[Mark Hayward <mahhayward@xxxxxxxxxx>]

Hi,
Our Mosquitto broker had some out of memory issues a few days back. I have found some errors which are attached. Any ideas what may have caused this?

Thanks MH
=============================
Broker server = CentOS release 6.9 (Final), 15GB RAM

/var/log/mosquito.log
1454674266: mosquitto version 1.4.7 (build date 2015-12-27 20:16:39+0000) starting
1454674266: Config loaded from /etc/mosquitto/mosquitto.conf.
1454674266: Opening ipv4 listen socket on port 1883.
1454674266: Opening ipv6 listen socket on port 1883.

Errors from /var/log/messages
Dec 30 00:31:10 broker kernel: mosquitto invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Dec 30 00:31:10 broker kernel: mosquitto cpuset=/ mems_allowed=0
Dec 30 00:31:10 broker kernel: Pid: 21815, comm: mosquitto Not tainted 2.6.32-358.11.1.el6.x86_64 #1
Dec 30 00:31:10 broker kernel: Call Trace:
Dec 30 00:31:10 broker kernel: [<ffffffff810cb541>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Dec 30 00:31:10 broker kernel: [<ffffffff8111cd40>] ? dump_header+0x90/0x1b0
Dec 30 00:31:10 broker kernel: [<ffffffff8121d24c>] ? security_real_capable_noaudit+0x3c/0x70
Dec 30 00:31:10 broker kernel: [<ffffffff8111d1c2>] ? oom_kill_process+0x82/0x2a0
Dec 30 00:31:10 broker kernel: [<ffffffff8111d101>] ? select_bad_process+0xe1/0x120
Dec 30 00:31:10 broker kernel: [<ffffffff8111d600>] ? out_of_memory+0x220/0x3c0
Dec 30 00:31:10 broker kernel: [<ffffffff8112c2ac>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Dec 30 00:31:10 broker kernel: [<ffffffff81160aaa>] ? alloc_pages_vma+0x9a/0x150
Dec 30 00:31:10 broker kernel: [<ffffffff81143d2b>] ? handle_pte_fault+0x76b/0xb50
Dec 30 00:31:10 broker kernel: [<ffffffff81434931>] ? sock_aio_read+0x1a1/0x1b0
Dec 30 00:31:10 broker kernel: [<ffffffff8114434a>] ? handle_mm_fault+0x23a/0x310
Dec 30 00:31:10 broker kernel: [<ffffffff810474c9>] ? __do_page_fault+0x139/0x480
Dec 30 00:31:10 broker kernel: [<ffffffff8151364e>] ? do_page_fault+0x3e/0xa0
Dec 30 00:31:10 broker kernel: [<ffffffff81510a05>] ? page_fault+0x25/0x30
Dec 30 00:31:10 broker kernel: Mem-Info:
Dec 30 00:31:10 broker kernel: Node 0 DMA per-cpu:
Dec 30 00:31:10 broker kernel: CPU 0: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: CPU 1: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: CPU 2: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: CPU 3: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: CPU 4: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: CPU 5: hi: 0, btch: 1 usd: 0
Dec 30 00:31:10 broker kernel: Node 0 DMA32 per-cpu:
Dec 30 00:31:10 broker kernel: CPU 0: hi: 186, btch: 31 usd: 58
Dec 30 00:31:10 broker kernel: CPU 1: hi: 186, btch: 31 usd: 35
Dec 30 00:31:10 broker kernel: CPU 2: hi: 186, btch: 31 usd: 21
Dec 30 00:31:10 broker kernel: CPU 3: hi: 186, btch: 31 usd: 32
Dec 30 00:31:10 broker kernel: CPU 4: hi: 186, btch: 31 usd: 53
Dec 30 00:31:10 broker kernel: CPU 5: hi: 186, btch: 31 usd: 47
Dec 30 00:31:10 broker kernel: Node 0 Normal per-cpu:
Dec 30 00:31:10 broker kernel: CPU 0: hi: 186, btch: 31 usd: 166
Dec 30 00:31:10 broker kernel: CPU 1: hi: 186, btch: 31 usd: 112
Dec 30 00:31:10 broker kernel: CPU 2: hi: 186, btch: 31 usd: 35
Dec 30 00:31:10 broker kernel: CPU 3: hi: 186, btch: 31 usd: 79
Dec 30 00:31:10 broker kernel: CPU 4: hi: 186, btch: 31 usd: 72
Dec 30 00:31:10 broker kernel: CPU 5: hi: 186, btch: 31 usd: 42
Dec 30 00:31:10 broker kernel: active_anon:3578393 inactive_anon:443476 isolated_anon:0
Dec 30 00:31:10 broker kernel: active_file:128 inactive_file:1014 isolated_file:0
Dec 30 00:31:10 broker kernel: unevictable:0 dirty:7 writeback:3 unstable:0
Dec 30 00:31:10 broker kernel: free:33347 slab_reclaimable:2459 slab_unreclaimable:10855
Dec 30 00:31:10 broker kernel: mapped:277 shmem:5 pagetables:12489 bounce:0
Dec 30 00:31:10 broker kernel: Node 0 DMA free:15636kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15220kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 30 00:31:10 broker kernel: lowmem_reserve[]: 0 3510 16120 16120
Dec 30 00:31:10 broker kernel: Node 0 DMA32 free:64944kB min:14704kB low:18380kB high:22056kB active_anon:2695584kB inactive_anon:570724kB active_file:192kB inactive_file:4304kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3595040kB mlocked:0kB dirty:8kB writeback:0kB mapped:1096kB shmem:0kB slab_reclaimable:136kB slab_unreclaimable:332kB kernel_stack:8kB pagetables:3640kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:384 all_unreclaimable? yes
Dec 30 00:31:10 broker kernel: lowmem_reserve[]: 0 0 12609 12609
Dec 30 00:31:10 broker kernel: Node 0 Normal free:52560kB min:52816kB low:66020kB high:79224kB active_anon:11618096kB inactive_anon:1203180kB active_file:348kB inactive_file:144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:12911840kB mlocked:0kB dirty:0kB writeback:4kB mapped:52kB shmem:20kB slab_reclaimable:9700kB slab_unreclaimable:43092kB kernel_stack:2360kB pagetables:46316kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:424 all_unreclaimable? yes
Dec 30 00:31:10 broker kernel: lowmem_reserve[]: 0 0 0 0
Dec 30 00:31:10 broker kernel: Node 0 DMA: 1*4kB 0*8kB 1*16kB 2*32kB 1*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15636kB
Dec 30 00:31:10 broker kernel: Node 0 DMA32: 192*4kB 120*8kB 67*16kB 38*32kB 14*64kB 5*128kB 2*256kB 25*512kB 35*1024kB 5*2048kB 0*4096kB = 64944kB
Dec 30 00:31:10 broker kernel: Node 0 Normal: 992*4kB 707*8kB 440*16kB 312*32kB 167*64kB 69*128kB 24*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 52312kB
Dec 30 00:31:10 broker kernel: 58389 total pagecache pages
Dec 30 00:31:10 broker kernel: 57147 pages in swap cache
Dec 30 00:31:10 broker kernel: Swap cache stats: add 2757137, delete 2699990, find 10981274/10994874
Dec 30 00:31:10 broker kernel: Free swap = 0kB
Dec 30 00:31:10 broker kernel: Total swap = 4194288kB
Dec 30 00:31:10 broker kernel: 4190192 pages RAM
Dec 30 00:31:10 broker kernel: 78284 pages reserved
Dec 30 00:31:10 broker kernel: 8051 pages shared
Dec 30 00:31:10 broker kernel: 4072434 pages non-shared