Re: [mosquitto-dev] Multithreading

[Wiebe Cazemier <wiebe@xxxxxxxxxxxx>]

----- Original Message -----
> From: "Wiebe Cazemier" <wiebe@xxxxxxxxxxxx>
> To: "mosquitto-dev eclipse" <mosquitto-dev@xxxxxxxxxxx>
> Sent: Tuesday, 8 October, 2019 21:50:14
> Subject: [mosquitto-dev] Multithreading
>
> Hi,
> 
> After bench-marking some MQTT servers, it was clear that Mosquitto is one of the
> fastest, it's just single threaded. I decided to have a look at the code, and
> it seems to me multithreading could be implemented. With edge-triggered epoll,
> you can call epoll_wait() on several threads. One could do pthread_create() for
> each CPU core at [1] where it now calls 'while(run)' in mosquitto_main_loop(),
> to have a loop per CPU. This will process connections rapidly, without context
> switching. What manner of mutexing on the subscription hierarchy, message
> store, log file and such are necessary, or whether it should be converted to a
> producer/consumer model, I'm still looking into. And I'm just learning the
> intricacies of the source code first, of course.
> 
> I'm working towards a first (sloppy) proof of concept. I'm happy to continue my
> efforts, but like to just mention it, as to not reinvent the wheel or go down
> paths that have been tried. I know there are some obstructions, like the
> Windows build without epoll, and auth plugins would need to be thread-safe, but
> one thing at a time.
> 
> I do have one question already. There are some (dummy) mutexes present in
> 'struct mosquitto' already, used in packet_mosq.c, 'int packet__write(struct
> mosquitto *mosq)' for instance. I read on an old wiki [2] that even though not
> threaded, code was written with threading in mind, but I'm not following the
> logic. Mutexes here suggest multiple threads are working on the same
> connection?
> 
> Regards,
> 
> Wiebe
> 
> 
> [1]
> https://github.com/eclipse/mosquitto/blob/a9d1a9f11524d117804635f3e44869f69f294968/src/loop.c#L211
> [2] https://wiki.eclipse.org/Mosquitto/Multi-Thread

Hi again,

I wrote a concept using epoll. See [1] (and its commit messages). It's based on having one thread per CPU (so no thread per client, which doesn't scale), and a separate thread for listening to the listensockets, to accept connections. This is to avoid the thundering herd problem.

I applied some locking like the UTHash docs suggest, but the store and mosquitto_db are still globally manipulated, and sockets are now read/written from multiple threads. So, if you connect too many clients (I wrote a quick mqtt stress tester in C++/Qt for that), it crashes. This could be solved by a lot of mutexes, which will cause lock contention, or perhaps with shared queues, but that takes a refactor.

A question is how do we deal with different platforms? I have no Windows or Mac C experience, so I don't know if I can do this with Winsock or kqueue if needed. I haven't even looked into if I can make this work with just 'poll'. I did avoid EPOLLONESHOT, EPOLLET and EPOLLEXCLUSIVE, so I hope it's doable.

Or we use libevent; I've never used it, and I don't know if the Mosquitto code easily ports to it.

Before I continue with the work, I'd like to know if you're open to merging this in Mosquitto (when done), and whether I can get some help for the other platforms. 

Anyway, feedback is welcome.

Regards,

Wiebe



[1] https://github.com/halfgaar/mosquitto/commits/wiebe-threads-demo