Re: [mosquitto-dev] Cassandra as a retained message store

["Laing, Michael" <michael.laing@xxxxxxxxxxx>]

OK. It looks promising so far. The difficulty is scaling the levels.

I can now handle 5 levels and am working on getting it to 8 over the next few days, which is enough to begin trying our use cases. Could go to more.

The nice thing is that, up to 4 levels, the 'start of query response' from Cassandra to the client is flat, regardless of query complexity - under 200 microseconds. And the messages are sequential from memory/SSD, so quick to send.

Increasing to 8 levels, filtering work is required of Cassandra, but it will still begin delivery in under a millisecond.

Michael

On Wed, Nov 26, 2014 at 11:52 AM, Ian Skerrett <ian.skerrett@xxxxxxxxxxx> wrote:

Michael,

I would definitely be interested in hearing updates from you, especially the solution around query. I think your use case is what a lot of other people will have too so it would be good to share experiences. It will also be interesting to see the different approaches people take to the query of MQTT messages.

 

Ian

 

 

 

 

From: mosquitto-dev-bounces@xxxxxxxxxxx [mailto:mosquitto-dev-bounces@xxxxxxxxxxx] On Behalf Of Laing, Michael
Sent: November-26-14 10:24 AM
To: mosquitto-dev@xxxxxxxxxxx
Subject: [mosquitto-dev] Cassandra as a retained message store

 

I am experimenting with MQTT as a way to extend the nytfabrik at the edge, complementing and perhaps replacing our custom WebSocket/SockJS protocol.

 

We use RabbitMQ with topic routing on the nytfabrik global backbone mesh, so we are familiar with wildcarding, subscription, etc.

 

And we have lots of 'retained' messages - 10's of millions growing to 100's of millions.

 

This is no problem for Cassandra but, as we look to support MQTT, we need a technical solution for query with wildcards on retained messages.

 

Cassandra has a limited query language but can be very fast. I was looking for a way to run 'close to the metal' (no additional layers) and now have a working prototype architecture and algorithms for store and query that are simple, fast and which scale. So I think that technical blocker is overcome for our usage cases.

 

Anyway, if this is of interest, I'll keep this list informed, and maybe solicit some advice.

 

We're not sure yet how we'll add MQTT to the nytfabrik, but most likely we would remix mosquitto with our own code, package it up, and run it under python on our autoscaling edge instances.

 

Cheers,

 

Michael

_______________________________________________
mosquitto-dev mailing list
mosquitto-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/mosquitto-dev