| Re: [paho-dev] Offline buffering in Python3 Paho MQTT client |
Lars Noodén <lars.nooden@xxxxxxxxx> writes:
> On 10/7/20 4:02 PM, Greg Troxel wrote:
>> You say "buffering", but once you talk about offline then this veers
>> into database sync. I think it's an important use case.
>> ...
>
> Thanks. I am unsure of the right terminology but buffering seems to be
> closest concept. Some systems are without networking for minutes or
> hours at a time yet still able to measure and have plenty of RAM in that
> context. This data is still useful to have even when it comes in very
> late.
I didn't mean to give you a hard time about terminology so much as raise
the database sync concept.
I completely agree about gathering data separately from connectivity and
having it arrive eventually.
> For this project, several systems feed timestamped temperature
> measurements into a broker. A database saves those incoming messages
> whenever they happen to arrive. Separately, another script does a
> database query for GNUplot for visualizations. Yet another script
> subscribes to MQTT and updates an e-paper display with the latest
> measurements. On the e-paper display different colors mark the
> staleness of the data, if it is not obvious from a discrepancy between
> the current time and the timestamp it displays. The e-paper does not
> benefit from the held MQTT messages, but the GNUplot rendering and other
> after-the-fact analysis can still use the data.
That sounds totally sensible, and what I was trying to get at and didn't
explain well is that the case of MQTT messages with timesamps and
messages without timestamps are very different. I would posit that MQTT
messages without timestamps and buffering are not really a good
combination.
>> Do you think this sort of offline/retransmission schemes need to
>> preserve ordering?
>
> Ordering would be useful but not necessary, at least for me, at least at
> the moment. However, I don't count on it. Due to device and network
> latency, which can be several seconds, even when all the devices are
> online at the same time, I have to use timestamps in the MQTT messages.
Once you have origin timestamps, then I don't think you need to care
about network ordering. The display needs to show "data from most
recent timestamp of all received messages", rather than "data from most
recent message".
> Anyway, I tried adding QoS when publishing,
>
> mqttclient.publish("temperatures", qos=2, payload=msg)
>
> but that does not seem to send a backlog of data after the longer
> outages finish.
I dimly remember a fixed smallish limit (100) of messages that can be
queued.
This is why I was suggesting on-disk or whatever persistence, sort of
like /var/spool/mqueue.
At least for me, network issues are correlated with power issues. I
realize there are devices without storage that are useful as sensors,
but it would be nice if those that did have storage could make use of
it.
I wonder if the right thing is to have a datbase abstraction usable by
the client, with one option being an in-RAM database.
Another option is a wrapper that does this and calls the existing
library, expecting error returns or acks for messages. The wrapper
could have the same interface, or better yet inherit from the same
public API so it can just be used instead.
Attachment:
signature.asc
Description: PGP signature