Our current design for internal data management is largely tainted by constraints of the deprecated AWTDA2 protocol. This mail explores ways to optimize and simplify the
:pushdata() API.
The current situation is that internally, we have SDB staging tables, in which data is pushed from assets to the agent, where it's consolidated and sent to the server according to policy settings. There are smart optimizations to encode historical series of timestamped data, but less efforts have been made to efficiently encode "simple" data trees, where there's only one value per path.
- The SDB table API allows fine-grained control of the process, for users who want to perform automated local consolidations, and/or control the encoding, e.g. by trading precision for bandwidth usage.
- The :pushdata() API, conversely, just accepts unstructured data, and makes the best of it by transparently declaring the tables needed to buffer them in the agent.
:pushdata() is simple to use, but often leads to ineffective encodings, partly because it makes the unwarranted assumption that most data are sent by large series to the server, partly because it retained some legacy, AWTDA2 related limitations.
*If* we accept the following assumptions:
- most data sent through :pushdata() are instantaneous single data points or data trees
-
developers using it don't care *yet* about finely optimizing bandwidth usage. If and when it becomes necessary for them, it's easy for them to move on to SDB tables API.
Then a much more simple/predictable/maintainable internal mechanism for :pushdata() becomes possible. The agent could simply serialize series of key/value records, as sent by assets to the agent, sort them by policy, and flush them whenever the policy is triggered. Whenever better performances are needed, switching to tables is easy, the only price to pay is that those tables' structure must be predeclared through racon:newtable(config).
Some work might be done, sooner or later, to optimize the sending of trees such as config queries from the server. The main optimization would be sharing common path prefixes; it could be done either through heuristics, or by using an offline int <-> path string dictionary, as optionally supported by M3DA. By making no firm promise about how hard :pushdata() tries to optimize the data it sends, we make it mor protocol agnostic, which might prove valuable if we later decide to support more protocols.
Another option would be to drop
:pushdata() altogether, and to delegate "simple data sending" to the device tree (see my previous mail).
-
pro: one less API to learn, document and maintain.
- cons:
- device tree writings don't naturally support the notion of sending policy. Everything pushed through the simple API would have to use the "default" policy, which might be an excessive simplification;
- translating device tree writings into SDB table operations might be less straightforward than pushdata -> SDB table translations (to be discussed).
--
Fabien Fleutot
