[linuxtools-dev] Modeled state in TMF/LTTng and pattern matching

[Michel Dagenais <michel.dagenais@xxxxxxxxxx>]

The recent discussion on the modeled state was extremely interesting and greatly benefited from the experience of people with different backgrounds. Thanks! This brings follow-up questions on other parts of a trace analysis system which interact or add to a modeled state system.

Among the features being targeted or already prototyped for trace analysis, quite a few access the modeled state or use Finite State Machines and thus define (implicitly) additional state information:

- Automated Fault Identification: the idea is to search for patterns of problematic behavior to raise alerts. It could be used for detecting programming errors or operational problems (performance bottleneck, host based intrusion detection...).

- Ad hoc filtering or querying of events.

- Trace Abstraction: replace patterns of low level events by higher level synthetic events. For example, the many events related to sequentially reading a file F could, for a high level view, be replaced by a synthetic event "sequentially reading file" F.

In all three cases, a pattern language could be useful, additional state information may be required during the pattern matching, and it may be useful to keep the links between the state transitions and the events causing them; the links may be useful to follow dependencies or to understand/debug the patterns.

It looks like it could be interesting to have a single notion of state used to model both the directly modeled state (explicitly required) as well as the state required for pattern matching in these different applications. This way, common code and tools may be used for storing, navigating, displaying and debugging the state.

Here are a few examples:

1) We want to check if there are attempts to write to a closed file descriptor. If we keep the list of opened file descriptors in the modeled state, this is equivalent to searching for write events where the file descriptor is absent from the file descriptor table in the modeled state. It can thus be expressed as a simple condition on the event type, event fields and modeled state.

2) We are looking for locking problems, a lock sometimes acquired in IRQ context and sometimes outside of IRQ context. If we add to the modeled state the locks and a flag telling if each lock has been accessed in IRQ and another for out of IRQ access, again it becomes a simple condition: lock acquire event type, event field telling which lock, state for that lock telling that is was acquired in IRQ context, and process context outside of IRQ.

3) When abstracting multiple read events into a "sequentially reading a file" event, we may want to keep track in the state of the previous read position and if all previous read operations were sequential. Then, upon closing the file, if the state tells that all read operations were sequential and the last read operation was the file end, then a "sequentially reading a file" abstract synthetic event could be created.

4) In an ad hoc query, one could be interested in finding if the system load ever got over a certain threshold. The system load could be added to the modeled state, in which case it becomes a simple condition to check on the modeled state.

Is a common state and associated common tools a good idea in this context of trace analysis tools? Simple applications (ad hoc queries, intrusion detection systems rules, scripting) typically use patterns (with implicit state in the underlying finite state machines) while larger applications may go for a more programming like model (explicit state variables, conditions for state changes, debugging tool). Do you think that common trace analysis tasks would be simpler or more difficult with such an explicit model for state variables?