Community
Participate
Working Groups
The live streaming support (reading trace when being streamed) is rudimentary. This is an umbrella bug for improvments of the live streaming support. Here is some items that need to be looked at: - It only supports local connection to lttng-relayd. - Trace events are read from the trace file on disk written by lttng-relayd is read. It might be beneficial for certain use case to get the events directly from the lttng-relayd. - Performance is slow - In Control view, no way to specify destination in tracing project. - During import and opening, no check for name collision of the trace name - Not possible yet to trace per PID for UST - Not possible to live view kernel and UST at the same time - XML Analysis views are not supported
(In reply to Bernd Hufmann from comment #0) > - Not possible to live view kernel and UST at the same time There was work done on this one but it depends on https://bugs.lttng.org/issues/853
> - Trace events are read from the trace file on disk written by lttng-relayd > is read. It might be beneficial for certain use case to get the events > directly from the lttng-relayd. No. Trace data should *always* be fetched from lttng-relayd. On a live session Tracecompass should not even be aware of trace files locally or remotely.
(In reply to Jonathan Rajotte Julien from comment #2) > > - Trace events are read from the trace file on disk written by lttng-relayd > > is read. It might be beneficial for certain use case to get the events > > directly from the lttng-relayd. > > No. Trace data should *always* be fetched from lttng-relayd. On a live > session Tracecompass should not even be aware of trace files locally or > remotely. This brings up an interesting point. Unlike a live analysis (à la LTTng-Analyses), Trace Compass needs to be able to seek back in a trace, since the user can seek at different locations in the trace, and we cannot usually keep the whole trace in memory. In this case, should it be Trace Compass's responsibility to write the CTF trace to disk? Or should they be fetched from the relayd every time?
After some discussions with the relayd maintainers, it's been confirmed that the relayd does support sending past events of the trace, as long as you know which stream/offset you are looking for. So it seems the best approach on the Trace Compass side would be to implement a new "CTF live" trace type, which does not read from files, but only communicates with a relay daemon. This new trace type could: - Upon connection, ask for all past trace events (or only the ongoing ones, maybe make it configurable?) - Saves the CTF index that are received locally to disk. - If an event request for past events is made, when the user scrolls up the event table for example, then a request for event is sent to the relayd, using the appropriate indexes. Another option could be to write the whole trace locally to disk, and seek back using this. However this would end up storing the complete trace in two locations. Since the relayd supports "seeking back", it would make sense to make use of it.
Thanks to everybody for your input to the discussions. Before we go into the implementation details, I would like to open a discussion about the use cases that the live trace reading feature in Trace Compass should support. In the current implementation it was assumed that all the relevant analysis available for a kernel trace or UST trace respectively are executed so that the user can analyse the trace while new data arrives. Some analysis are run when opening a the trace (e.g. kernel analysis), other ones are executed when certain views are opened (e.g. CPU usage). However, live mode is about monitoring certain key performance indicatior, events or conditions of an application or node, more about high-level information than low-level events (e.g. sched_switch). It is supposed to give users an overview on what is happening in the application. It might even trigger some action (e.g. notfiy user, enable more trace events) when a condition happens. Such key indicators can be for example cpu load, latencies/durations, memory consumption, missed deadlines, error events from application. The user will look at these indicators and when certain conditions occur the user then will dig deeper by analysizing the trace at the relevant timestamp. I see it as a type of report that is generated while an application is executed. How can this be achieved? When new events arrive certain filter or XML pattern matcher could be applied that will generate such reports. Another point I'd like to raise. When running analysis like the kernel analysis, supplementary files (state system) are generated. In live mode one might think that it's intersting to generate them when new events arrive. However, live trace reading is often used to monitor an application over several hours and creating large data sets on disk where Trace Compass runs won't scale. It think it's better to create reports and then use the original trace to dig deeper at the relevant timestamps when needed. The planned trace cutting feature will be very handy for that. Any thoughts?
