Re: [linuxtools-dev] [TMF] Advanced statistics view

[Alexandre Montplaisir <alexmonthy@xxxxxxxxxxxx>]

Hi François,

Very interesting! Some more comments below

On 13-02-26 04:07 PM, François Rajotte wrote:
> [...]
>
> By using the kernel state system, we have effectively halved the amount of
> information we have to store in the statistics state system.
> By doing so, we have also heavily coupled the kernel state system and the
> statistics state system. If a new event is added to the kernel state
> provider, the statistics provider must be aware of that change if the
> statistics must remain coherent. At the moment, this is not too problematic
> as only the sched_switch event provokes the state change used by the
> statistics.
> Still, the compromise between file size and the ease of maintenance is an
> interesting problem.

It's worth re-using what we already have. This new separate state system
could depend on the existing one, it shouldn't be a problem. The way to
find out when processes go in and out of CPUs shouldn't change (or if it
does change, it would be to fix/refine something, in which case the
separate statistics would also make use of).

>
> Here are some history tree file sizes I measured with the statistics state
> system using the current running time (without optimization)
> Trace 1 : 286 MB
> CPU usage stats history : 53 MB
> event count stats history: 572 MB
> kernel state history : 578 MB

Ouch, yeah the base kernel state system is quite large... I was getting
1.5x the size of the trace on average, 2x is getting quite big... There
might be optimizations to do there.

The "event count" statistics will be able to move to a partial history,
the should help a lot. Interesting to see that your CPU stats history
doesn't take much space already.

>
> Trace 2 : 10.6 MB
> CPU usage stats history : 2.24 MB
> event count stats history: 17.6 MB
> kernel state history : 12.2 MB
>
> Another minor problem is the storage of the cumulative time. The current
> history implementation only supports 32-bit integers and strings.
> Cumulative time can easily overflow an integer in long enough traces.
> I have not tried implementing 64-bit integer support. But according to my
> experience with the history tree format, this should not be very hard to
> do. In my implementation, overflows will occur without warning.

It would be very possible - dare I say "easy" - to add support for Long
state values. So that way you can save timestamps and cumulative times
as longs with no hassle. (String conversions, yuck!)

>
> The second part I worked on was the integration of this new information to
> the existing statistics view.
> Unfortunately, the current implementation of the statistics is very
> specific to the showing of event counts.
>
> I still wanted to have meaningful information displayed, so I worked around
> the limitations.
> Only longs can be displayed in the view, so instead of cpu usage shown in
> percentage, I have elected to display the raw count in nanoseconds.
> Also, there was no way to specify an organization for the statistics. The
> flat architecture of the event counts was pretty much hard-coded.
> Anyway, I feel like this part would require some work before it is
> integrated seamlessly in the current view.

Hmm yes, the current statistics view was all based around event counts.
If it's too specific it might be worth splitting away a EventCountViewer
for the existing statistics, and a new TimeStatsViewer or something,
that handles cumulative times and that kind of stuff (and would only be
available for trace types that decide to use it, like LTTng kernel
traces in our case).

>
> The code can be found here :
> http://git.dorsal.polymtl.ca/~frajotte?p=org.eclipse.linuxtools.git;a=shortlog;h=refs/heads/cpustats
>
> The lightweight version that uses the kernel state system can be found here:
> http://git.dorsal.polymtl.ca/~frajotte?p=org.eclipse.linuxtools.git;a=shortlog;h=refs/heads/cpustatslight
>
> Please keep in mind that this was mostly done as a proof of concept to show
> that it was possible. There are some hard-coded strings that should be
> refactored and no unit tests.
> I would love to have your feedback. I can also commit this on gerrit if you
> would prefer to review this contribution there.
>

Yeah once it's in shape for review it'll be easier on Gerrit. But it's
fine to discuss the design and high-level concepts first.


Cheers,
Alex