Re: [linuxtools-dev] TMF and pluggable state providers

[Alexandre Montplaisir <alexmonthy@xxxxxxxxxxxx>]

On 13-03-15 12:56 PM, Aaron Spear wrote:
> [...]
> The problem as I see it is the requirement to derive a new type of trace in order to get state.  I want to have a facility that can create state for *any* trace knowing the event schema, but irrespective of the trace format.  So, I wanted to simply use the CustomTxtTrace for instance without having to derive and then override TmfTrace.buildStateSystem.  This is why I added an extension point for plugins that can provide instances of "IStateSystemFactory", an object that can create state systems given an ITmfTrace instance.  So then you can have a plugin that provides state systems in whatever sort of way you wish.  I then additionally created a data driven way to parse state from any trace.
>
> Here is the changed code in TmfTrace.buildStateSystem, perhaps this will clarify what I am doing:
>
> [...]

We already have some sort of factory for state systems, in
StateSystemManager.java (it's been rewritten a couple times, and it's
definitely more of a factory now, we should rename it...)

When a user wants to create is own "state definitions", what they are
effectively building is the IStateChangeInput. The trace type should
already be a type supported by TMF (like a CustomTextTrace, a Generic
CTF Trace, etc.) The way I was envisioning it (but I could be all wrong,
of course), was to use the existing trace type, and then to use the
user's custom StateChangeInput to build a state system manually, and
then add it to that trace. We would just need to expose a way to .put()
stuff in the fStateSystems map.

Extension points are kind of a pain to implement and maintain. If we
need to add one, so be it, let's do so. But I'm just trying to see if we
could avoid it.

>
>>> 2) Using the infrastructure above, I built a framework that does data
>>> driven state extraction. The idea is that you have some metadata that
>>> describes your hierarchy, event types that correspond to context and state
>>> changes, and rules for how to extract the new state and context from the
>>> events. It uses the TMF abstraction for events, and so it works
>>> independent of how the underlying trace is structured which is nice.
>>>
>>> 3) I have an implementation of the data driven state extraction above that
>>> right now is populated using a simple convention whereby the factory looks
>>> for an XML file that describes the state data right next to the trace. The
>>> right way to do this is to use some nice GUI to either create rules or
>>> manually choose which state types you want to use for a given trace, but
>>> it works for now. Attached is the XML file used to create the VM trace
>>> above so you have some idea of how this works. Again, there are layers to
>>> this. In the case of the java function trace I actually have a derived
>>> trace type, so for that one the XML file can be in the plugin (or you can
>>> hard code it and not bother with the XML)
>> Awesome. State-definitions in XML or similar has been a hot topic
>> recently. In the base TMF framework we could have a special state
>> provider (implementing IStateChangeInput) that would read such a file
>> (and eventually, the custom contents that was defined in a GUI), that
>> could then be used to populate custom views.
> This is exactly what I did.  I created a generic "DataDrivenStateChangeInput" class that has data structures with mappings from event types to fields and such to extract to figure out state.  The state system factory then creates an instance of this class and uses another builder class that parses the XML to populate these data structures. It would be nice to have a GUI to create the rules too, though that would be quite a bit of effort for something probably not used all that often. (It is probably more often than not *programmers* using this tool anyway, so they must be able to handle XML)

I know many users who would be very happy to have a click-friendly GUI
to define their own states ;) But yeah, that can come much later, being
able to write XML for the first iteration is plenty good.

>
>>> My plans/hopes:
>>> -figure out data driven choice in columns in the display.  Right now the
>>> columns are generic and there are only two of them.  User should be able
>>> to choose somehow, or it comes with the particular event/state schema.
>>> -extend the view so that it can display multiple traces at the same time,
>>> whichever traces are open in editors currently and support state display
>>> should be able to be aggregated.
>> There is definitely some work to be done in making the views more
>> generic. Right now the Control Flow and Resource View have a lot of
>> redundant code, which could be abstracted in a base class. That base
>> class could then be the basic for the custom view, to which you simply
>> pass a map of <table entries, state attributes> to populate.
> That is more or less what I have done.  I added an "IStateSystemPresentationInfo" interface that you can get an instance of from IStateSystem that gives you information on the context hierarchy, the possible states and their colors and such, and methods to get the state presentation for a given ITmfStateValue (used from the view).  This interface also has methods that are used from DataDrivenStateChangeInput when doing the one time population of the state system itself.

That's great to hear!

>  
>> For a custom trace type we use internally, we use such a view with
>> "events" on it, which are simply states with a duration of 1 time unit.
>> The view then knows that this particular type of event must be shown at
>> one pixel (or maybe it's three? a fixed amount, in any case), no matter
>> the zoom level.
> I would guess this is similar to the "Time Chart" view? 

Yes, I wasn't very clear but it's a view that has both states (like the
Control Flow View) and "punctual" events at the same time. Underneath
everything is a "state", we just display the event-like ones differently.

> I do envision something that is similar, with some subset of events optionally being added in the control flow view.  It is a bit difficult to decide what should be added though.  As you know if there are lots of events that can get really time and resource intensive to plot them all.

Even if there are a lot of states, it's possible for the view to do at
most 1 state system query per pixel (since we wouldn't be able to show
more information anyway). You can look at
ITmfStateSystem.queryHistoryRange() with the 'resolution' parameter, and
at how the Control Flow and Resource Views use it, it can give some ideas.

>
>> Thanks a lot for working on this and contributing it! I don't know if
>> you are familiar with how Eclipse handles external contributions, it's
>> not too complicated but let's say it's not trivial either. Right now we
>> have passed the deadline for accepting "big" (>250 lines) contributions
>> from non-committers for Kepler. (That could be worked-around if VMWare
>> had a committer in the Linux Tools project, but I don't think this is
>> the case.)
>>
>> But you could always post what you have on Gerrit so people can start
>> trying it out and reviewing it, but we won't be able to merge it into
>> master until after Kepler is released, in June.
> I have been in fast prototyping mode for some time, but with an eye towards contributing this stuff.  So, to that end I have started to clean things up, rip out dead code and various FIXME comments I dumped in along the way and such, so hopefully some time soon I can put my branch on github at least so that we can start the dialogue in earnest.  I have no experience with Gerrit, but will happily learn (any links to how to get started there?)

GitHub is fine, especially if it's not ready to be merged yet. It only
needs to go on Gerrit when it's ready to go in. You can look at
http://wiki.eclipse.org/Gerrit , it gives a nice overview. The section
about getting the commit hook (to add a "Change-Id" line in the commit
messages, which Gerrit uses) is easy to miss though.


Thanks,
Alexandre