Re: [linuxtools-dev] TMF and pluggable state providers

[Aaron Spear <aspear@xxxxxxxxxx>]

Hi Alex,

Thanks for the positive feedback.  Some replies below.

> > 1) I changed the TMF code base so that it was possible to have a plugin
> > contribute a named factory that can be used to create state systems. The
> > base implementation of TmfTrace.buildStateSystem checks to see if there is
> > a property for a given trace that specifies which state system factory to
> > use and creates it using that if so. This is nice because you can still
> > have the state system be a function of the derived trace if you want to
> > (like it is for LTTng currently), but you can also have a state system
> > implementation that works with different traces regardless of the
> > underlying file format.
> 
> The way we've done it so far is to make the trace type object (so, the
> class that extends ITmfTrace) be the central point used to specify new
> trace types. It defines its event type, timestamps, statistics backend,
> state system(s), etc., if applicable. That way we don't need a separate
> extension points for timestamps, event types, event fields and so on.
> 
> For state systems, as you might have noticed, there is a base
> implementation in TmfTrace.buildStateSystem(), that can be extended by
> specific trace types if they want to add (or override) state systems to
> this trace. All assigned state systems have a global ID, so they can be
> accessible from anywhere (and not only those that have a reference of
> the specific object type).
> 
> I haven't seen your implementation so I can't comment, but is there
> anything that couldn't be done with the current method, and that
> required you to add an extension point for state systems too?

The problem as I see it is the requirement to derive a new type of trace in order to get state.  I want to have a facility that can create state for *any* trace knowing the event schema, but irrespective of the trace format.  So, I wanted to simply use the CustomTxtTrace for instance without having to derive and then override TmfTrace.buildStateSystem.  This is why I added an extension point for plugins that can provide instances of "IStateSystemFactory", an object that can create state systems given an ITmfTrace instance.  So then you can have a plugin that provides state systems in whatever sort of way you wish.  I then additionally created a data driven way to parse state from any trace.

Here is the changed code in TmfTrace.buildStateSystem, perhaps this will clarify what I am doing:

 protected void buildStateSystem() throws TmfTraceException {

        /**
         * derived classes may overload this method to specify dedicated state
         * system behavior.
         *
         * The base class does now optionally allow "mixing in" an
         * implementation of a state provider that can come from other plugins.
         * To do this we get a list of factories that support this particular
         * trace instance.
         */
        IResource resource = this.getResource();
        String selectedStateFactoryId = null;

        try {
            // get the property that has the id of the selected state factory if it exists
            selectedStateFactoryId = resource.getPersistentProperty(TmfCommonConstants.TRACE_SELECTED_STATE_PROVIDER);
        } catch (CoreException e) {
            throw new TmfTraceException(e.toString(), e);
        }

        IStateSystemFactory selectedStateFactory = null;

        if (selectedStateFactoryId != null) {
            // get the factory by string id
            selectedStateFactory = StateSystemFactoryManager.getInstance().getFactoryById(selectedStateFactoryId);
        } else {
            // There is no factory selected in the preference.  Get the list of all factories
            // that advertise working for this particular trace
            List<IStateSystemFactory> availableStateSystemFactories = StateSystemFactoryManager.getInstance().getFactoriesForTrace(this);
            if (availableStateSystemFactories.size() > 0) {
                // TODO we need a mechanism to allow the user to select which
                // factory they want to use in the case that there are multiple
                // that will work.  This is probably a rare use case anyway, so
                // simply select the first one for nows
                selectedStateFactory = availableStateSystemFactories.get(0);
            }
        }
        if (selectedStateFactory != null) {
            ITmfStateSystem ss = selectedStateFactory.createStateSystem(this);
            if (ss != null) {
                fStateSystems.put(selectedStateFactory.getStateSystemId(), ss);
                try {
                    resource.setPersistentProperty(TmfCommonConstants.TRACE_SELECTED_STATE_PROVIDER, selectedStateFactory.getStateSystemId());
                } catch (CoreException e) {
                    throw new TmfTraceException(e.toString(), e);
                }
            }
        }
    }

> > 2) Using the infrastructure above, I built a framework that does data
> > driven state extraction. The idea is that you have some metadata that
> > describes your hierarchy, event types that correspond to context and state
> > changes, and rules for how to extract the new state and context from the
> > events. It uses the TMF abstraction for events, and so it works
> > independent of how the underlying trace is structured which is nice.
> >
> > 3) I have an implementation of the data driven state extraction above that
> > right now is populated using a simple convention whereby the factory looks
> > for an XML file that describes the state data right next to the trace. The
> > right way to do this is to use some nice GUI to either create rules or
> > manually choose which state types you want to use for a given trace, but
> > it works for now. Attached is the XML file used to create the VM trace
> > above so you have some idea of how this works. Again, there are layers to
> > this. In the case of the java function trace I actually have a derived
> > trace type, so for that one the XML file can be in the plugin (or you can
> > hard code it and not bother with the XML)
> 
> Awesome. State-definitions in XML or similar has been a hot topic
> recently. In the base TMF framework we could have a special state
> provider (implementing IStateChangeInput) that would read such a file
> (and eventually, the custom contents that was defined in a GUI), that
> could then be used to populate custom views.

This is exactly what I did.  I created a generic "DataDrivenStateChangeInput" class that has data structures with mappings from event types to fields and such to extract to figure out state.  The state system factory then creates an instance of this class and uses another builder class that parses the XML to populate these data structures. It would be nice to have a GUI to create the rules too, though that would be quite a bit of effort for something probably not used all that often. (It is probably more often than not *programmers* using this tool anyway, so they must be able to handle XML) 

> > My plans/hopes:
> > -figure out data driven choice in columns in the display.  Right now the
> > columns are generic and there are only two of them.  User should be able
> > to choose somehow, or it comes with the particular event/state schema.
> > -extend the view so that it can display multiple traces at the same time,
> > whichever traces are open in editors currently and support state display
> > should be able to be aggregated.
> 
> There is definitely some work to be done in making the views more
> generic. Right now the Control Flow and Resource View have a lot of
> redundant code, which could be abstracted in a base class. That base
> class could then be the basic for the custom view, to which you simply
> pass a map of <table entries, state attributes> to populate.

That is more or less what I have done.  I added an "IStateSystemPresentationInfo" interface that you can get an instance of from IStateSystem that gives you information on the context hierarchy, the possible states and their colors and such, and methods to get the state presentation for a given ITmfStateValue (used from the view).  This interface also has methods that are used from DataDrivenStateChangeInput when doing the one time population of the state system itself.
 
> For a custom trace type we use internally, we use such a view with
> "events" on it, which are simply states with a duration of 1 time unit.
> The view then knows that this particular type of event must be shown at
> one pixel (or maybe it's three? a fixed amount, in any case), no matter
> the zoom level.

I would guess this is similar to the "Time Chart" view?  I do envision something that is similar, with some subset of events optionally being added in the control flow view.  It is a bit difficult to decide what should be added though.  As you know if there are lots of events that can get really time and resource intensive to plot them all.

> Thanks a lot for working on this and contributing it! I don't know if
> you are familiar with how Eclipse handles external contributions, it's
> not too complicated but let's say it's not trivial either. Right now we
> have passed the deadline for accepting "big" (>250 lines) contributions
> from non-committers for Kepler. (That could be worked-around if VMWare
> had a committer in the Linux Tools project, but I don't think this is
> the case.)
> 
> But you could always post what you have on Gerrit so people can start
> trying it out and reviewing it, but we won't be able to merge it into
> master until after Kepler is released, in June.

I have been in fast prototyping mode for some time, but with an eye towards contributing this stuff.  So, to that end I have started to clean things up, rip out dead code and various FIXME comments I dumped in along the way and such, so hopefully some time soon I can put my branch on github at least so that we can start the dialogue in earnest.  I have no experience with Gerrit, but will happily learn (any links to how to get started there?)

thanks,
Aaron