Re: [linuxtools-dev] [TMF] New Features in TMF about filters expressions

[Alexandre Montplaisir <alexmonthy@xxxxxxxxxxxx>]

Hi Florian,

This is definitely interesting! However, with the recent discussions of
an eventual build-your-own-state-provider GUI, we are starting to wonder
if it's really worth exposing all the complexity of the statesystem
(attribute tree, paths, etc.) to the end-user.

Another hurdle is that right now the events table cannot be easily
extended. The number of columns is hard-coded, and to add more you have
to do some weird hacks. This is something we will want to fix for sure.
Each trace type should eventually be able to define its own columns. We
could provide a bunch of them, and specify which one are shown by
default. I'm thinking of column management like foobar2k or Clementine here.


I see nothing wrong with your syntax. What you describe in your first
point could be used in the eventual GUI to build custom state systems.
Custom columns would also go well with this. The goal here being to
allow users to add states or information for any trace type (like UST
traces coming from any userspace application).
Custom filters are a bit redundant I think. You can define a custom
column, and then filter on that. I think it's better to have one way of
doing things, or else people get confused... But of course, we would
have to see once we get to the actual implementation.

As for state entries/exits, this could go straight into the plugin. The
implementation underneath could use the filter expressions, if it makes
it easier.
Just a note about file opening though, you cannot assume that a file is
opened right at the "sys_open" event. You have to wait for the matching
exit_syscall : if the file opening fails, the return value of the
syscall will indicate so, and no file will be opened by the process. I
would propose assigning the "file opened" state at the exit_syscall ;
it's only then that the process can start using the descriptor.

Finally, as time goes on I'm believing less and less in state
information living in the CTF metadata. Not that it would be a bad idea,
but it would not be very flexible. I'm thinking this information should
belong in the viewer/analysis tool. There is no point creating state
systems containing a LOT of information if the viewer cannot make use of
it. It is also possible to define *different* state providers for the
same trace type: for example, you can get a LOT of stuff from a kernel
trace. But you might not want to build the same state system, depending
if you want to do memory analysis, network analysis, or just seeing
which process uses the most CPU.


tl;dr
We need to get custom number of columns working first. Then a good
contender would be custom column definition using a syntax like the one
you propose. Once this is working and well-oiled, we could re-use that
syntax in the custom state-provider-builder (which would give custom
control flow views etc. for any trace types, no coding involved). Then
there is nothing between us and world domination.

Time for lunch so I didn't really have time to proofread... If some
things don't make sense let me know!

Cheers,
Alexandre


On 13-03-12 10:23 AM, Florian Wininger wrote:
> Hi all,
>
> I'm looking to implement new features in TMF about filter expressions
> using the state system. I would appreciate receiving your feedback on
> the proposal.
>
> Currently, we can only specify constraints on the event context and payload.
> For example : /event_type == sys_read && fd == 3/
>
>   * *Specify filter expression using state information*
>
> The first feature is to add the use of state system attributes, to be
> mixed with payload and context information. For example : //event/type
> == "sys_read" && ///system/${/event/pid}/files/${/event/fd}/pathname ==
> "/etc/passwd"/
>
> I want to propose a syntax to make this expression. The nodes in the
> attribute tree are referenced using a hierarchical path. This path is a
> sequence of names and can be expressed with the usual notation, e.g.
> /processes/cpu/state. Each path component can be a constant string, or a
> quoted string to escape special characters. Alternatively a component
> can also be a number which is the put in []. This is used for directory
> nodes where all childs use integers as names, which is more efficient
> than treating numbers as strings and using strings. Finally, a component
> can be a variable where the value of the component is obtained by
> referencing a path; the $ notation is used as in shell scripts. We use
> the C/Java syntax, '==' to compare and '=' to assign.
>
> /processes/[4]/state == RUNNING
> /processes/[4]/state_start_time == 012032432
> /processes/[4]/PPID == 1
> /processes/[4]/user_mode/cumulative_time == 5789357
> /cpu/[0]/current_process == /processes/[1]
> /processes/[${/processes/[4]/PPID}]/childs_cumulative_time
>
> The model of the attribute tree is extended to make available as well
> some dynamically computed information related to the current context
> (channel, event). This additional information is not assigned nor saved
> in the history tree. For example, some information is associated with
> the current channel such as the CPU. In addition, all the fields of the
> current event as well are made available through the attribute tree
> syntax. For example, the following information may be accessed through
> the associated paths.
>
> /event/timestamp
> /event/type
> /event/TID
> /event/CPU
> /event/field/next_process
> /event/field/poll_array/[4]
>
> These values can be used in logical and arithmetic expressions for
> expressing conditions or assignments. Conditions are primarily used in
> filters, for example in the detailed event list view. They are also used
> as guards for state changes or as trigger for creating abstract events.
>
>   * *Specify expressions to determine state entry / exit*
>
> The second feature is to provide expression for state change.
> Currently this logic is coded in Java in files such as :
> org.eclipse.linuxtools.lttng2.kernel.core/src/org/eclipse/linuxtools/internal/lttng2/kernel/core/stateprovider/CtfKernelStateInput.java
>
> It would be nice to be able to provide expressions for that. This way,
> adding state for specific needs would not require programming and we
> could import/export new types of state systems easily (as part of the
> CTF metadata for example).
>
> Upon event of type sys_open:
> /processes/${/event/pid}/files/${/event/fd}/pathname = name
> /processes/${/event/pid}/files/${/event/fd}/mode = mode
>
>
> Once we can easily refer to state using such expressions, several
> additional features become relatively easy to add such as new "virtual"
> columns in the detailed event list computed based on an expression
> instead of being "event context" or "payload information". In addition,
> such expressions could become part of the CTF metadata to describe
> states, state entry expressions, colors to associate with specific
> states in the control flow view...
>
> I would like to have the developers' opinion on these two features. Do
> you see any major problem or easy simplification for this syntax?
>
> Thanks,
> Florian
>
> ------------------------------------------------------------------------------------------------
>
> Complete examples for transcription of the CtfKernetStateInput.java :
>
> case 1: // "exit_syscall":
> /* Clear the current system call on the process */
> /Threads/[${/CPUs/[${/event/Cpu}]/Current_thread}]/System_call = NULL
>
> /* Put the process' status back to user mode */
> /Threads/[${/CPUs/[${/event/Cpu}]/Current_thread}]/Status =
> PROCESS_STATUS_RUN_USERMODE
>
> /* Put the CPU's status back to user mode */
> /CPUs/[${/event/Cpu}]/Status = CPU_STATUS_RUN_USERMODE
>
>
> case 8: // "sched_process_fork":
> /* Assign the PPID to the new process */
> /Threads/[${/event/child_tid}]/PPID = /event/parent_tid
>
> /* Set the new process' exec_name */
> /Threads/[${/event/child_tid}]/Exec_name = /event/child_comm
>
> /* Set the new process' status */
> /Threads/[${/event/child_tid}]/Status = PROCESS_STATUS_WAIT_FOR_CPU
>
> /* Set the process' syscall name, to be the same as the parent's */
> /Threads/[${/event/child_tid}]/System_call =
> /Threads/[${/event/parent_tid}]/System_call
>
>
> case 11: // "lttng_statedump_process_state":
> /* Set the process' name */
> if( /Threads/[/event/tid]/Exec_name == Null)
>     /Threads/[/event/tid]/Exec_name = /event/name
>
> /* Set the process' PPID */
> if( /Threads/[/event/tid]/PPID == Null)
>     /Threads/[/event/tid]/PPID = /event/ppid
>
> /* Set the process' status */
> if( /Threads/[/event/tid]/Status == Null)
>     if(/event/status == 2)
>         /Threads/[/event/tid]/Status = PROCESS_STATUS_WAIT_FOR_CPU
>     else if(/event/status == 5)
>         /Threads/[/event/tid]/Status = PROCESS_STATUS_WAIT_BLOCKED
>     else
>         /Threads/[/event/tid]/Status = PROCESS_STATUS_UNKNOWN
>
>
>
>
>
> _______________________________________________
> linuxtools-dev mailing list
> linuxtools-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/linuxtools-dev