Bug 249227 - [console][cdi] Add support for "verbose" tracing of MI commands in console
Summary: [console][cdi] Add support for "verbose" tracing of MI commands in console
Status: RESOLVED FIXED
Alias: None
Product: CDT
Classification: Tools
Component: cdt-debug-dsf-gdb (show other bugs)
Version: 0 DD 1.1   Edit
Hardware: All All
: P3 enhancement with 1 vote (vote)
Target Milestone: 6.0   Edit
Assignee: Marc Khouzam CLA
QA Contact: Marc Khouzam CLA
URL:
Whiteboard:
Keywords: api
Depends on: 240092
Blocks: 266863
  Show dependency tree
 
Reported: 2008-09-30 17:04 EDT by Pawel Piech CLA
Modified: 2014-01-29 22:17 EST (History)
6 users (show)

See Also:
cdtdoug: iplog-
marc.khouzam: review? (pawel.1.piech)


Attachments
Simple (insufficient) fix (1.17 KB, patch)
2008-10-03 06:02 EDT, Marc Khouzam CLA
cdtdoug: iplog-
Details | Diff
Improved debug traces (2.66 KB, patch)
2008-10-03 12:41 EDT, Marc Khouzam CLA
cdtdoug: iplog-
Details | Diff
Adding "verbose console" option to the launch (7.58 KB, patch)
2008-10-06 14:52 EDT, Marc Khouzam CLA
cdtdoug: iplog-
Details | Diff
Temporary hiding of "verbose" option (870 bytes, patch)
2008-10-10 10:34 EDT, Marc Khouzam CLA
cdtdoug: iplog-
Details | Diff
Simple suggestion (3.60 KB, patch)
2008-10-16 11:10 EDT, Marc Khouzam CLA
cdtdoug: iplog-
Details | Diff
Prototype of dedicated trace console (17.06 KB, patch)
2009-02-19 16:30 EST, Marc Khouzam CLA
no flags Details | Diff
Improved prototype of dedicated trace console (16.10 KB, patch)
2009-02-19 23:42 EST, Marc Khouzam CLA
no flags Details | Diff
Improved prototype 2 (16.21 KB, patch)
2009-02-20 13:51 EST, Marc Khouzam CLA
no flags Details | Diff
TracingConsoleManager solution (35.10 KB, patch)
2009-02-28 03:18 EST, Marc Khouzam CLA
no flags Details | Diff
Cleanup of old Verbose launch option (6.87 KB, patch)
2009-02-28 10:25 EST, Marc Khouzam CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Pawel Piech CLA 2008-09-30 17:04:41 EDT
CDI-GDB has a "verbose" option which causes the GDB debugger to print the MI traffic in the console belonging to the GDB process.  This option should be supported in DSF-GDB as well.
Comment 1 Daniel Jacobowitz CLA 2008-10-01 17:30:26 EDT
FYI, this is the only thing from your list of missing CDI-GDB features that would be a big problem for us to switch.  We use this all the time, for both development and support.
Comment 2 Marc Khouzam CLA 2008-10-03 06:02:07 EDT
Created attachment 114171 [details]
Simple (insufficient) fix

(In reply to comment #1)
> FYI, this is the only thing from your list of missing CDI-GDB features that
> would be a big problem for us to switch.
 
Since fixing this feature can help using DSF/GDB, I thought I'd give it a shot.

This attempt is only to get the traces to print, not to configure if they should print or not, which is simple.

It turns out that printing the traces to the console is non-trivial (at least for me :-)) A very simple solution is illustrated by this patch.
But it has the following problems:
1 - The --thread/--frame options will not be printed because the CommandHandle.toString() does not know if these options were used or not.  We can try to fix this problem but there is a more fundamental problem, I think

2 - The traces are printed once the reply to the command is received and after DSF has done some parsing/handling.  But Daniel mentioned that these traces would be used for GDB support, and therefore I think they should reflect as closely as possible what is sent and received from GDB.  For instance, if a command was to crash GDB (far-fetched example ;-)), there would be no answer to the command, and therefore, no trace of the command; that is a problem for troubleshooting.

That last point also makes me think that our current debug traces are insufficient.  I think we should straight out print what is sent to GDB and what is received from GDB (except maybe for the prompt.)

So, this patch is for documentation/discussion as I don't feel it is sufficient.  I will post another solution.
Comment 3 Marc Khouzam CLA 2008-10-03 06:13:55 EDT
Small note to say that using a combination of commandSent() and commandDone() to print the traces is better but not sufficient still.  This is because -thread-select and -frame-select do not trigger commandSent() (maybe they should?)
Comment 4 Pawel Piech CLA 2008-10-03 12:36:09 EDT
It's funny how such a simple feature can be such a challange.  I agree that the simple solution is not sufficient, and our tracing implementation is also somewhat inaccurate.

To address this properly I think it would be best to make the MI protocol stream be available from AbstractMIControl using the IStreamsProxy interface.  IStreamsProxy is normally returned by IProcess object and is implemented by internal platform classes.  Once we have this streams proxy, we would then also need to implement our own IProcess object representing the GDB process, instead of implementing java.lang.Process as we do now.  In this IProcess implementation we could then easily switch between displaying the CLI interface or the raw MI interface.  This solution would also allow the user to send MI commands in addition to monitoring.
Comment 5 Marc Khouzam CLA 2008-10-03 12:41:27 EDT
Created attachment 114208 [details]
Improved debug traces

(In reply to comment #4)
> To address this properly I think it would be best to make the MI protocol
> stream be available from AbstractMIControl using the IStreamsProxy interface. 
[...]

Thanks for the guidance, I'll try to look into it soon.

This patch is a suggestion on improving our debug traces.  The traces when we send a command to GDB used to be in a executor call.  I didn't know why...  Is there a reason for that?  (I removed it in the patch.)
Comment 6 Pawel Piech CLA 2008-10-03 14:02:49 EDT
The traces (which ultimately use System.out) are synchronized using the executor thread as anything else, but I don't think this is strictly necessary.
Comment 7 Marc Khouzam CLA 2008-10-03 14:32:44 EDT
(In reply to comment #6)
> The traces (which ultimately use System.out) are synchronized using the
> executor thread as anything else, but I don't think this is strictly necessary.

Since it is not "strictly necessary" :-) I'll remove it because I like the idea of the trace coming out at the same time as when we send it to GDB.

Committed the "Improved debug traces" patch.

Comment 8 Marc Khouzam CLA 2008-10-06 14:52:08 EDT
Created attachment 114346 [details]
Adding "verbose console" option to the launch

we are not quite ready with this feature, but since I had the changes needed to add the option in the launch, I figure I'd commit them.  Once we have the feature ready, we'll just have to hook them to the launch option.
Comment 9 Pawel Piech CLA 2008-10-09 13:55:59 EDT
Should address the separation of the back end process first.
Comment 10 Marc Khouzam CLA 2008-10-10 10:34:33 EDT
Created attachment 114801 [details]
Temporary hiding of "verbose" option

We have agreed that this bug needs to wait for bug 240092.  Furthermore, we currently believe this bug will require changes in interface.  Since we won't have time to complete this bug for M3 (API freeze), it will probably not make it into the 1.1 release.

Therefore, this patch makes the "verbose" option of the launch invisible until it actually works.

Committed
Comment 11 Marc Khouzam CLA 2008-10-16 11:10:18 EDT
Created attachment 115257 [details]
Simple suggestion

(In reply to comment #4)
> To address this properly I think it would be best to make the MI protocol
> stream be available from AbstractMIControl using the IStreamsProxy interface. 

I played around with this idea, and I think I finally understood it.  The ideas was to allow AbstractCLIProcess to get access to the MI protocol streams so as to print their content, right?

The idea is quite elegant.  However, I think it is not sufficient.  The problem is that the IStreamsProxy interface allows to write to the MI input stream, but does not give a monitor for it.  Therefore, when AbstractMIControl writes a command to GDB, AbstractCLIProcess will not see this command but only the result of it, coming in through the OutputStreamMonitor.

Or is there some way to monitor the InputStream also?


In the end, I'm thinking that simply giving AbstractMIControl access to the Console OutputStream through AbstractCLIProcess may be the easiest thing to do.  This is what the patch illustrates.  It does add two new interface methods in the MI plugin though...

Let me know what you think.
(Is the M3 build done?)

> This solution would also allow the user to send MI
> commands in addition to monitoring.

I think we can address this part separately in AbstractCLIProcess.
Comment 12 Marc Khouzam CLA 2008-10-16 13:18:21 EDT
(In reply to comment #11)
> > This solution would also allow the user to send MI
> > commands in addition to monitoring.
> 
> I think we can address this part separately in AbstractCLIProcess. 

I have opened bug 251101 to track this.
Comment 13 Pawel Piech CLA 2008-10-16 18:29:21 EDT
(In reply to comment #11)
> Created an attachment (id=115257) [details]
> Simple suggestion
> 
> (In reply to comment #4)
> > To address this properly I think it would be best to make the MI protocol
> > stream be available from AbstractMIControl using the IStreamsProxy interface. 
> 
> I played around with this idea, and I think I finally understood it.  The ideas
> was to allow AbstractCLIProcess to get access to the MI protocol streams so as
> to print their content, right?
> 
> The idea is quite elegant.  However, I think it is not sufficient.  The problem
> is that the IStreamsProxy interface allows to write to the MI input stream, but
> does not give a monitor for it.  Therefore, when AbstractMIControl writes a
> command to GDB, AbstractCLIProcess will not see this command but only the
> result of it, coming in through the OutputStreamMonitor.
> 
> Or is there some way to monitor the InputStream also?
Yes, I actually imagined that these stream monitors would exist in addition to the existing input/output streams.  So the output stream monitor would echo the 
stuff written to the input stream as well (stuff written using IStreamsProxy.write() would not be echoed).

I wanted to bypass AbstractCLIProcess altogether and instead directly implement the IProcess interface... either by creating a separate IProcess entry in the Debug view or by somehow switching the proxy in the gdb's IProcess implementation.

So looking at your solution, I don't think it's far off, I do think it makes more sense to implement the streams proxy in the control rather than the backend service.
Comment 14 Marc Khouzam CLA 2009-02-16 10:55:58 EST
I finally got back to this.

I was thinking that we could improve the user's experience compared to CDI.  I never liked that the CDI verbose console interleaved the GDB MI traces with the console that is presented to the user.

Maybe we should keep the MI traces isolated instead of reusing the gdb console.    With this, we could have the MI traces always on and therefore always available.

What we could do is:
1- have a dedicated console for the MI traces.  The difficulty here is that this special console should not keep popping up to the front whenever something is printed.

2- we could simply dump the traces to a file.  This is easier to manage from the point of view of the UI, gives more persistence and avoids having to copy/paste from the console.  The complication I was thinking of is supporting multiple launches at the same time, but it should not be too bad.

or

we can simply do what CDI does and intermix the traces in the GDB console.

Any opinions?
Comment 15 Pawel Piech CLA 2009-02-17 12:12:36 EST
(In reply to comment #14)
I think it's a good idea to prototype.  To integrate with the console view at a lower level you should take a look at org.eclipse.ui.console.IConsoleManager.  You may need to implement something equivalent to org.eclipse.debug.internal.ui.views.console.ProcessConsoleManager.
Comment 16 Marc Khouzam CLA 2009-02-19 16:30:31 EST
Created attachment 126220 [details]
Prototype of dedicated trace console

Here is a prototype (trace output only).
It creates a special DebugTraceConsole which is always present when launching DSF.   AbstractMIControl prints all traces to that console.  The new console never jumps to the front by itself and it follows the launch lifecycle (as the ProcessConsole does).

What I haven't looked at yet is if we want to accept input from that console.

Any comments?
Comment 17 Marc Khouzam CLA 2009-02-19 22:18:55 EST
(In reply to comment #16)
> What I haven't looked at yet is if we want to accept input from that console.

After thinking about this, I realized that if we accepted input on such a DebugTraceConsole, we would still need to deal with it properly before sending a command to GDB, so that DSF-GDB could handle the answer properly.  What that means is that it would be the exact same thing as typing to our current cli console.

Therefore, I feel we should not read any of the input given by the user on such a dedicated Trace console.
Comment 18 Marc Khouzam CLA 2009-02-19 23:42:45 EST
Created attachment 126257 [details]
Improved prototype of dedicated trace console

I cleaned up the patch a bit.

The really important change is that I made the new Traces console read-only.  I realized that if it was not read-only, I would need a thread to read whatever the user was typing to avoid a memory leak.  As I mentioned before, I feel that there is no value in the user being to write to that console, so read-only is fine in my eyes.

In this patch, I'm 'illegally' extending MessageConsole.  Here is why:
- IOConsole.setName() is protected so the only way to call it is to extend IOConsole.
- but to make an IOConsole read-only, I need to call IOConsolePage.setReadOnly() which is in an internal package.
- a MessageConsole is already read-only, so I extended that, which violates its @noextend tag
- without getting help from the platform, the only other solution I see is to redefine my own local copy of IOConsolePage.
- sigh...

Finally, I was thinking that having the traces on all the time was nice, but could be annoying for some people.  Instead of making it a launch option, since the console is not very intrusive, I was thinking that it should be a Preference.
Comment 19 Marc Khouzam CLA 2009-02-19 23:45:42 EST
Oh yeah, and I added timestamps to the traces.  I was 100% sure I should, so I'm open to disagreements :-)
Comment 20 Marc Khouzam CLA 2009-02-20 13:51:54 EST
Created attachment 126334 [details]
Improved prototype 2 

I changed my mind about the console being read-only.
When looking at traces, I like to have the ability to insert delimiters between traces, like blank like or --------------------- etc.

So, I added an InputReadJob() to the DebugTracingConsole, which just swallows what the user types.
That also allowed me to inherit from IOConsole and remove my whole 'illegal' extension problem.

Last thing to add is the preference setting.  I'll probably wait for feedback before doing this.
Comment 21 Pawel Piech CLA 2009-02-20 17:48:20 EST
I don't have time to try to run this right now, but I really like this approach.
Comment 22 Marc Khouzam CLA 2009-02-20 20:51:08 EST
(In reply to comment #19)
> Oh yeah, and I added timestamps to the traces.  I was 100% sure I should, so
> I'm open to disagreements :-)

This should have said "I was NOT 100% sure"

(In reply to comment #21)
> I don't have time to try to run this right now, but I really like this
> approach.

At least I know I'm no way off :-)
I'll get the preferences to work before committing anything.
This feature is all in DSF-GDB right now (although in the MI part also.)
The remaining work is the preferences.
All preferences we have know are part of DSF, so I'm not sure how to keep
this new preference in DSF-GDB.  But since I want to get this in for M6, I think it is safer to do it in DSF-GDB, at least for now.

One thing I wanted to add was that I tried to make this feature is such a way that it could easily be customizable.  For example, I was able to, with a two-line change in the CommandControl service, to use System.out as the trace stream instead of a new console, which gave the exact same traces as what we've always had, but using the new trace 'infrastructure'.
Comment 23 Marc Khouzam CLA 2009-02-23 11:28:44 EST
Using preferences in DSF is not very clear to me.

A preference store is in the UI plugin (GdbUIPlugin in this case.)  However, I need to use the information in the non-ui plugin (Gdb plugin.)

I was thinking of reading the preferenceStore in LaunchVMProvider and then calling the GDBControl service to notify it that the traces property has changed.  I'm not entirely sure that is what I am supposed to do though.

Even worse is that I need to know the trace preference when GDBControl is initialized so as to start the traces right away.  However, I don't know what gets initialized first, the LaunchVMProvider, or the GDBControl service.  From setting a breakpoint, I saw the LaunchVMProvider being initialized first, which prevents me from calling the GDBControl service right away.

So, I don't know how to have GDBControl know about the trace preference right at the start.

Thanks for any guidance.


Comment 24 Pawel Piech CLA 2009-02-23 11:43:30 EST
(In reply to comment #23)

You can retrieve preferences in a non-UI plugin just as well, IMO you should put reading of the preferences right into GDBControl.  You may want to invent some kind of service event that the LaunchVMProvider can listen to to discover when MI logging gets initiated.  This way the GDBControl could start MI logging at any time in response to the preferences changed event.
Comment 25 Marc Khouzam CLA 2009-02-23 12:59:05 EST
(In reply to comment #24)
> You can retrieve preferences in a non-UI plugin just as well, IMO you should
> put reading of the preferences right into GDBControl.  

I didn't know you could do that.  But now that you mentioned it, I found
Platform.getPreferencesService()
which allows any plugin to access the preferences of other plugins

Thanks!

> You may want to invent
> some kind of service event that the LaunchVMProvider can listen to to discover
> when MI logging gets initiated.  This way the GDBControl could start MI logging
> at any time in response to the preferences changed event.

The only reason I was including LaunchVMProvider in this change was because I thought I had to read preferences in a UI-plugin.  But if I don't, then there is not need to include LaunchVMProvider at all.

Comment 26 Marc Khouzam CLA 2009-02-24 09:49:20 EST
(In reply to comment #24)

> You can retrieve preferences in a non-UI plugin just as well, IMO you should
> put reading of the preferences right into GDBControl.  You may want to invent
> some kind of service event 

I may have to do something like that if I want to keep dependencies on UI plugins within other UI plugins.  This new tracing is done within a console which is part of eclipse.ui.  So either our gdb plugin accesses eclipse.ui, or I keep it in gdb.ui, but then I'll need some event like you suggest.

I'll try the event.
Man, this feature is more complex than I thought.
Comment 27 Marc Khouzam CLA 2009-02-28 03:18:51 EST
Created attachment 127073 [details]
TracingConsoleManager solution

Well, this turned out to be a much bigger effort than I expected :-)

It turned out to be tricky to avoid using org.elclipse.ui.console from our non-ui plugins.  I finally managed it by imitating the ProcessConsoleManager class.

So this solution is to have a TracingConsole for any launch that is tagged ITracedLaunch (basically GdbLaunch for now.)  Tracing is controlled by a preference in a new GDB preference page, under the DSF preference page.

I find the solution very elegant, if I can say so myself.  The last part that bothers me is the way the Tracing Stream is passed back to the AbstractMIControl class.  I envision a more generic way where any class could have access to a Tracing Stream.  I was thinking of using the Launch to achieve this.  But this will have to wait for later.

I used the ICommandControlInitializedDMEvent to notify a TracingConsole that the AbstractMIControl service was ready.  I really should use a new event that would trigger a little earlier, but since I'm hoping for an eventual better solution on that front, I'm leaving it for now.

I used the DSF executor to print the traces because I was not sure about concurrency issues.  I would like to eventually discuss this, but there is no hurry.

What I'm left with now is to remove the 'Verbose' option I had added in the Launch, since we won't be using it.

I'm committing this patch.

Comments, whenever possible, are more than welcome.
Comment 28 Marc Khouzam CLA 2009-02-28 10:25:02 EST
Created attachment 127081 [details]
Cleanup of old Verbose launch option

No longer need the Verbose console launch option.

This patch removes it (it was not being displayed, but it was in the code.)
Committed.
Comment 29 Marc Khouzam CLA 2009-02-28 11:02:37 EST
This feature now works for GDB.
It would be nice to migrate it to DSF or even platform, but that should be another bug.

Pawel, can you review?