Community
Participate
Working Groups
Using CDT, I am trying to debug an application using a small ARM board as the target. I consistently get this error: http://nova.polymtl.ca/~simark/ss/fileikFgVT.png The launch is configured to use an SSH remote connection and to upload the binary to the target. If I choose to "Skip download to target path" (it sounds like it should say upload... anyway), then it works. When CDT tries to upload the binary itself, I can see that the binary is correctly uploaded (the md5sum matches), but the file is not executable: $ ls -ltr ... -rw-rw-r-- 1 simark simark 32919 Mar 15 14:02 TestArm
Can you attach the screenshot to this bug. The day you loose your account at polymtl.ca we loose the info for this bug. Doesn't CDT explicitly set the executable bit for the file?
I made the following patch: https://git.eclipse.org/r/#/c/68396/ It makes the remote launch works for my scenario, but I am not 100% that it's the right fix. AFAIU, the procedure to prepare a remote launch looks like this: 1. Upload the file using sftp 2. Start a remote shell and execute chmod +x 3. Start a remote shell and execute gdbserver My initial thought was that we don't wait until we have confirmation that #2 is complete before continuing with #3. So gdbserver would try (and fail) to execute a file that is not executable yet. I would find it surprising however that gdbserver's shell and gdbserver would start faster than the time required by chmod to do its work. But it's plausible, since there's nothing that guarantees the order of execution right now, so let's assume this is what's happening. Even when we get the error, presumably because gdbserver couldn't execute the file, the chmod process should still finish eventually and leave us with an executable file on the filesystem. However, that's not what I see, the file is still non-executable. Because of this, I don't think I fully understand the problem, and therefore I am not sure my patch is right. It might be that it changes the timing in some way and I get lucky, without fixing the real problem...
Created attachment 260320 [details] Screenshot
(In reply to Marc Khouzam from comment #1) > Can you attach the screenshot to this bug. The day you loose your account > at polymtl.ca we loose the info for this bug. > > Doesn't CDT explicitly set the executable bit for the file? Yes, it runs chmod on the target. Although it doesn't check if the command worked, so it could fail and we wouldn't know.
Here is the relevant code, by the way: http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/tree/cross/org.eclipse.cdt.launch.remote/src/org/eclipse/cdt/launch/remote/RSEHelper.java#n191
I replaced the call to chmod by a small wrapper that logs its invocation: chmod $@ echo "$(date) chmod $@" >> /tmp/chmod-log And it appears that when the problem occurs, chmod is not called. So the problem may revolve around the fact that the second ssh shell execution (to start gdbserver) messes with the first one (that executes chmod). I tried to dive in the RSE code, but it's not clear to me what happens. It explains at least why the file ends up non-executable, the chmod command never actually gets executed on the target.
(In reply to Simon Marchi from comment #6) > I replaced the call to chmod by a small wrapper that logs its invocation: > > chmod $@ > echo "$(date) chmod $@" >> /tmp/chmod-log > > And it appears that when the problem occurs, chmod is not called. So the > problem may revolve around the fact that the second ssh shell execution (to > start gdbserver) messes with the first one (that executes chmod). I tried > to dive in the RSE code, but it's not clear to me what happens. > > It explains at least why the file ends up non-executable, the chmod command > never actually gets executed on the target. Could it be that ssh command is sent, but not executed on target side? I have seen a similar issue with Bug 467833 where I have ended implementing an workaround on eclipse side.
(In reply to Simon Marchi from comment #0) > Using CDT, I am trying to debug an application using a small ARM board as > the target. I consistently get this error: > > http://nova.polymtl.ca/~simark/ss/fileikFgVT.png > > The launch is configured to use an SSH remote connection and to upload the > binary to the target. If I choose to "Skip download to target path" (it > sounds like it should say upload... anyway), then it works. When CDT tries > to upload the binary itself, I can see that the binary is correctly uploaded > (the md5sum matches), but the file is not executable: > > $ ls -ltr > ... > -rw-rw-r-- 1 simark simark 32919 Mar 15 14:02 TestArm The setup is close to the setup from Bug 467833.
> Could it be that ssh command is sent, but not executed on target side? I > have seen a similar issue with Bug 467833 where I have ended implementing an > workaround on eclipse side. It could be, but I don't really know how to debug that. You seem to have some experience debugging this kind of scenario, do you have a some pointers oh how you do that? Would sniffing with Wireshark help me at all, or I won't see anything useful because it's encrypted?
(In reply to Simon Marchi from comment #9) > > Could it be that ssh command is sent, but not executed on target side? I > > have seen a similar issue with Bug 467833 where I have ended implementing an > > workaround on eclipse side. > > It could be, but I don't really know how to debug that. > > You seem to have some experience debugging this kind of scenario, do you > have a some pointers oh how you do that? Would sniffing with Wireshark help > me at all, or I won't see anything useful because it's encrypted? It did help to start ssh daemon with additional verbose setting to confirm that packets do reach into target applications. BTW, do you experience similar issue with CDT 8.8.1? (using RSE ssh capabilities)
(In reply to Teodor Madan from comment #10) > It did help to start ssh daemon with additional verbose setting to confirm > that packets do reach into target applications. Some things appear in the log, but I can't really make sense out of it. Also, I don't know if those are related to the "chmod" connection or "gdbserver" connection (assuming they use different connections). > BTW, do you experience similar issue with CDT 8.8.1? (using RSE ssh > capabilities) I just checkout out CDT' 8.8 branch. The first two launches worked somehow. but now it's constantly failing as well.
(In reply to Simon Marchi from comment #11) > (In reply to Teodor Madan from comment #10) > > BTW, do you experience similar issue with CDT 8.8.1? (using RSE ssh > > capabilities) > > I just checkout out CDT' 8.8 branch. The first two launches worked somehow. > but now it's constantly failing as well. Make sure that you rebuild RSE for Bug 467833. Also, to activate workaround you have to pass "-vmargs -DRSE_SHELL_READY_PING=200" to eclipse
(In reply to Simon Marchi from comment #11) > (In reply to Teodor Madan from comment #10) > > It did help to start ssh daemon with additional verbose setting to confirm > > that packets do reach into target applications. > > Some things appear in the log, but I can't really make sense out of it. > Also, I don't know if those are related to the "chmod" connection or > "gdbserver" connection (assuming they use different connections). AFAIR, I had to rebuild sshd with debug define to output channel messages as well.
(In reply to Teodor Madan from comment #12) > Make sure that you rebuild RSE for Bug 467833. Also, to activate workaround > you have to pass "-vmargs -DRSE_SHELL_READY_PING=200" to eclipse I am not sure what you mean by having to rebuild RSE. The version I use already has the code that handles RSE_SHELL_READY_PING it seems. So I just tested adding -DRSE_SHELL_READY_PING=200 to my test Eclipse launch, it seems like it does fix the problem. I can see briefly (before it disappears) the "ping"'s being displayed on the remote shell view.
(In reply to Simon Marchi from comment #14) > So I just tested adding -DRSE_SHELL_READY_PING=200 to my test Eclipse > launch, it seems like it does fix the problem. I can see briefly (before it > disappears) the "ping"'s being displayed on the remote shell view. To me this is a confirmation of the same target SW/HW stack issue for opening a ssh channel. And the proper fix would be in o.e.remote to have a similar ssh opening handshake implementation.
*** Bug 510605 has been marked as a duplicate of this bug. ***
Can someone explain why is this bug having such a low priority? There was a workaround for RSE but now (eclipse neon.2) there's no way to start a remote debug session...
There are 2 bugs related to launching GDB remotely: 1. The two procedures that are executing remote commands RemoteHelper.execCmdInRemoteShell and RemoteHelper.remoteShellExec don't wait for shell prompt after the command is executed. For this reason new commands try to execute before the previous command has finished and the system gets blocked because there is no shell prompt available. To fix this I wrote a new procedure waitForShellPrompt that should be called before the output stream is flushed. private static void waitForShellPrompt(Channel channel, InputStream is) throws Exception{ int SIZE = 1024; byte[] buffer = new byte[SIZE]; String line=""; while(true){ while (is.available() > 0) { int i = is.read(buffer, 0, 1024); if (i < 0) { break; } line = new String(buffer, 0, i); line= line.trim(); // System.out.println(line); } if(line.contains("logout")) { break; } if (channel.isClosed()) { break;} if (!line.isEmpty() && (line.endsWith("#") || line.endsWith(">") || line.endsWith("$") || line.endsWith(":"))) {break;} try { Thread.sleep(500); } catch (InterruptedException e) { e.printStackTrace(); } } } insert a call to the above procedure before the output stream is flushed as below: IRemoteCommandShellService shellService = conn.getService(IRemoteCommandShellService.class); IRemoteProcess remoteProcess = null; Process p = null; try { remoteProcess = shellService.getCommandShell(IRemoteProcessBuilder.ALLOCATE_PTY); p = new RemoteProcessAdapter(remoteProcess); InputStream is =p.getInputStream(); OutputStream os = remoteProcess.getOutputStream(); os.write(remoteCommand.getBytes()); waitForShellPrompt(((JSchProcessBuilder) remoteProcess.getProcessBuilder()).getChannel(), is); os.flush(); } catch (IOException e) { abort(Messages.RemoteRunLaunchDelegate_7, e, ICDTLaunchConfigurationConstants.ERR_INTERNAL_ERROR); } catch (Exception e){ e.printStackTrace(); } monitor.done(); return p; } It works for me but I do not know if it is the best solution as various machines can have various shell prompts and I check only for #, $, :, > prompts. This procedure should be called from both remoteShellExec and execCmdInRemoteShell. 2. The second bug is related to the RemoteGdbLaunchDelegate.launch method. Apparently a listener on the monitoring stream that returns the GDB server output has been attached to the stream AFTER the command that starts GDB server has been executed. For this reason when we try to start GDB for the first time we'll not receive the response from the server because the LISTENER IS NOT triggered. However when we start the GDB server for the second time then the listener will be triggered and we'll receive the response from the server. I suggest the fix below: IStreamsProxy streams = iProcess.getStreamsProxy(); IStreamMonitor monitorStream = streams.getOutputStreamMonitor(); if (monitorStream.getContents().contains("Listening on port")) { synchronized (lock) { gdbServerReady[0] = true; lock.notifyAll(); }}; This code should be inserted before the listener is added to the stream, so we can check if the GDB is started and its output is on the pipe already. Practically, we do what the listener was supposed to do for the first execution of the GDB server. //below is the original code that must follow after my fix: monitorStream.addListener(new IStreamListener() { @Override public void streamAppended(String text, IStreamMonitor monitor) { if (text.contains("Listening on port")) { //$NON-NLS-1$ synchronized (lock) { gdbServerReady[0] = true; lock.notifyAll(); } monitor.removeListener(this); } } }); Let me know what you think about my observations and how do you propose to solve these bugs. Thanks!
I'm really glad we have an attempt to fix the remote debugging issue but my opinion is that we might need a better approach to determine if the shell prompt is available and ready to interpret commands. Unix shell prompts are configurable and checking within a set of known patterns guarantees your solution will work only in *some* situations. I'm wondering if probing the console readiness with an 'echo <some_control_text>' command wouldn't be better approach for the implementation of waitForShellPrompt method. That's just a suggestion that will work in all Unix environments. The best approach would be to find if the console is ready by querying some objects states but I have no idea if this is possible. Mihai, regarding the second problem reported, I suggest you open a bug in bugzilla as well. This a separate problem than the one affecting remote application debug. Let's handle them separately. Doug/Marc/Simon/Teo, would you please comment on the possible fixes we have for the first problem Mihai tried to fix? Thanks, Adrian
(In reply to Mihai Furis from comment #18) > There are 2 bugs related to launching GDB remotely: I'll answer to both separately. > 1. The two procedures that are executing remote commands > RemoteHelper.execCmdInRemoteShell and > RemoteHelper.remoteShellExec don't wait for shell prompt after the command > is executed. > For this reason new commands try to execute before the previous command has > finished and the system gets blocked because there is no shell prompt > available. To fix this I wrote a new procedure waitForShellPrompt that > should be called before the output stream is flushed. > > private static void waitForShellPrompt(Channel channel, InputStream is) > throws Exception{ > int SIZE = 1024; > byte[] buffer = new byte[SIZE]; > String line=""; > while(true){ > while (is.available() > 0) { > int i = is.read(buffer, 0, 1024); > if (i < 0) > { break; } > line = new String(buffer, 0, i); > line= line.trim(); > // System.out.println(line); > } > if(line.contains("logout")) > { break; } > if (channel.isClosed()) > { break;} > if (!line.isEmpty() && (line.endsWith("#") || > line.endsWith(">") || > line.endsWith("$") || > line.endsWith(":"))) > {break;} > try > { Thread.sleep(500); } > catch (InterruptedException e) > { e.printStackTrace(); } > } > } > > insert a call to the above procedure before the output stream is flushed as > below: > > IRemoteCommandShellService shellService = > conn.getService(IRemoteCommandShellService.class); > IRemoteProcess remoteProcess = null; > Process p = null; > > try { remoteProcess = > shellService.getCommandShell(IRemoteProcessBuilder.ALLOCATE_PTY); > p = new RemoteProcessAdapter(remoteProcess); > InputStream is =p.getInputStream(); > OutputStream os = remoteProcess.getOutputStream(); > os.write(remoteCommand.getBytes()); > waitForShellPrompt(((JSchProcessBuilder) > remoteProcess.getProcessBuilder()).getChannel(), is); > os.flush(); Note that you'd probably want to flush the output stream before waiting for the prompt. If for some reason the command that you wrote is buffered locally, we'll end up waiting for the prompt forever. > } catch (IOException e) > { abort(Messages.RemoteRunLaunchDelegate_7, e, > ICDTLaunchConfigurationConstants.ERR_INTERNAL_ERROR); } catch (Exception e){ > e.printStackTrace(); } > monitor.done(); > return p; > } > It works for me but I do not know if it is the best solution as various > machines can have various shell prompts and I check only for #, $, :, > > prompts. This procedure should be called from both remoteShellExec and > execCmdInRemoteShell. As you and Adrian have mentioned, this would probably not be acceptable, since the prompts can vary. Also, some commands can have a trailing (#, $, :, >) in their output, so you'd get a false positive. I think the only safe way is to have the remote shell exit once the command execution is complete, and have CDT wait for the shell to exit. The first part could be done by either: - appending something like "; exit" to the command (or maybe "; exit $?" if we want the shell to exit with the return value of the command) - shells end when their input stream is closed/reaches EOF, so perhaps we can just close our output stream. Intuitively, I'd say that we could make CDT wait on the remove shell process by calling .waitFor() on the IRemoteProcess object, but I see this comment in IRemoteProcess.java, and I am not sure what to think about it: /** * Wait until the process has terminated * * Note: some implementations (e.g. JSch) will not work correctly if the remote process generates stdout or stderr but the * calling thread does not read the corresponding input or error streams. That's to be investigated. But in theory, it should provide us with a safe way to wait for command completion.
(In reply to Mihai Furis from comment #18) > 2. The second bug is related to the RemoteGdbLaunchDelegate.launch method. > Apparently a listener on the monitoring stream that returns the GDB server > output has been attached to the stream AFTER the command that starts GDB > server has been executed. For this reason when we try to start GDB for the > first time we'll not receive the response from the server because the > LISTENER IS NOT triggered. However when we start the GDB server for the > second time then the listener will be triggered and we'll receive the > response from the server. Just to be sure, are you talking about the listener that we install at RemoveGdbLaunchDelegate.java:150, where it says // Listen process' output to determine gdbserver is up and running. ? I am not sure I understand why things would be different for a second invocation of gdbserver. The same code is executed, and in the second run it's a completely new listener. > I suggest the fix below: > IStreamsProxy streams = iProcess.getStreamsProxy(); > IStreamMonitor monitorStream = streams.getOutputStreamMonitor(); > if (monitorStream.getContents().contains("Listening on port")) { > synchronized (lock) { > gdbServerReady[0] = true; > lock.notifyAll(); > }}; > This code should be inserted before the listener is added to the stream, so > we can check if the GDB is started and its output is on the pipe already. > Practically, we do what the listener was supposed to do for the first > execution of the GDB server. > //below is the original code that must follow after my fix: > monitorStream.addListener(new IStreamListener() { > @Override > public void streamAppended(String text, IStreamMonitor monitor) { > if (text.contains("Listening on port")) { //$NON-NLS-1$ > synchronized (lock) > { gdbServerReady[0] = true; > lock.notifyAll(); } > monitor.removeListener(this); > } > } > }); > > Let me know what you think about my observations and how do you propose to > solve these bugs. Thanks! I don't really understand the change you are proposing. May I suggest you create gerrit changes to show code changes? Or at least patches attached to the bug? It's very difficult to read code in bugzilla comments, since there's no formatting. Thanks! Simon
New Gerrit change created: https://git.eclipse.org/r/100894
Hello, I created the fix and send it for review. Please let me know what you think about it. I need this fix committed on both Oxygen and Neon (all versions). Sincerely, Mihai Furis
