[ptp-dev] RemoteToolsFileStore

[Roland Schulz <roland@xxxxxxx>]

Hi,

Dieter and me are currently working on getting the RemoteToolsFileStore to work correctly.

For this we have so far submitted three patches (bugs: 300889,300891,300623).

The main problem before our patches was that the order of file operations wasn't guaranteed by the RemoteTools. (*)

To fix this problem (and also speeding up the file operations - sftp is faster than exeucting remote shell commands like "cat") we implemented all file operations using sftp.

The one remaining problem we have is

1) Many of the file operations are executed in the "main" thread.  (**)

2) Eclipse sometimes interrupts the "main" thread (e.g. when Display.syncExec is called by a worker thread)

3) Jsch does not gracefully handle being interrupted (***)

As we can't do much about these problems directly, we think we need to wrap the file operations to prevent the interrupt to reach the jsch library.

The only solution we see is the following workaround. It would never execute sftp operations on the main thread to guarantee that those are not interrupted.

This would be done by:

- If a FileTools method is executed on the main thread, it uses a separate thread for the file operation and waits on the completion of that thread.

- During this waiting a wait-dialog could be displayed and could be used to allow the user to cancel.

- The InterruptExecption is ignored for this waiting on the separate thread. After the file operation is finished the thread interrupt status is set.

- For getInputStream/getOutputStream a pair of PipedInputStream/PipedOutputStream is used to guaranteed that the file operation is running in the separate thread and not the main thread. The PipedInputStream is overwritten to guarantee that it itself does not throw a InterruptIOExecption. 

As this is a large workaround just to prevent functions as Display.syncExec to interfere with the sftp communication we would like to ask for ideas of how to better solve this problem.

Thanks

Dieter and Roland

Details:

*) I.e. it was possible that a code using the RemoteTools to write a file and then checking whether that file exists, got the answer that the file does not exist even though the file was written successfully. 

This was caused by the file checking operation (stat) could execute before the file write operation was done.

This in turn was caused by the working of jsch. The jsch channels are independent and since the file writing operation (FileTools.getOutputStream) used the exec channel while the stat operation (FileTools.fetchRemoteAttr) used the sftp channel the order of operation wasn't guaranteed.

**) It should be avoided as much as possible that remote file operations are done on the main thread, since it can freeze the GUI on slow connections. But won't be avoidable in all cases. This is because some file operations are done directly by the eclipse core from the main thread. e.g.:

org.eclipse.ptp.remotetools.internal.ssh.FileTools.getInputStream(FileTools.java:276)

org.eclipse.ptp.remote.remotetools.core.RemoteToolsFileStore.openInputStream(RemoteToolsFileStore.java:322)

org.eclipse.core.internal.localstore.FileSystemResourceManager.read(FileSystemResourceManager.java:713)

org.eclipse.core.internal.resources.SaveManager.restoreMetaInfo(SaveManager.java:796)

org.eclipse.core.internal.resources.SaveManager.restoreMetaInfo(SaveManager.java:776)

org.eclipse.core.internal.resources.SaveManager.restore(SaveManager.java:662)

org.eclipse.core.internal.resources.SaveManager.startup(SaveManager.java:1352)

***) It throws SftpExection and the file is then in a undefined state. We have written an email to the jsch list.

Caused by: 4: 

at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:1751)

at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:1011)

at com.jcraft.jsch.ChannelSftp.get(ChannelSftp.java:989)

at org.eclipse.ptp.remotetools.internal.ssh.FileTools.getInputStream(FileTools.java:284)

... 81 more

Caused by: java.io.InterruptedIOException

at java.io.PipedInputStream.read(PipedInputStream.java:312)

at java.io.PipedInputStream.read(PipedInputStream.java:361)

at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2325)

at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2351)

at com.jcraft.jsch.ChannelSftp._stat(ChannelSftp.java:1732)

--
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309