Community
Participate
Working Groups
This is a change request. I'm using TerminalView and trying to add some functionality that I need for our product. One feature I'm implementing is file transfer using different protocol based on serial connection. In order to not repeat RXTX wrapper code I'd like to get access to ITerminalConnection to us its service. How to do it? Typical use case for that: User needs action to be added to the terminal instance. This action causes transfer IPL or OS image to remote target. It would be nice to use terminal transport services (e.g. InputStream and OutputStream for serial connection) and for setting session parameters to use standard UI that terminal view already provides.
Created attachment 94374 [details] Patch to add requested feature What's done in this patch: - created notification interface to inform registred listeners about change of connection state. I added this functionality to ITerminalViewControl interface and ITerminalView interface (this one works just as a client of ITerminalViewControl notification mechanism. I'd prefer instead of that add this notification feature to ITerminalConnector interface, however it would cause more changes that I dare to present in this patch. - exposed instance of ITerminalConnectorInfo associated with terminal view. I'd prefer to expose ITerminalConnector instead of that, but two obstacles prevented me of doing that: a) cannot get connection protocol ID b) no method to check current state (for ITerminalConnectorInfo I at least can use isInitialized() which is not enough but better than nothing. - exposed InputStream instance for ITerminalConnector in order to support bi-directional transfer protocol This changes let me implement data transfer framework which use terminal communication servises to implement various data transmit protocols based on serial communication layer.
Why does ITerminalView extends IViewPart? I think adding ITerminalConnector.getInputStream is dangerous, because there is typically a thread in th connector that reads the input stream and writes the bytes to the terminal. I wonder how you use this in your extension... There are probably two threads trying to read from this input stream, this sounds like causing trouble....
(In reply to comment #2) > Why does ITerminalView extends IViewPart? I could live without that, it happened historically. As for now the reason I would keep it on this way is that it is more convenient in ...ActionDelegate implementation: it lets me to avoid either casting in hacker's manner or implementation of an adapter method in TerminalView. On the other hand it seemed to me harmless, taking in account that the only implementation of it extends ViewPart. However, I'm ready to revoke this change as not principal one. > > I think adding ITerminalConnector.getInputStream is dangerous, because there is > typically a thread in th connector that reads the input stream and writes the > bytes to the terminal. I wonder how you use this in your extension... There are > probably two threads trying to read from this input stream, this sounds like > causing trouble.... > Actually I use inputstream in implementation of transfer protocol in order to get flow control packets from the target. In this case client (i.e. transfer protocol implementation) has to completely control input, and terminal itself is not expected to receive this data and output them to display. Maybe, to guarantee full control over the input stream it would be better to synchronize its usage It is easy to do on the server side, terminal transport service. It should be commented in the interface definition that InputStream must be synchronized during the period when client expects receiving data.
I think what you need for the input stream is described in bug 165893 comment #19
I fully agree: It's not sufficient to just hijack a terminal's InputStream. What we need is a Decorator to intercept both InputStream and OutputStream. If that's not done, a user could destroy an ongoing file transmission by typing some random characters in the terminal while the transmission is ongoing, right? However the use-case on this bug is slightly simpler than bug 165893 because here, we're not concerned of contributing arbitrary decorators by extension point; we rather use contributed menus/actions which use API to operate on the current connector of the current Terminal. That simplifies the Ordering problem, because via API we're not concerned about the order of a decorator stack (first come first serve). We might also not be concerned about stacking multiple stream decorators yet, although this looks like a nice generalization if we can do it. I also checked the console framework, but it doesn't seem to support decorators; however they have IOConsole with IOConsoleInputStream and IOConsoleOutputStream, supporting multiple outputs and a single input on one console. The special streams basically add some buffering but I think that our PipedInputStream is actually better. http://help.eclipse.org/help33/topic/org.eclipse.platform.doc.isv/reference/api/org/eclipse/ui/console/IOConsole.html
Actually, looking at the proposed use case from bug 185348 / attachment 94254 [details] again: it seems to me that you're really only interested in getting your own channel connected for transmission, independent of a Terminal. Would it help if you instantiated your own (serial) connector? And reuse the TerminalSettingsDlg to configure your connection? Along the lines of bug 185348 comment 5 and http://dev.eclipse.org/newslists/news.eclipse.dsdp.tm/msg00229.html
(In reply to comment #6) > Actually, looking at the proposed use case from bug 185348 / attachment 94254 [details] > again: it seems to me that you're really only interested in getting your own > channel connected for transmission, independent of a Terminal. > > Would it help if you instantiated your own (serial) connector? And reuse the > TerminalSettingsDlg to configure your connection? > > Along the lines of bug 185348 comment 5 and > http://dev.eclipse.org/newslists/news.eclipse.dsdp.tm/msg00229.html Not at all. The process of loading image could be multi-step and includes several times interaction with target via terminal. And these operations are never overlapped with data flow. What do you think if we add just releaseInputStream() together with getInputStream(). If client gets ownership over input stream, terminal cannot use it any more until client does not release it. I think that the same is valid for outputStream. This is a sort of a simple synchronization, and it doesn't seem to require a big changes.
(In reply to comment #6) > Actually, looking at the proposed use case from bug 185348 / attachment 94254 [details] > again: it seems to me that you're really only interested in getting your own > channel connected for transmission, independent of a Terminal. > > Would it help if you instantiated your own (serial) connector? And reuse the > TerminalSettingsDlg to configure your connection? > > Along the lines of bug 185348 comment 5 and > http://dev.eclipse.org/newslists/news.eclipse.dsdp.tm/msg00229.html > Moreover, during the overall process terminal and client share one and the same physical channel, so each of them must get ultimate access to this resource.
...that is why a decorator is a good approach: your decorator (FilterOutputStream) would see all the communication and it could then grab the data it needs and write it to a file. There are two directions of communication: remoteToTerminal and terminalToRemote. Both are represented by an output stream: * remoteToTerminal is used to write to the terminal (when the remote site wants to display a string in the terminal, this channel is used): ITerminalControl.getRemoteToTerminalOutputStream() * terminalToRemote is used to write to the remote site (if a key is pressed the key is sent to the remote site on this channel: ITerminalConnector.getOutputStream() I assume there are two scenarios for a file transfer: 1. the remote site sends a file to the terminal and 2. the terminal sends a file to the remote site. I also assume that there is a special maker in the stream (escape sequence) to signal the beginning and the end of a file transfer. Let's look at both cases (for simplicity, let's assume the special marker is a single char MARKER_CHAR): 1. remote to terminal: the remote site wants to send a file. it sends the begin file marker. Your write method in your FilterOutputStream would look like this: public class RemoteToTerminalFilterOutputStream extends FilterOutputStream { private static final int MARKER_CHAR = 1; public RemoteToTerminalFilterOutputStream(OutputStream out) { super(out); } boolean inFileTransfer; OutputStream fileStream; public void write(int b) throws IOException { // the marker begins and ends th file transfer. // in real life, the marker would be a multi-byte // sequence (plus a way to "escape" the marker)). // And there might be some more markers in the transfer // mode to denote the filename etc. if(b==MARKER_CHAR) { inFileTransfer=!inFileTransfer; if(inFileTransfer) { // begin the transfer fileStream=askUserForOutputFile(); } else { // transfer is done fileStream.close(); } } else { if(inFileTransfer) // write to the file if we are in the transfer mode fileStream.write(b); else super.write(b); } } private OutputStream askUserForOutputFile() { return ... } } 2. terminal to remote (you send a file): public class TerminalToRemoteFilterOutputStream extends FilterOutputStream { private static final byte MARKER_CHAR = 1; public TerminalToRemoteFilterOutputStream(OutputStream out) { super(out); } private boolean inFileTransfer; public void sendFile(InputStream in) throws IOException { // during the transfer do not allow writing to the stream synchronized(this) { inFileTransfer=true; } try { // use the decorated output stream directly // In real life, the marker would be more than // one char and there should be an escape sequence // to "quote" the marker.... out.write(MARKER_CHAR); copyStream(in,out); out.write(MARKER_CHAR); } finally { // make sure to end the transfer mode synchronized(this) { inFileTransfer=false; } } } private void copyStream(InputStream is, OutputStream os) throws IOException { byte bytes[]=new byte[1024]; int n; // read until the thread gets interrupted.... while( (n=is.read(bytes))!=-1) { os.write(bytes,0,n); } } synchronized public void write(int b) throws IOException { // ignore input from the terminal during file transfer // it could also be buffered and sent to the terminal // after the file transfer had finished if(inFileTransfer) return; super.write(b); } } The remaining question is, how to decorate the two streams. Alex, before I go into this, I want to know if this is what you are trying to do.....
Here the two files with better formatting: public class RemoteToTerminalFilterOutputStream extends FilterOutputStream { private static final int MARKER_CHAR = 1; public RemoteToTerminalFilterOutputStream(OutputStream out) { super(out); } boolean inFileTransfer; OutputStream fileStream; public void write(int b) throws IOException { // the marker begins and ends th file transfer. // in real life, the marker would be a multi-byte // sequence (plus a way to "escape" the marker)). // And there might be some more markers in the transfer // mode to denote the filename etc. if(b==MARKER_CHAR) { inFileTransfer=!inFileTransfer; if(inFileTransfer) { // begin the transfer fileStream=askUserForOutputFile(); } else { // transfer is done fileStream.close(); } } else { if(inFileTransfer) // write to the file if we are in the transfer mode fileStream.write(b); else super.write(b); } } private OutputStream askUserForOutputFile() { return ...; } } public class TerminalToRemoteFilterOutputStream extends FilterOutputStream { private static final byte MARKER_CHAR = 1; public TerminalToRemoteFilterOutputStream(OutputStream out) { super(out); } private boolean inFileTransfer; public void sendFile(InputStream in) throws IOException { // during the transfer do not allow writing to the stream synchronized(this) { inFileTransfer=true; } try { // use the decorated output stream directly // In real life, the marker would be more than // one char and there should be an escape sequence // to "quote" the marker.... out.write(MARKER_CHAR); copyStream(in,out); out.write(MARKER_CHAR); } finally { // make sure to end the transfer mode synchronized(this) { inFileTransfer=false; } } } private void copyStream(InputStream is, OutputStream os) throws IOException { byte bytes[]=new byte[1024]; int n; // read until the thread gets interrupted.... while( (n=is.read(bytes))!=-1) { os.write(bytes,0,n); } } synchronized public void write(int b) throws IOException { // ignore input from the terminal during file transfer // it could also be buffered and sent to the terminal // after the file transfer had finished if(inFileTransfer) return; super.write(b); } }
I'd concentrate on the second case (terminal to remote) because the first one is out of my scope. If you ask me to specify my task in a couple of phrases, it would be sound like that: - Client gets full control on the input channel receiving ALL remote target->terminal traffic, target itself can or cannot send escaped control data (such as ACK/NAK for example or whatever ) like it takes place in simple "copy file comN" case. - Client gets full control on the output channel - terminal "thinks" that it still controls input and output channel, but these are just proxy i/o streams that client provides as a replacement. Client can decide which part of incoming trafic should be displayed on the terminal. Client should be able to disable/enable user's keyboard input (maybe by sending ESC sequences). It can also generate some extra stuff on the terminal such as user request about error processing. What do you think about that approach? Is it flexible, too flexible, just silly or what? Sorry, it is not direct answer to you question, I just would like to find common ground before switching to implementation issues.
What do you mean by "client". Is the (ab)using the terminal connection for the file transfer? You also have to understand that there is no InputStream in the current design. There are two output streams. One from the remote site to the target and one from the target to the remote site. If the ITerminalConnector would provide an InputStream, then the terminal would be responsible to create a thread that reads from the input stream. This might cause some problems with some connectors, because only the connector knows which thread(s) can use the input stream. In addition, the connector knows better how to handle errors on the input stream. If the terminal would read from the input stream, it would be very hard to deal with errors (e.g, closing the connection and opening a new one). Therefore each connector (often in a class called Connection) has something like a loop that reads from the input stream and writes it to the ITerminalControl.getRemoteToTerminalOutputStream(): private void readDataForever(InputStream in) throws IOException { // read the data byte bytes[]=new byte[8*1024]; int n; while( (n=in.read(bytes))!=-1) { fControl.getRemoteToTerminalOutputStream().write(bytes,0,n); } } I also don't understand how you want to replace the streams. I think it is quite difficult to replace a stream while the communication is already going on. That's why I propose the decorator approach. But maybe you can explain to me how you would replace the stream while the communication is going on....
> What do you mean by "client". Is the (ab)using the terminal connection for the file transfer? I meant: Is the "cleint" (ab)using the terminal connection for the file transfer?
This is a component which communicates on some way with remote target using terminal communication ad user interaction abilities. It's the client of terminal service.
(In reply to comment #12) > You also have to understand that there is no InputStream in the current design. > There are two output streams. One from the remote site to the target and one > from the target to the remote site. > OK, I understand that. This is a connector who has input stream. That's good as well :-) > I also don't understand how you want to replace the streams. I think it is > quite difficult to replace a stream while the communication is already going > on. That's why I propose the decorator approach. But maybe you can explain to > me how you would replace the stream while the communication is going on.... > Maybe I don't understand something, but is it a problem? For terminal it is just switching a buffer, isn't it? The most important thing is to control which data goes to terminal, and which doesn't, doesn't matter how it is implemented. Please correct me if I'm wrong. Anyway, let it be decorated stream. The purpose of my previous message was to explain what features I need to have. If you think it is feasible (I mean that at the end of the story I'll get what I need) , go ahead with this implementation. If you need any help, I'd be happy to participate in this work.
(In reply to comment #12) I would like to to give an illustration of my verbal description. It is just a raw not polished fragments of code. I'm talking now only about seriall connection. I add two methods to ITerminalConnector interface: /** * Depends on the connector implementation. Should be invoked * each time user does not need to grab input from the remote * target. */ void releaseInputStream(); /** * @return a stream used to write to the terminal. Any bytes written to this * stream appear in the terminal or are interpreted by the emulator as * control sequences. Returns null if input is not grabbed. */ OutputStream getTerminalOutputStream(); Small changes in TerminalConnectorExtension: public OutputStream getTerminalOutputStream() { return getConnector().getTerminalOutputStream(); } public void releaseInputStream() { getConnector().releaseInputStream(); } Their implementation in SerialConnector: private boolean fInputGrabbed = false; public InputStream getInputStream() { fInputGrabbed = true; return fInputStream; } public void releaseInputStream() { fInputGrabbed = false; } /* this method is not a part of public interface */ boolean isInputGrabbed() { return fInputGrabbed; } public OutputStream getTerminalOutputStream() { if(isInputGrabbed()) return fControl.getRemoteToTerminalOutputStream(); return null; } Change a bit serialEvent method in SerialPortHandler public void serialEvent(SerialPortEvent event) { switch (event.getEventType()) { case SerialPortEvent.DATA_AVAILABLE: if(!fConn.isInputGrabbed()) onSerialDataAvailable(null); break; } } Now what program that use this service does: public void setTransport(ITerminalConnector transport) { ostream = transport.getOutputStream(); controlData = transport.getInputStream(); }
Sorry, my message was interrupted accidentally. Below is the end of the message. Now what program that use this service does: private OutputStream ostream; // To remote target private InputStream controlFlow; // From remote target private OutputStream termOutputStream; public void setTransport(ITerminalConnector transport) { ostream = transport.getOutputStream(); controlFlow = transport.getInputStream(); termOutputStream = transport.getTerminalOutputStream(); } ...... void processBlock(byte[] buffer) { writeControlDataPerBlock(buffer); ostream.write(buffer, 0, buffer.length); ..... for(int code = controlFlow.read(); code >= 0; ) { switch (code) { case ABORT_CKSUM : // Process error break; case ABORT_SEQ : // Process error break; case ABORT_PROTOCOL : // Process error break; default: // This is not a part of protocol exchange - // send it to the terminal termOutputString.write(code); } } ... // Don't forget to release input stream: } finally { ... transport.releaseInputStream(); ... } Admittedly, I didn't try it yet, but it seems to me that at least for serial connection it should work ok. I'd like to know your opinion.
Alex, I think I know what you need and I am working on a solution.....
Alex, there are many ways a file transfer could work on a shared connection. Lets assume the file we want to send has "0123456789" as content. And the user is typing "abcdefgh". And let's assume the transfer starts after the user has typed 'b' and during the transfer he's typing "cde" and after the transfer "fgh". Here are different solutions I can think of: 1. The file is send in one big junk use '[' to begin the transfer and ']' to end the transfer (it could also be a begin marker with a size). 1.a. The bytes sent during the trensfer are dropped: "ab[0123456789]fgh" 1.b. The bytes sent during the transfer are delayed: "ab[0123456789]cdefgh" 2. The file is sent in chunks and the bytes send during the transfer are send in between: "ab[0123]cd[4567]e[89]fgh" 2.a The intercepting bytes ("cde") are sent as soon as they arrive. If there are no intercepting bytes the file is send as one long string. 2.b The transfer is always chunked [0123][4567][89] In a more complex world, there could be multiple file transfers going on in parallel, where each chunk has a header telling which file it belongs to. Which leads to the most general design, where multiple streams are multiplexed in chunks over one channel... And for the controlFlow: there could be other characters coming on the control flow stream. So you's have to filter out your specific control characters and the other characters have to flow to the receiver. It is essentially the same design but this time you remove the chunks that belong to you from the stream.... Does this somehow describe what you need?
(In reply to comment #19) > Alex, there are many ways a file transfer could work on a shared connection. > Lets assume the file we want to send has "0123456789" as content. And the user > is typing "abcdefgh". And let's assume the transfer starts after the user has > typed 'b' and during the transfer he's typing "cde" and after the transfer > "fgh". Here are different solutions I can think of: > > 1. The file is send in one big junk use '[' to begin the transfer and ']' to > end the transfer (it could also be a begin marker with a size). > 1.a. The bytes sent during the trensfer are dropped: "ab[0123456789]fgh" > 1.b. The bytes sent during the transfer are delayed: "ab[0123456789]cdefgh" > 2. The file is sent in chunks and the bytes send during the transfer are > send in between: "ab[0123]cd[4567]e[89]fgh" > 2.a The intercepting bytes ("cde") are sent as soon as they arrive. If there > are no intercepting bytes the file is send as one long string. > 2.b The transfer is always chunked [0123][4567][89] > > In a more complex world, there could be multiple file transfers going on in > parallel, where each chunk has a header telling which file it belongs to. Which > leads to the most general design, where multiple streams are multiplexed in > chunks over one channel... > > And for the controlFlow: there could be other characters coming on the control > flow stream. So you's have to filter out your specific control characters and > the other characters have to flow to the receiver. It is essentially the same > design but this time you remove the chunks that belong to you from the > stream.... > > Does this somehow describe what you need? > I follow standard multi-tier level. Each tier knows what belongs to it, then all the rest follows to upper level (if control data that tier received doesnt say other). Next level does the same. So on each level control flow contains only not stripped on the lower tiers stuff. For example, your application that controls serial port doesn't receive control bytes that serial drivers exchange to each other. I think that to say that there is a great variety of transfer protocols based on RS232 would be a bit exagerration because this is very old and stable communication environment, and one could call barely more than half a dozen of different transfer methods. So I agree with your last paragraph: this is what I'm talking about.
Michael, Martin, I just finished testing together with our tester my patch. (I updated the previous one after the discussion in this bug and fixed synchronization issues for Remote-to-local stream), and it looks like it works fine now (admittedly, we tested it on Windows platform only so far). I just was wondering about your plans on this issue. Would you are still planning to fix it soon, I appreciate it and can wait for the generic solution. If this is not in your closest plans and you are somehow curious to look at this solution, I could once again to create a patch and send it to you. I just wouldn't do this work if it is against your plans or interests, because you have been changing terminal stuff quite intensively last week, and it would takes a wile to create new patch. "No" is also OK:-)
We definitely want to bring the "contributed services over terminal connections" idea into TM 3.0. In fact, some work-in-progress has just been checked into the org.eclipse.tm.terminal.tests plugin. The difference between our work and your patch is, that while * In your case, it's totally up to the clients who writes on the stream and when, so the resulting stream is timing dependent (which may be fine in most cases). * In our case, we're trying to make the process of merking two streams into one, or splitting one stream up into two, configurable. That is, our StreamInterceptor class allows the contributor to provide an algorithm which selects who's writing to the stream and when. Some example trivial implementations of the Interceptor are: - SerializingInterceptor: Write from Stream A until it's closed, then append everything from Stream B (essentiall B is blocked while A is busy writing). - ImmediateInterceptor: Write bytes from Stream A or Stream B onto the output as they come along. That's essentially what you are doing. It's similar for the other way round, demultiplexing a single Stream onto two outputs. Here, the trivial implementatio is - DuplicatingInterceptor: Every byte from IN is copied onto both A and B. Some implementation is still missing, and we'll happily accept your feedback about the idea, as well as what we currently have. One part of the work in progress is rationalizing the synchronization / threading story behind the multiplexers.
Thank you Martin, sounds good for me. It looks more generic and shouldn't add much overhead. I'm waiting for implementation.
tentatively planned for M5 to allow some API review.
Moving deferred 3.3 api items to 3.4
Hi Michael, I want to upload a binary file to the device connect through serial port. When the data is sent, the device will send success/failure message for every packet. I have to interrupt with the data received. Is there any work around/patch before 3.4 release? It is very urgent requirement. I have also rised a Topic in the forum. http://www.eclipse.org/forums/index.php/m/768501/#msg_768501
If it's urgent I'm afraid you'll have to work on the feature yourself, or use an external terminal program that supports xyzmodem. As of today the assignment to 3.4 is tentative and we can't guarantee getting this in.
The Terminal component of the Eclipse Ecosystem has a new home. The Terminal is now part of the Eclipse CDT project[1]. This change means a new Git repo[2], P2 site[3] and Bugzilla component. The terminal will continue to be delivered as part of the quarterly Simultaneous Release of Eclipse as well. The marketplace entry[4] had not been updated in a few years. It will once again install the latest release of the terminal on the latest release of the whole IDE (currently 2020-03). If this bug is no longer relevant, please feel free to comment or close the bug. If you can confirm if this issues still occurs in the latest release please do let me know in a comment. [1] https://wiki.eclipse.org/CDT/User/NewIn911 [2] https://git.eclipse.org/c/cdt/org.eclipse.cdt.git (in the terminal directory) [3] current release is 9.11 - P2 site https://download.eclipse.org/tools/cdt/releases/9.11/ [4] https://marketplace.eclipse.org/content/tm-terminal (This comment was added to all open terminal bugs along with changing the Product/component pair to CDT/terminal.)
