Community
Participate
Working Groups
The RSE FTP Service is currently single-threaded: directory access as well as any downloads all share a single thread only. This is necessary because Commons Net is single threaded and requires all access be synchronized (see http://wiki.apache.org/jakarta-commons/Net/FrequentlyAskedQuestions) What we could do, however, would be to keep multiple connections open to the remote side: one connection for dir/list commands only, the other connection(s) for data transfer only. The number of connections to try opening could be user customizable. Having two connections by default by default could be simple enough, and have the large benefit of keeping the LIST channel open for important fast operations all the time. The approach of keeping list-channel separate from (multiple) download-channels would be analogous to what we do for SSH. Such implementation would be simple enough, but perhaps produce additional overhead because of doing fresh login for every download; it's also likely that after lots of remote login attempts the server would not allow any more. So, the the overhead of logging in to the remote is higher on FTP, and many FTP servers do not allow too many parallel sessions. Therefore, * additional parallel sessions should be established at connect time and kept open; * the number of parallel sessions should be configurable * a keepalive for the sessions may be needed. A different alternative might be switching to a different FTP client library that supports multiple parallel downloads in itself.
