[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [ecf-dev] E-intro [Was Efficient downloads]

Hi Scott,

thanks for your quick reply.

We need a very robust file transfer implementation for our application. I believe that if we cooperate on this we'll create something really good.

Purpose:
- have downloads as fast as possible (optimize downloads for case that the download capacity is bigger than single location's upload capacity)
- have downloads reliable (download everything or nothing, avoid saving corrupted files if possible)
- have downloads convenient (automatic recovery, discovery of alternatives)
- let downloads have good UI (verbose enough, good cancel/pause/resume response)


Let me list features we are looking for (just a brainstorming, no specification yet):
- multiple protocol support (http/https/ftp/file...)
- pause/resume using partial content requests (connection can be closed/reopened before an attempt to resume)
- resume from a different location (e.g. different mirror)
- multi threaded download of one artifact from one location (starting at different content offsets)
- multi threaded download of one artifact from several locations at the same time
- multi threaded download of several artifacts at the same time (using either single threaded or multi threaded single artifact downloads)
- retrieving information from special headers (like Content-Disposition)
- detecting URL redirections to final mirrors
- detecting approximate user's geographical position
- collecting download statistics (slow/unreliable mirror discovery)
- dynamic switching between mirrors in case of slow download
- automatic recovery after connection timeout
- good API for verbose progress bars (ability to create a good UI)
- good API for statistics processing
- configurable behavior (enable configuring features described above)
- ...and perhaps many more (?)


I don't know how much from this you already cover. I'll study what you currently provide first. All the features don't have to be implemented at the same time, we can define development stages to start with more important issues and continue with less important ones.

Best regards
 Filip Hrbek

Note: I'll be offline from this Friday until next Sunday, so don't worry if I don't answer your emails during next week. I'll be definitely back from Monday (11th June).



Scott Lewis napsal(a):
Hi Thomas and Filip,

Thanks! The help is most welcome.
The filetransfer API is in plugin: org.eclipse.ecf.filetransfer.
The bundle docs (with pointers to javadocs) is: http://wiki.eclipse.org/index.php/ECF_API_Docs#File_Transfer_API


There are also pointers in wiki to project sets source and test code.

There is a base provider plugin: org.eclipse.ecf.provider.filetransfer. This plugin has an extension point that allows other plugins to insert implementations associated with various protocols. See the extension point org.eclipse.ecf.provider.filetransfer.fileTransferProtocolFactory. The httpclient implementation uses this extension point (and is defined in another plugin: org.eclipse.ecf.provider.filetransfer.http).

The basic implementation that uses jobs for doing pause/resume (IFileTransferPausable) is in org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer. This can/could be used (and appropriate methods overidden) or replaced...up to you as to what makes sense.

Filip please let me and everyone know if you have any questions/comments/suggested or needed changes...and thanksinadvance! I would/will be curious myself about what http/https impl you are looking to use and what specific http/https protocol features you will be using as well.

Scott

Thomas Hallgren wrote:
Hi Scott,
We, (the Buckminster team) would like to offer our help in implementing protocol-level pause/resume and recovery capabilities for the ECF http/https file transfer. I'd like you to meet Filip Hrbek. He is the Buckminster guy who currently works on improving our materialization. He will be our main resource for this, should you accept our help. So Filip, meet Scott. Scott meet Filip.


Regards,
Thomas Hallgren



Scott Lewis wrote:
Hi Thomas,

Thomas Hallgren wrote:
Hi Scott,
Thanks for a very elaborate answer. This looks very promising. Well structured, low overhead. Only thing missing if I read you correctly is true pause/resume on protocol level (and the failure recovery that I think will come as a bonus). Do you have any sense what it would take to implement that?

It depends upon each protocol and it's corresponding support in (some) protocol library. I assume you are most interested in http/https. I'll take a look at the httpclient 3.0.1 support for pause/resume and failure recovery.


Scott


_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev

_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev

_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev