[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [ecf-dev] E-intro [Was Efficient downloads]

Hi Filip,

A couple of comments interspersed.

Filip Hrbek wrote:
Hi Scott,

thanks for your quick reply.

We need a very robust file transfer implementation for our application. I believe that if we cooperate on this we'll create something really good.

Purpose:
- have downloads as fast as possible (optimize downloads for case that the download capacity is bigger than single location's upload capacity)
- have downloads reliable (download everything or nothing, avoid saving corrupted files if possible)
- have downloads convenient (automatic recovery, discovery of alternatives)
- let downloads have good UI (verbose enough, good cancel/pause/resume response)


Let me list features we are looking for (just a brainstorming, no specification yet):
- multiple protocol support (http/https/ftp/file...)

This is what ECF is fundamentally about, so the filetransfer API should help with this.


- pause/resume using partial content requests (connection can be closed/reopened before an attempt to resume)

This will be up to the implementation to provide, of course. I *think* that the IFileTransferPausable as currently defined gives sufficient semantics for what you want to do here, but if not please let us know.


- resume from a different location (e.g. different mirror)

Hmm. Don't know how you are going to accomplish that without something quite different from normal http, but sounds interesting.


- multi threaded download of one artifact from one location (starting at different content offsets)
- multi threaded download of one artifact from several locations at the same time
- multi threaded download of several artifacts at the same time (using either single threaded or multi threaded single artifact downloads)

ECF's structure is based upon asynchronous download...i.e. each request has a job/thread associated with it that is responsible for asynchronously sending back events (data received, failed, completed, etc) to a listener (IFileTransferListener). So this shouldn't be a problem as it's built into the fabric of ECF.


- retrieving information from special headers (like Content-Disposition)
- detecting URL redirections to final mirrors

I'm not sure what you are going to use to implement this, but would be curious to find out.


- detecting approximate user's geographical position
- collecting download statistics (slow/unreliable mirror discovery)
- dynamic switching between mirrors in case of slow download
- automatic recovery after connection timeout

I don't think these have any API implications (except perhaps the collecting download stats), but if I'm wrong about that please let me know.


- good API for verbose progress bars (ability to create a good UI)

The asynchronous structure of ECF's filetransfer API should be well-suited to this, as progress bars can simply be updated in listener code...


- good API for statistics processing

I think you would need to describe what statistics are desired here. We can easily add adapter interfaces for collecting statistics associated with a given file retrieval/all to ecf or individual providers, but would need to know what stats are of interest.


- configurable behavior (enable configuring features described above)
- ...and perhaps many more (?)

I don't know how much from this you already cover. I'll study what you currently provide first. All the features don't have to be implemented at the same time, we can define development stages to start with more important issues and continue with less important ones.

Yes, I agree that's certainly the right approach. Please feel free to ask any questions as you study what's there.


Thanks,

Scott


Best regards Filip Hrbek

Note: I'll be offline from this Friday until next Sunday, so don't worry if I don't answer your emails during next week. I'll be definitely back from Monday (11th June).



Scott Lewis napsal(a):
Hi Thomas and Filip,

Thanks! The help is most welcome.
The filetransfer API is in plugin: org.eclipse.ecf.filetransfer.
The bundle docs (with pointers to javadocs) is: http://wiki.eclipse.org/index.php/ECF_API_Docs#File_Transfer_API


There are also pointers in wiki to project sets source and test code.

There is a base provider plugin: org.eclipse.ecf.provider.filetransfer. This plugin has an extension point that allows other plugins to insert implementations associated with various protocols. See the extension point org.eclipse.ecf.provider.filetransfer.fileTransferProtocolFactory. The httpclient implementation uses this extension point (and is defined in another plugin: org.eclipse.ecf.provider.filetransfer.http).

The basic implementation that uses jobs for doing pause/resume (IFileTransferPausable) is in org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer. This can/could be used (and appropriate methods overidden) or replaced...up to you as to what makes sense.

Filip please let me and everyone know if you have any questions/comments/suggested or needed changes...and thanksinadvance! I would/will be curious myself about what http/https impl you are looking to use and what specific http/https protocol features you will be using as well.

Scott

Thomas Hallgren wrote:
Hi Scott,
We, (the Buckminster team) would like to offer our help in implementing protocol-level pause/resume and recovery capabilities for the ECF http/https file transfer. I'd like you to meet Filip Hrbek. He is the Buckminster guy who currently works on improving our materialization. He will be our main resource for this, should you accept our help. So Filip, meet Scott. Scott meet Filip.


Regards,
Thomas Hallgren



Scott Lewis wrote:
Hi Thomas,

Thomas Hallgren wrote:
Hi Scott,
Thanks for a very elaborate answer. This looks very promising. Well structured, low overhead. Only thing missing if I read you correctly is true pause/resume on protocol level (and the failure recovery that I think will come as a bonus). Do you have any sense what it would take to implement that?

It depends upon each protocol and it's corresponding support in (some) protocol library. I assume you are most interested in http/https. I'll take a look at the httpclient 3.0.1 support for pause/resume and failure recovery.


Scott


_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev

_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev

_______________________________________________ ecf-dev mailing list ecf-dev@xxxxxxxxxxx https://dev.eclipse.org/mailman/listinfo/ecf-dev
_______________________________________________
ecf-dev mailing list
ecf-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/ecf-dev