[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [equinox-dev] [prov] processing steps, restartable downloads and ECF

Scott Lewis wrote on 09/10/2007 12:58:10 PM:

> Actually, I'm a little surprised that you have so far passed the
> ProcessingSteps as output streams directly to the ECF OutputStream,
> as I was expecting that you would have a temporary file to receive
> the file contents, and then when the file reception is done *then*
> apply the ProcessingSteps.  

Is there something wrong with passing the data along immediate?  I guess it would be a trade off.  Some steps only work on whole files while others can be streamed.  Streaming is more effecient since you have the bytes right there.  there may be some buffering (e.g., performance) but there are several opportunities for optimization.  We could also add a step at the beginning of a chain that buffers up the data in a file and then pumps it through.  This sort of step could be used to do the restarting but if ECF already had (or could have) something, that seems like more fun.

> But in any event, we can add impl support for pause/resume/caching
> etc to the ECF receive implementations w/o changing API to support
> required use cases.  I would appreciate a little better
> understanding of the existing ProcessingSteps and their function...
> so could someone point me at the relevant packages/classes and I'll
> take more of a look?

The current code is in my workspace as I have been doing some significant reworking and have not finished.  I will post to this list when it is released.

> Seems like this would also be a good topic for the upcoming Equinox
> Summit:  what enhancements are needed for file transfer both at API
> and impl:  e.g. pause/resume enhancements, file caching,
> monitoring/transfer statistics collection?, support more/other providers, etc.

yes.  I added this to the list.  Under your name :-)