Re: [jetty-dev] Reactive Streams

[Simone Bordet <sbordet@xxxxxxxxxxx>]

Hi,

On Tue, Jun 2, 2015 at 4:16 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:
>
>
> On 2 June 2015 at 03:08, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:
>>
>> On Mon, Jun 1, 2015 at 2:48 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:
>> > On 1 June 2015 at 18:21, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:
>>
>> Using ByteBuffer is an implementation detail.
>
>
> hmmmm.... I think it is more than that.  ByteBuffers vs byte[] is a big
> difference if you are going to allow for partial consumption in onNext()!
> You can partially consume a ByteBuffer, but you can't partially consume a
> byte[]!

You don't pass a byte[] if that alone does not convey the semantic or
the implementation needs you want.

> Sure you have the String in memory, but that does not mean that you have the
> memory available to convert that string into something else in a single
> chunk.
>
> Consider if you have a huge binary file that has been compressed, B64
> encoded and then sent over email.   The publisher gets the email so it has
> the entire B64 string in memory and can deliver it to the chain of
> Processors.  There is a B64 decoder processor (approx doubles memory) and a
> decompressing processor (can multiple memory arbitrarily), so the memory
> commitment for processing a single string is unbounded and the processing by
> the ultimate subscriber may take time.   Thus those intermediate Processers
> are going to have to block or consume unbounded memory.

Not really.

They can consume the string a chunk at a time, B64 decode, uncompress,
store the bytes into a ByteBuffer, and pass that ByteBuffer to the
subscriber, if there is demand.

It's a 1-to-N transformation in the number of items: from 1 String to
N ByteBuffers.
No need to block.

> Still can't see that.   Can you implement my FragmentingProcesser for
> Strings without blocking or doubling memory commitment?

I'll give it a go.

-- 
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.