Re: [jetty-dev] Reactive Streams

[Greg Wilkins <gregw@xxxxxxxxxxx>]

On 2 June 2015 at 03:08, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

On Mon, Jun 1, 2015 at 2:48 PM, Greg Wilkins <gregw@xxxxxxxxxxx> wrote:
> On 1 June 2015 at 18:21, Simone Bordet <sbordet@xxxxxxxxxxx> wrote:

Using ByteBuffer is an implementation detail.

hmmmm.... I think it is more than that.  ByteBuffers vs byte[] is a big difference if you are going to allow for partial consumption in onNext()!   You can partially consume a ByteBuffer, but you can't partially consume a byte[]!

 

> For a start, it would not work with byte[] or String as there
> are no mutable pointers that can communicate how much has been consumed.

That is true, but the APIs give you back that information for bytes,
which is what you will eventually end up to if you work with sockets.

But even imagining a non-socket system that consumes strings, you
already have those Strings allocated in memory.
So you should not worry about memory, and how to backpressure parts of
a huge String, when you have already all of it in memory.

Sure you have the String in memory, but that does not mean that you have the memory available to convert that string into something else in a single chunk.

Consider if you have a huge binary file that has been compressed, B64 encoded and then sent over email.   The publisher gets the email so it has the entire B64 string in memory and can deliver it to the chain of Processors.  There is a B64 decoder processor (approx doubles memory) and a decompressing processor (can multiple memory arbitrarily), so the memory commitment for processing a single string is unbounded and the processing by the ultimate subscriber may take time.   Thus those intermediate Processers are going to have to block or consume unbounded memory.

 

Bottom line for me is this: you can backpressure easily with this
model, but not with a single byte precision. But you can never have
single byte precision, so there is no point in wanting that.
You can easily implement a proxy with proper backpressure, which in my
experience is the toughest use case to get right.

Still can't see that.   Can you implement my FragmentingProcesser for Strings without blocking or doubling memory commitment?

--

Greg Wilkins <gregw@xxxxxxxxxxx>  - an Intalio.com subsidiary
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.