Re: [mosquitto-dev] [RSMB] Strange memory behavior

[Mohamed HAMZAOUI <requinham@xxxxxxxxx>]

Paul,

I don't totally agree with you. When I talk about marshalling, I don't
refer to splitting message or marshalling it in applicative layer.
Marshalling is a normal step that will be done by down layer. My
remark is about how applicative layer (Eclipse Mosquitto) tell the
down layer to send a specific message. In current case, I think the
broker internally copy payload and create a "real" new instance in
memory for every subscriber. My proposition is to create many copy but
keep just one in memory like implicit sharing mechanism (also called
shadowing). Broker don't need to hard copy non writable instance of a
data.

Obviously, this proposition don't concern only big message but all message.

Regards,
Mohamed

On Tue, Oct 18, 2016 at 2:55 PM, Paul Fremantle
<paul.fremantle@xxxxxxxxxx> wrote:
> Mohamed
>
> In general, wouldn't it make sense to split up your message and re-assemble
> it later?
>
> There are a lot of systems that support efficient distribution of large
> messages in constant memory, but in general its very hard to combine the
> following three things:
> 1) large messages
> 2) reliable delivery
> 3) multiple consumers.
>
> Paul
>
> On 18 October 2016 at 14:22, Mohamed HAMZAOUI <requinham@xxxxxxxxx> wrote:
>>
>> Hi Roger,
>>
>> I just checkout master git on https://github.com/eclipse/mosquitto and
>> when testing the last working version, I detect this behavior :
>> 1- At starting the broker allocate 37240 KB of virtual memory and 4756
>> of resident memory (in the last version just 4340 virtual and 1360
>> resident)
>> 2- When transferring my message of 100 MB, the memory peak is about
>> 242048 for one publisher and one subscriber and about 550 MB for one
>> publisher and 4 subscriber. It's not bad compared to latest version
>> but I think there are a means to marshal message from only one
>> instance in memory ?
>>
>> Regards,
>> Mohamed
>>
>>
>>
>> On Tue, Oct 18, 2016 at 12:54 PM, Roger Light <roger@xxxxxxxxxx> wrote:
>> > Hi Mohamed,
>> >
>> >> I don't understand exceeding 300 MB to transfer 100MB message size
>> >> between one publisher and one subscriber ! Then I tried to test RSMB
>> >> code but more problems.
>> >
>> > This is down to some unnecessary copying of the payload which is
>> > already changed in the development branch.
>> >
>> > Cheers,
>> >
>> > Roger
>> > _______________________________________________
>> > mosquitto-dev mailing list
>> > mosquitto-dev@xxxxxxxxxxx
>> > To change your delivery options, retrieve your password, or unsubscribe
>> > from this list, visit
>> > https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
>> _______________________________________________
>> mosquitto-dev mailing list
>> mosquitto-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/mosquitto-dev
>
>
>
>
> --
> Paul Fremantle
> Part-time PhD student - School of Computing
> email: paul.fremantle@xxxxxxxxxx, paul@xxxxxxxxxxxxx
> twitter: pzfreo / skype: paulfremantle / blog: http://pzf.fremantle.org
> Co-Founder, WSO2
> Apache Member and Committer
> 07740 199 729
>
> _______________________________________________
> mosquitto-dev mailing list
> mosquitto-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/mosquitto-dev