Re: [paho-dev] Throttling on massive message publish in Java client

[Bin BJ Zhang <zhbinbj@xxxxxxxxxx>]

Hi Lars,

Regarding "There are no more message IDs available" exception,
yes, you are right, although a message ID is assigned to QoS0 messages,
the actual MQTT packet sent to the wire will not contain the message ID for QoS0 messages.
The message ID is also used by paho client as an internal ID which is not good idea in this case.

And the reason why you didn't get the exception with the sync client is: the sync client will
wait for the message delivery completion, and the message ID is released then. Actually, the sync client
is just a wrapper of async client with a waitForCompletion.

I think we may need a improvement regarding the message ID to avoid it get exhausted in a high traffic environment.

Best Regards,
Bin Zhang
--------------------------------------------------------------------------------------------
WebSphere MQ, IBM China Software Development Lab
E-Mail:  zhbinbj@xxxxxxxxxx
-------------------------------------------------------------------------------------------

Lars Schuetze ---01/28/2015 11:50:01 PM---Hi Ian, thank you very much for your response.

From: Lars Schuetze <dev@xxxxxxxxxxxxxxx>
To: General development discussions for paho project <paho-dev@xxxxxxxxxxx>
Date: 01/28/2015 11:50 PM
Subject: Re: [paho-dev] Throttling on massive message publish in Java client
Sent by: paho-dev-bounces@xxxxxxxxxxx

---

Hi Ian,

thank you very much for your response.

So, for 1)
I did tests with mosquitto, as well as with Mosca, but configured it
to use mosquitto as backend. There is a broker effect and pure
mosquitto is way faster.
[1] shows a graph where the y-axis is the latency (where start = 0) so
it should be a simple line. But at the end it starts going horrible
slow and the latency is increasing rapidly. But that will be a problem
of the TCP packets flying around. This was tested with AsyncClient and
QoS0.

Since I tested it on the same machine the many connections (client <->
broker (TCP), broker <-> webclient (websocket, so TCP)) seems to be
too much.
Having the full pipeline and make use of mosquitto as a parent broker
for Mosca I end up with having packages sent from the beginning (ca.
2600mps arriving), but after stopping it still sends packages for some
time. So this is a Mosca problem and I will not go deeper into this.

For 2)
If I hit this exception I make the client to wait 500ms and then
continue to send.
However, I get a second exception, too: There are no more message IDs
available (which is set to max 65534). But I read up the standard and
there is no need for package identifiers[3] when using QoS0 but
MqttWireMessage[2] always returns true in this regard (and thus is
causing exceptions when hitting [4]). I am not so much familar with
your code base, so it could be I am mixing the standards package
identifier with your message identifier.

I dont get the exception with the sync client, but it sends barely
1600 mps (but there is almost no latency).

Best Regards,
    Lars


[1] https://drive.google.com/file/d/0B2mHWmdGeL4pZHJXOWM2V18xMGM/view?usp=sharing
[2] http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/tree/org.eclipse.paho.client.mqttv3/src/main/java/org/eclipse/paho/client/mqttv3/internal/wire/MqttWireMessage.java#n143
[3] http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.pdf , line
(numbers on the left) 1231 (or page 52)
[4] http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.java.git/tree/org.eclipse.paho.client.mqttv3/src/main/java/org/eclipse/paho/client/mqttv3/internal/ClientState.java#n416

2015-01-27 22:43 GMT+01:00 Ian Craggs <icraggs@xxxxxxxxxxxxxxxxxxxxxxx>:
> Hi Lars,
>
> two things to investigate first come to my mind.
>
> 1) Try similar tests with Mosquitto as well, to see if we can see any broker
> effect.  This can happen especially with QoS 0 as there is no throttling
> available in MQTT for QoS 0 (no acks).
>
> 2) In the async client, what action are you taking when you get the "max
> messages inflight" condition?  Do you see that condition with the sync
> client as well?
>
> Ian
>
>
>
> On 01/27/2015 12:55 PM, Lars Schuetze wrote:
>>
>> Hello,
>>
>> I am using Ecipse Paho Java (1.0.1) implementation in a height
>> throughput environment.
>> I did some experiments and am experiencing some weird things.
>>
>> The scenario is a client using Paho, and a broker using Mosca (NodeJS
>> implementation, said to handle 20k msg/s). We have a websocket browser
>> application for visualization.
>> I have tried the synchronous and asynchronous client as well as
>> different QoS parameters.
>>
>> First of all, the async client is much much slower when using QoS 1
>> then using the synchronous client directly, which should be
>> equivalent. But I am achieving the below listed values:
>>
>> MqttClient (QoS1): 1500 - 1700 mps
>> MqttAsyncClient (QoS1): 20 - 100 mps
>> MqttAsyncClient (QoS0):
>>      - first second 1-2mps then 0 mps
>>      - some seconds pause
>>      - then 4000 - 8000 mps
>>
>> The pictures [1,2] are in the done with the MqttAsyncClient and QoS 1.
>> Where [1] shows the counted packages per second arriving at the
>> broker. As you can see there are some packages sent (99, 1131) until
>> the throughout drops dramatically. After pausing the client the
>> throughput stays some seconds low until it explodes. In [2] you see
>> the graph in which the x-axis is time and y-axis is millisecond
>> between send <-> receive (broker) <-> receive (webclient). The huge
>> increase in time is the point in time where I made the client stop
>> sending. This is where in [1] those (1675, 2040, 1577) packages begin
>> to fly in.
>>
>> The scenario is the very same when using the async client with QoS 0.
>>
>> I've been using Fedora 21, Ubuntu 14.10 for testing this. Those
>> pictures where taken in a local environment. Using Wireshark I
>> calculated that by requesting the client to send 10 bytes messages the
>> TCP package arrived will be 113 bytes. But doing the math I cannot see
>> that my 100/10 Ethernet will be the limiting factor.
>>
>> Maybe you have some ideas on how to investigate further.
>>
>> Regards,
>>      Lars
>>
>> [1]
>> https://drive.google.com/file/d/0B2mHWmdGeL4pM0J4VHBlTlBUd0E/view?usp=sharing
>> [2]
>> https://drive.google.com/file/d/0B2mHWmdGeL4peHFsdEFPbHdCNE0/view?usp=sharing
>> _______________________________________________
>> paho-dev mailing list
>> paho-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe
>> from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/paho-dev
>
>
> --
> Ian Craggs
> icraggs@xxxxxxxxxx                 IBM United Kingdom
> Paho Project Lead; Committer on Mosquitto
>
> _______________________________________________
> paho-dev mailing list
> paho-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/paho-dev
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev