Re: [paho-dev] MQTT Conformance/Interoperability Testing status

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [paho-dev] MQTT Conformance/Interoperability Testing status
From: Nicolas Jouanin <nico@xxxxxxxxxxxxxxx>
Date: Wed, 15 Jul 2015 22:35:44 +0200
Delivered-to: paho-dev@xxxxxxxxxxx
List-archive: <https://dev.eclipse.org/mailman/private/paho-dev>
List-help: <mailto:paho-dev-request@eclipse.org?subject=help>
List-subscribe: <https://dev.eclipse.org/mailman/listinfo/paho-dev>, <mailto:paho-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://dev.eclipse.org/mailman/options/paho-dev>, <mailto:paho-dev-request@eclipse.org?subject=unsubscribe>
Hi,

Just tested on a private instance, everything is fine. Sorry about that.

$ python3 client_test.py —hostname=localhost -p 1883
clean up starting
clean up finished
Basic test starting
Basic test succeeded
Retained message test starting
Retained message test succeeded
This server is not queueing QoS 0 messages for offline clients
Offline message queueing test succeeded
Will message test succeeded
Overlapping subscriptions test starting
This server is publishing one message for all matching overlapping subscriptions, not one for each.
Overlapping subscriptions test succeeded
Keepalive test starting
Keepalive test succeeded
Redelivery on reconnect test starting
Redelivery on reconnect test succeeded
test suite succeeded


> Le 15 juil. 2015 à 22:27, Roger Light <roger@xxxxxxxxxx> a écrit :
> 
> Hi Nicolas,
> 
> I've replied on irc, but for others - the answer for those errors is
> that you can't run the test suite on a broker that is doing other
> tasks. Run a local copy and test against it.
> 
> I can't comment on how complete the tests are.
> 
> Cheers,
> 
> Roger
> 
> 
> On Wed, Jul 15, 2015 at 8:59 PM, Nicolas Jouanin <nico@xxxxxxxxxxxxxxx> wrote:
>> Hi,
>> 
>> I was wondering what was the current status of broker testing suite. I mean is it currently reliable for a broker compliance testing ?
>> I’d like to use it to validate my own broker implementation, but some tests are failing. Also I’ve tried to run the conformance tests on test.mosquito.org and it also fails on some test (see results below).
>> 
>> 
>> python3 client_test.py --hostname=test.mosquitto.org -p 1883
>> hostname test.mosquitto.org port 1883
>> clean up starting
>> ERROR:root:receive: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826208>)
>> Traceback (most recent call last):
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
>>    packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
>>    packet.unpack(buffer)
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
>>    self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
>>    buf = buffer[2:2+length].decode("utf-8")
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
>> ERROR:root:call: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826248>)
>> Traceback (most recent call last):
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 155, in __call__
>>    self.receive(callback)
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
>>    packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
>>    packet.unpack(buffer)
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
>>    self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
>>    buf = buffer[2:2+length].decode("utf-8")
>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
>> deleting retained message for topic frl/2130
>> deleting retained message for topic /ohotkuba/19791981jh
>> deleting retained message for topic /ohotkuba/19791981ih
>> deleting retained message for topic /casa/corredor/luz
>> deleting retained message for topic /casa/sala/controlador/luz
>> deleting retained message for topic /casa/sala/luz/cor
>> deleting retained message for topic /casa/sala/slider
>> deleting retained message for topic /casa/jardim/luz
>> deleting retained message for topic /casa/garagem/luz
>> deleting retained message for topic /Schlenz/AndroidCI
>> deleting retained message for topic /Schlenz/WebCI
>> deleting retained message for topic /Schlenz/iOSCI
>> deleting retained message for topic /jolicode/coffee_pot
>> deleting retained message for topic /node0/dht0/temperature
>> deleting retained message for topic /node0/dht0/humidity
>> deleting retained message for topic /node0/gas0/pollution
>> deleting retained message for topic /dcc/jodo
>> deleting retained message for topic /_mytopic/date
>> deleting retained message for topic /openhab/in/Date/state
>> deleting retained message for topic /openhab/in/Weather_Temperature/state
>> deleting retained message for topic /openhab/in/Light_GF_Living_Table/state
>> deleting retained message for topic /openhab/in/Light_GF_Corridor_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_GF_Kitchen_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_GF_Kitchen_Table/state
>> deleting retained message for topic /openhab/in/Light_GF_Corridor_Wardrobe/state
>> deleting retained message for topic /openhab/in/Light_GF_Toilet_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_GF_Toilet_Mirror/state
>> deleting retained message for topic /openhab/in/Light_FF_Bath_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_FF_Bath_Mirror/state
>> deleting retained message for topic /openhab/in/Light_FF_Corridor_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_FF_Office_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_FF_Child_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_FF_Bed_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_C_Corridor_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_C_Staircase/state
>> deleting retained message for topic /openhab/in/Light_C_Washing_Ceiling/state
>> deleting retained message for topic /openhab/in/Light_C_Workshop/state
>> deleting retained message for topic /openhab/in/Light_Outdoor_Garage/state
>> deleting retained message for topic /openhab/in/Light_Outdoor_Terrace/state
>> deleting retained message for topic /openhab/in/Light_Outdoor_Frontdoor/state
>> deleting retained message for topic /openhab/in/Heating_GF_Corridor/state
>> deleting retained message for topic /openhab/in/Heating_GF_Toilet/state
>> deleting retained message for topic /openhab/in/Heating_GF_Living/state
>> deleting retained message for topic /openhab/in/Heating_GF_Kitchen/state
>> deleting retained message for topic /openhab/in/Heating_FF_Bath/state
>> deleting retained message for topic /openhab/in/Heating_FF_Office/state
>> deleting retained message for topic /openhab/in/Heating_FF_Child/state
>> deleting retained message for topic /openhab/in/Heating_FF_Bed/state
>> deleting retained message for topic /openhab/in/Temperature_Setpoint/state
>> deleting retained message for topic /openhab/in/Window_GF_Frontdoor/state
>> deleting retained message for topic /openhab/in/Window_GF_Kitchen/state
>> deleting retained message for topic /openhab/in/Window_GF_Living/state
>> deleting retained message for topic /openhab/in/Window_GF_Toilet/state
>> deleting retained message for topic /openhab/in/Window_FF_Bath/state
>> deleting retained message for topic /openhab/in/Window_FF_Bed/state
>> deleting retained message for topic /openhab/in/Window_FF_Office_Window/state
>> deleting retained message for topic /openhab/in/Window_FF_Office_Door/state
>> deleting retained message for topic /openhab/in/Garage_Door/state
>> deleting retained message for topic /openhab/in/Temperature_GF_Corridor/state
>> deleting retained message for topic /openhab/in/Temperature_GF_Toilet/state
>> deleting retained message for topic /openhab/in/Temperature_GF_Living/state
>> deleting retained message for topic /openhab/in/Temperature_GF_Kitchen/state
>> deleting retained message for topic /openhab/in/Temperature_FF_Bath/state
>> deleting retained message for topic /openhab/in/Temperature_FF_Office/state
>> deleting retained message for topic /openhab/in/Temperature_FF_Child/state
>> deleting retained message for topic /openhab/in/Temperature_FF_Bed/state
>> deleting retained message for topic /openhab/in/Weather_Temp_Max/state
>> deleting retained message for topic /openhab/in/Weather_Temp_Min/state
>> deleting retained message for topic /home
>> deleting retained message for topic /home/temperature
>> deleting retained message for topic /openHABBruegge/out/Slim_motion/state
>> deleting retained message for topic /openHABBruegge/out/Slim_temp/state
>> deleting retained message for topic /openHABBruegge/out/Slim_door/state
>> deleting retained message for topic /openHABBruegge/out/Slim_battery/state
>> deleting retained message for topic /v1/p0/starling/3810/status
>> deleting retained message for topic /v1/p0/buyAtmel/aeef/status
>> deleting retained message for topic /v1/p0/atmeld/aeef/status
>> clean up finished
>> Basic test starting
>> Basic test succeeded
>> Retained message test starting
>> Traceback (most recent call last):
>>  File "client_test.py", line 142, in retained_message_test
>>    aclient.disconnect()
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
>>    assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
>> AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n   "2,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n   "6,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
>> Retained message test failed
>> Traceback (most recent call last):
>>  File "client_test.py", line 218, in offline_message_queueing_test
>>    aclient.disconnect()
>>  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
>>    assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
>> AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n   "2,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n   "6,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
>> Offline message queueing test failed
>> Traceback (most recent call last):
>>  File "client_test.py", line 178, in will_message_test
>>    assert len(callback2.messages) == 1, callback2.messages  # should have the will message
>> AssertionError: [('Topic/C', b'qos 1', 1, True, 1), ('Topic/C', b'client not disconnected', 2, False, 2)]
>> Will message test failed
>> Overlapping subscriptions test starting
>> Traceback (most recent call last):
>>  File "client_test.py", line 253, in overlapping_subscriptions_test
>>    assert len(callback.messages) in [1, 2]
>> AssertionError
>> Overlapping subscriptions test failed
>> Keepalive test starting
>> Keepalive test succeeded
>> Redelivery on reconnect test starting
>> Traceback (most recent call last):
>>  File "client_test.py", line 309, in redelivery_on_reconnect_test
>>    assert len(callback2.messages) == 2, "length should be 2: %s" % callback2.messages
>> AssertionError: length should be 2: [('TopicA/B', b'', 1, False, 2), ('TopicA/C', b'qos 2', 2, True, 1), ('TopicA/C', b'', 2, False, 3)]
>> Redelivery on reconnect test failed
>> test suite failed
>> _______________________________________________
>> paho-dev mailing list
>> paho-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.eclipse.org/mailman/listinfo/paho-dev
> _______________________________________________
> paho-dev mailing list
> paho-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/paho-dev
Follow-Ups:
- Re: [paho-dev] MQTT Conformance/Interoperability Testing status
  - From: Ian Craggs
References:
- [paho-dev] MQTT Conformance/Interoperability Testing status
  - From: Nicolas Jouanin
- Re: [paho-dev] MQTT Conformance/Interoperability Testing status
  - From: Roger Light
Prev by Date: Re: [paho-dev] MQTT Conformance/Interoperability Testing status
Next by Date: [paho-dev] Callback exception
Previous by thread: Re: [paho-dev] MQTT Conformance/Interoperability Testing status
Next by thread: Re: [paho-dev] MQTT Conformance/Interoperability Testing status
Index(es):
- Date
- Thread
Breadcrumbs
