Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [iot-wg] IoT Connector project proposal
Good points, Julien. Let me elaborate:

> Yes you can use your bus for doing synchronous call, 
> but it's not going to solve any problem here, just adding latency.

Actually an ability to decouple protocol adapters from service implementation is a huge advantage. For example:
a) your service consumes from AMQP only, while you can attach many protocol adapters to the same bus. In practice it means that you can easily expose your service using a variety of APIs. So I can create a service Foo consuming from IoT Connector (AMQP) only and  at the same time expose this service via MQTT, REST, CoAP, WebSockets, etc. All the protocol conversion are automatically handled by IoT connector + protocol adapters.
b) this model of architecture is "cloud-friendly" (aka "PaaS-friendly"). You can easily scale out your application by scaling IoT connector, protocol adapters and services. I can imagine putting those components into Docker+Kubernates and taking advantage of auto-scaling features.
c) this model make it easier to provide IoT infrastructure and business logic (services) separately. It is extremely important from the business point if view, as it allows you to distinguish dedicated IoT infrastructure teams and dedicated services teams.
d) IoT infrastructure (IoT connector + protocol adapters) communicating indirectly with the services is easier to extract and reuse. It makes it cheaper and more stable then IoT infrastructure created from scratch for every project.
e) IoT infrastructure working independently from services is easier to productize, and therefore - monetize.

> When the device connect or send a message you need to read 
> the scenario state and make a decision on the next message to 
> send (simple HTTP call works here, without needing to install 
> any middleware).

Services communicating via HTTP are less resistant for failures. If service A calls service B without middleware, it is the responsibility of a service A to retry if B is down for a moment. Client side load balancing is usually more complicated to develop and maintain (even with libraries like Apache Camel or Netflix Ribbon), than relying on the messaging to do this job for you.

Messaging gives you also some communication patterns that make it easier to scale out communication (like fan-out or competing consumer). 

Finally relying on messaging is much easier for a large scale PaaS/cloud deployments where new nodes can be easily added or destroyed, as you don't need service discovery, load balancing, etc.

> In my opinion bus/queue is something you can keep out of the picture 
> in the first step.

Actually I believe it is essential for us to agree on the communication pattern before we proceed. I personally would say +1 for AMQP 1.0 as it is interoprable between all platforms (Java, .Net, Ruby, Python, Node.js etc.). It is also a mature protocol with many existing implementations that have proved to be solid and scalable.

Cheers!

pt., 27.11.2015 o 20:15 użytkownik Julien Vermillard <jvermillard@xxxxxxxxx> napisał:
Yes you can use your bus for doing synchronous call, but it's not going to solve any problem here, just adding latency.
When the device connect or send a message you need to read the scenario state and make a decision on the next message to send (simple HTTP call works here, without needing to install any middleware).

But using the same bus for storing telemetry values is more interesting, you can use it as a buffer between the devices and the database or the rule engine. also it's helping you to persist received message even if something is wrong in your data ingestion pipeline (like swallowing a traffic storm).

In my opinion bus/queue is something you can keep out of the picture in the first step.


--
Julien Vermillard

On Fri, Nov 27, 2015 at 7:11 PM, Henryk Konsek <hekonsek@xxxxxxxxx> wrote:
Hi Julien,

Actually using request/reply pattern with messaging systems is pretty common and works well for this kind of scenarios. It is only the matter of an extra latency added because device read request is now send from end-client to LeshanServer through AMQP instead of calling it directly. However this kind of latency is usually really small, as your massaging infrastructure deployed in a data center or cloud is usually very fast. I've been working with customers using our messaging technologies for a while and honestly I don't recall a single situation when messaging request/replay latency was an issue. 

As I understand with an architecture proposed by Kai we would have an end-client (like end-customer project willing to read device data) sending AMQP request/reply to Leshan wrapped as LWM2M protocol adapter. Adapter is able to receive the AMQP request, executes something like "leshanServer.send(client, new ReadRequest(resource))" and sends the AMQP response back to end-client. That would be really fast operation.

Keep also in mind that AMQP offers brokerless communication. With open source technologies like Apache Qpid Dispatch Router [1] you can connect two AMQP clients directly without the "broker middleman". That form of message exchange is extremely fast.

So I wouldn't be so concerned about using AMQP request/reply.

Cheers!


pt., 27.11.2015 o 18:15 użytkownik Julien Vermillard <jvermillard@xxxxxxxxx> napisał:
Hi Kai,
first question:
Scenario for complex operation like firmware upgrade or device configuration are in the scope?

Because if I read between the line an async queue based architecture won't be able to handle very synchronous operation like:

wait the device to connect,
when the device is there ask it's version,
wait for the response,
send the firmware download operation for the binary patch corresponding to the received version,
wait for the response,
poll the download status
...
...

if you put an async construction like rabbitmq between the device protocol gateway and something implementing this scenario it won't work because the device will probably left when you will unqueue the next event and you need to send the next command before the device is gone.
You can't enqueue whole command at once because you have control flow (loop and if).
If you put the state machine between the protocol adapter and the bus it will work, if you expect to put this logic on the northbound it won't.

Sending telemetry data from the device to a database or a rule engine in an async way is easy (even if it's where you probably have a lot of data).
Command flow control is a totally different beast, it needs synchronous behaviour.
You can try to use your bus/RabbitMQ in a synchronous way (RPC style constructions with ephemeral response queue) but a good old database will be a better choice.

I hope it does make sense :)

--
Julien Vermillard

On Fri, Nov 27, 2015 at 3:48 PM, Hudalla Kai (INST/ESY) <Kai.Hudalla@xxxxxxxxxxxx> wrote:
Hi,

we have created a draft version of a proposal [1] for starting a project under the Eclipse IoT umbrella for creating the IoT Connector component which is a central part of the IoT Server Platform [2]. We would like to encourage all interested parties to review and comment on the proposal so that we can make sure to create something that will be useful :-)

Please feel free to add yourself to the list of interested parties if you would like to see the component become reality and/or would like to collaborate on its development.

[1] https://wiki.eclipse.org/IoT/IoTConnectorProposal
[2] https://wiki.eclipse.org/IoT/IoTServerPlatform

Mit freundlichen Grüßen/ Best regards
 
Kai Hudalla

Bosch Software Innovations GmbH
Schöneberger Ufer 89
10785 Berlin
GERMANY
www.bosch-si.de

Mobile +49 (151) 54336335 
Phone +49 (30) 726 112-145
kai.hudalla@xxxxxxxxxxxx
 
Registered office: Berlin, Register court: Amtsgericht Charlottenburg, HRB 148411 B
Executives: Dr.-Ing. Rainer Kallenbach; Michael Hahn


_______________________________________________
iot-wg mailing list
iot-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/iot-wg

_______________________________________________
iot-wg mailing list
iot-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/iot-wg
--

_______________________________________________
iot-wg mailing list
iot-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/iot-wg

_______________________________________________
iot-wg mailing list
iot-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/iot-wg
--
Henryk Konsek
http://about.me/hekonsek

Back to the top