The purpose of this document is to define the technology that can be used for the RPC channel that OpenMDM® components require.

1. License

Eclipse Public License - v 1.0

THE ACCOMPANYING DOCUMENT IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM CONSTITUTES RECIPIENT’S ACCEPTANCE OF THIS AGREEMENT.

1. DEFINITIONS

"Contribution" means:

  1. in the case of the initial Contributor, the initial code and documentation distributed under this Agreement, and

  2. in the case of each subsequent Contributor:

    1. changes to the Program, and

    2. additions to the Program;

where such changes and/or additions to the Program originate from and are distributed by that particular Contributor. A Contribution 'originates' from a Contributor if it was added to the Program by such Contributor itself or anyone acting on such Contributor’s behalf. Contributions do not include additions to the Program which: (i) are separate modules of software distributed in conjunction with the Program under their own license agreement, and (ii) are not derivative works of the Program.

"Contributor" means any person or entity that distributes the Program.

"Licensed Patents" mean patent claims licensable by a Contributor which are necessarily infringed by the use or sale of its Contribution alone or when combined with the Program.

"Program" means the Contributions distributed in accordance with this Agreement.

"Recipient" means anyone who receives the Program under this Agreement, including all Contributors.

2. GRANT OF RIGHTS

  1. Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, distribute and sublicense the Contribution of such Contributor, if any, and such derivative works, in source code and object code form. .Subject to the terms of this Agreement, each Contributor hereby grants Recipient a non-exclusive, worldwide, royalty-free patent license under Licensed Patents to make, use, sell, offer to sell, import and otherwise transfer the Contribution of such Contributor, if any, in source code and object code form. This patent license shall apply to the combination of the Contribution and the Program if, at the time the Contribution is added by the Contributor, such addition of the Contribution causes such combination to be covered by the Licensed Patents. The patent license shall not apply to any other combinations which include the Contribution. No hardware per se is licensed hereunder.

  2. Recipient understands that although each Contributor grants the licenses to its Contributions set forth herein, no assurances are provided by any Contributor that the Program does not infringe the patent or other intellectual property rights of any other entity. Each Contributor disclaims any liability to Recipient for claims brought by any other entity based on infringement of intellectual property rights or otherwise. As a condition to exercising the rights and licenses granted hereunder, each Recipient hereby assumes sole responsibility to secure any other intellectual property rights needed, if any. For example, if a third party patent license is required to allow Recipient to distribute the Program, it is Recipient’s responsibility to acquire that license before distributing the Program.

  3. Each Contributor represents that to its knowledge it has sufficient copyright rights in its Contribution, if any, to grant the copyright license set forth in this Agreement.

3. REQUIREMENTS

A Contributor may choose to distribute the Program in object code form under its own license agreement, provided that:

  1. it complies with the terms and conditions of this Agreement; and

  2. its license agreement:

    1. effectively disclaims on behalf of all Contributors all warranties and conditions, express and implied, including warranties or conditions of title and non-infringement, and implied warranties or conditions of merchantability and fitness for a particular purpose;

    2. effectively excludes on behalf of all Contributors all liability for damages, including direct, indirect, special, incidental and consequential damages, such as lost profits;

    3. states that any provisions which differ from this Agreement are offered by that Contributor alone and not by any other party; and

    4. states that source code for the Program is available from such Contributor, and informs licensees how to obtain it in a reasonable manner on or through a medium customarily used for software exchange.

When the Program is made available in source code form:

  1. it must be made available under this Agreement; and

  2. a copy of this Agreement must be included with each copy of the Program.

Contributors may not remove or alter any copyright notices contained within the Program.

Each Contributor must identify itself as the originator of its Contribution, if any, in a manner that reasonably allows subsequent Recipients to identify the originator of the Contribution.

4. COMMERCIAL DISTRIBUTION

Commercial distributors of software may accept certain responsibilities with respect to end users, business partners and the like. While this license is intended to facilitate the commercial use of the Program, the Contributor who includes the Program in a commercial product offering should do so in a manner which does not create potential liability for other Contributors. Therefore, if a Contributor includes the Program in a commercial product offering, such Contributor ("Commercial Contributor") hereby agrees to defend and indemnify every other Contributor ("Indemnified Contributor") against any losses, damages and costs (collectively "Losses") arising from claims, lawsuits and other legal actions brought by a third party against the Indemnified Contributor to the extent caused by the acts or omissions of such Commercial Contributor in connection with its distribution of the Program in a commercial product offering. The obligations in this section do not apply to any claims or Losses relating to any actual or alleged intellectual property infringement. In order to qualify, an Indemnified Contributor must: a) promptly notify the Commercial Contributor in writing of such claim, and b) allow the Commercial Contributor to control, and cooperate with the Commercial Contributor in, the defense and any related settlement negotiations. The Indemnified Contributor may participate in any such claim at its own expense.

For example, a Contributor might include the Program in a commercial product offering, Product X. That Contributor is then a Commercial Contributor. If that Commercial Contributor then makes performance claims, or offers warranties related to Product X, those performance claims and warranties are such Commercial Contributor’s responsibility alone. Under this section, the Commercial Contributor would have to defend claims against the other Contributors related to those performance claims and warranties, and if a court requires any other Contributor to pay any damages as a result, the Commercial Contributor must pay those damages.

5. NO WARRANTY

EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the appropriateness of using and distributing the Program and assumes all risks associated with its exercise of rights under this Agreement , including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and unavailability or interruption of operations.

6. DISCLAIMER OF LIABILITY

EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

7. GENERAL

If any provision of this Agreement is invalid or unenforceable under applicable law, it shall not affect the validity or enforceability of the remainder of the terms of this Agreement, and without further action by the parties hereto, such provision shall be reformed to the minimum extent necessary to make such provision valid and enforceable.

If Recipient institutes patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Program itself (excluding combinations of the Program with other software or hardware) infringes such Recipient’s patent(s), then such Recipient’s rights granted under Section 2(b) shall terminate as of the date such litigation is filed.

All Recipient’s rights under this Agreement shall terminate if it fails to comply with any of the material terms or conditions of this Agreement and does not cure such failure in a reasonable period of time after becoming aware of such noncompliance. If all Recipient’s rights under this Agreement terminate, Recipient agrees to cease use and distribution of the Program as soon as reasonably practicable. However, Recipient’s obligations under this Agreement and any licenses granted by Recipient relating to the Program shall continue and survive.

Everyone is permitted to copy and distribute copies of this Agreement, but in order to avoid inconsistency the Agreement is copyrighted and may only be modified in the following manner. The Agreement Steward reserves the right to publish new versions (including revisions) of this Agreement from time to time. No one other than the Agreement Steward has the right to modify this Agreement. The Eclipse Foundation is the initial Agreement Steward. The Eclipse Foundation may assign the responsibility to serve as the Agreement Steward to a suitable separate entity. Each new version of the Agreement will be given a distinguishing version number. The Program (including Contributions) may always be distributed subject to the version of the Agreement under which it was received. In addition, after a new version of the Agreement is published, Contributor may elect to distribute the Program (including its Contributions) under the new version. Except as expressly stated in Sections 2(a) and 2(b) above, Recipient receives no rights or licenses to the intellectual property of any Contributor under this Agreement, whether expressly, by implication, estoppel or otherwise. All rights in the Program not expressly granted under this Agreement are reserved.

This Agreement is governed by the laws of the State of New York and the intellectual property laws of the United States of America. No party to this Agreement will bring a legal action under this Agreement more than one year after the cause of action arose. Each party waives its rights to a jury trial in any resulting litigation.

2. Introduction

TBD

2.1. Conventions and Terms

2.1.1. Typography

A fixed width, non-serif typeface (sample) indicates the term is a Java package, class, interface, or member name. Text written in this typeface is always related to coding. Emphasis (sample) is used the first time an important concept is introduced. Its explanation usually follows directly after the introduction.

2.1.2. Key Words

This specification consistently uses the words may, should, and must. Their meaning is well-defined in [1] Bradner, S., Key words for use in RFCs to Indicate Requirement Levels. A summary follows.

  • must – An absolute requirement. Both the Framework implementation and bundles have obligations that are required to be fulfilled to conform to this specification.

  • should – Recommended. It is strongly recommended to follow the description, but reasons may exist to deviate from this recommendation.

  • may or can – Optional. Implementations must still be interoperable when these items are not implemented.

3. Requirements

3.1. Required

These requirements are an absolute must-have, that is, a technology must be found that can deliver a solution that fulfils the requirement.

3.2. Optional

These requirements are nice-to-have, that is, they influence the implementation options but do not restrict component development in any way.

4. Candidate Technologies

Some candidate technologies support more than one serialization mechanism. We’ll list all RPC technologies first, regardless of their preference of serialization mechanism. the most popular and widely available serialization options will be shown next.

4.1. Candidate RPC Technologies

4.1.1. Java RMI

The Java Remote Method Invocation (Java RMI is a Java API that performs the object-oriented equivalent of remote procedure calls (RPC), with support for direct transfer of serialized Java classes and distributed garbage collection.

  1. The original implementation depends on Java Virtual Machine (JVM) class representation mechanisms and it thus only supports making calls from one JVM to another. The protocol underlying this Java-only implementation is known as Java Remote Method Protocol (JRMP).

  2. In order to support code running in a non-JVM context, a CORBA version was later developed.

Usage of the term RMI may denote solely the programming interface or may signify both the API and JRMP, IIOP, or another implementation, whereas the term RMI-IIOP (read: RMI over IIOP) specifically denotes the RMI interface delegating most of the functionality to the supporting CORBA implementation.

4.1.2. Hessian

The Hessian binary web service protocol makes web services usable without requiring a large framework, and without learning yet another alphabet soup of protocols. Because it is a binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.

4.1.3. Kryonet

KryoNet is a Java library that provides a clean and simple API for efficient TCP and UDP client/server network communication using NIO. KryoNet uses the Kryo serialization library to automatically and efficiently transfer object graphs across the network.

4.1.4. Kryonetty

KryoNetty is a Java library that provides clean and simple API for efficient TCP and UDP client/server network communication using Netty. KryoNetty uses the Kryo serialization library to automatically and efficiently transfer object graphs across the network.

4.1.5. Finagle

Twitter’s Finagle is an extensible RPC system for the JVM, used to construct high-concurrency servers. Finagle implements uniform client and server APIs for several protocols, and is designed for high performance and concurrency. Most of Finagle’s code is protocol agnostic, simplifying the implementation of new protocols.

Finagle is written in Scala, but provides both Scala and Java idiomatic APIs.

4.1.6. Protobuf-rpc-pro

Protobuf-rpc-pro provides an java implementation for Google’s Protocol Buffer RPC services. The implementation builds upon Netty for low-level NIO.

Features
  • TCP connection keep-alive.

  • Bi-directional RPC calls from client to server and server to client.

  • SSL socket layer encryption option.

  • Data Compression option.

  • RPC call cancellation.

  • RPC call timeout.

  • Out-of-Band RPC server replies to client.

  • Non RPC Protocol Buffer messaging for peer to peer communication.

  • Protocol Buffer wire protocol.

  • Semantics for calls handling on TCP connection closure.

  • Pluggable logging facility.

For more details see the wiki page http://code.google.com/p/protobuf-rpc-pro/wiki/Features.

4.2. Candidate Serialization Technologies

Classes are expected to follow these rules:

  • implement the java.io.Serializable interface.

  • define a serialVersionUID field.

4.2.1. Built-in Java

Default Serializer

This is the default serializer. All field types must implement the java.io.Serializable interface too. Serialization occurs when writing an instance "as is" to a java.io.ObjectOutputStream using a single writeObject() invocation. Deserialization occurs when reading from a java.io.ObjectInputStream using a single readObject() invocation.

Manual Serializer

This serialization differs form the default by micromanaging the fields and types that are written to java.io.ObjectOutputStream and read from a stream java.io.ObjectInputStream. Classes need not implement java.io.Serializable but the values must. The custom serialization mechanism may omit some fields that are not needed.

4.2.2. Apache Avro

Apache Avro™ is a data serialization system.

Avro provides:

  • Rich data structures.

  • A compact, fast, binary data format.

  • A container file, to store persistent data.

  • Remote procedure call (RPC).

  • Simple integration with dynamic languages. Code generation is not required to read or write data files nor to use or implement RPC protocols. Code generation as an optional optimization, only worth implementing for statically typed languages.

Avro relies on schemas. When Avro data is read, the schema used when writing it is always present. This permits each datum to be written with no per-value overheads, making serialization both fast and small. This also facilitates use with dynamic, scripting languages, since data, together with its schema, is fully self-describing.

When Avro data is stored in a file, its schema is stored with it, so that files may be processed later by any program. If the program reading the data expects a different schema this can be easily resolved, since both schemas are present.

When Avro is used in RPC, the client and server exchange schemas in the connection handshake. (This can be optimized so that, for most calls, no schemas are actually transmitted.) Since both client and server both have the other’s full schema, correspondence between same named fields, missing fields, extra fields, etc. can all be easily resolved.

Avro schemas are defined with JSON. This facilitates implementation in languages that already have JSON libraries.

4.2.3. Apache Thrift

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.

4.2.4. Protocol Buffers

Protocol Buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages – Java, C++, or Python.

4.2.5. ProtoStuff

Protostuff, a serialization library with built-in support for forward-backward compatibility (schema evolution) and validation.

4.2.6. Hessian

The Hessian binary web service protocol makes web services usable without requiring a large framework, and without learning yet another alphabet soup of protocols. Because it is a binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.

4.2.7. Kryo

Kryo is a fast and efficient object graph serialization framework for Java. The goals of the project are speed, efficiency, and an easy to use API. The project is useful any time objects need to be persisted, whether to a file, database, or over the network.

Kryo can also perform automatic deep and shallow copying/cloning. This is direct copying from object to object, not object→bytes→object.

4.3. Features

The following tables summarizes features and characteristics of each candidate technology

Table 1. Serialization Technologies
Table 2. Serialization Technologies
Feature Java Avro Thrift Protobuf Protostuff Hessian Kryo

License

GPL

ASLv2

ASLv2

New BSD

ASLv2

ASLv2

New BSD

Requires IDL

Target Language

Java

C++

Python

4.4. Serialization Benchmarks

The jvm-serializers project contains extensive tests for multiple serializing technologies. These are the selected technologies and their versions

Java

JDK8u40

Avro

1.7.7

Thrift

0.9.2

Protobuf

2.6.0

Protostuff

1.3.2

Hessian

4.0.38

Kryo

3.0.0

Benchmark tests were run with the following hardware/JVM/OS

JVM

java version "1.8.0_40"
Java™ SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot™ 64-Bit Server VM (build 25.40-b25, mixed mode)

OS

Mac OS X 10.9.5 x86_64

CPU

2.6GHz Intel Core i7

MEM

16GB 1600 MHz DDR3

Raw Benchmark Data
technology                         create     ser   deser   total   size
avro-generic                          357    1801    1019    2820    221
avro-specific                          84    1751    1246    2997    221
hessian                                58    3340    3491    6831    500
java-built-in                          57    5034   24653   29687    889
java-built-in-serializer               58    5172   27292   32464    889
java-manual                            59     788     617    1405    255
kryo-flat-pre                          59     571     706    1277    212
kryo-flat                              58     731     949    1680    268
kryo-manual                            58     482     558    1039    211
kryo-opt                               57     555     675    1230    209
kryo-serializer                        57    1612    1409    3020    286
protobuf/protostuff                    70     432     703    1135    239
protobuf/protostuff-runtime            57     516     710    1227    241
protobuf                              243    1119     576    1695    239
protostuff-graph                       71     707     746    1454    239
protostuff-graph-runtime               59     813     791    1605    241
protostuff-manual                      57     424     717    1141    239
protostuff                             69     453     700    1153    239
protostuff-runtime                     58     568     757    1325    241
thrift-compact                         86    1383     844    2227    240
thrift                                 83    1565     793    2358    349
benchmark total
Figure 1. Benchmark - Total time
benchmark size
Figure 2. Benchmark - Serialized Size

5. Summary

6. References

[1] Bradner, S., Key words for use in RFCs to Indicate Requirement Levels
http://tools.ietf.org/html/rfc2119