| Re: [science-iwg] Eclipse January Workshop, June 20th @EclipseCon France |
|
Hi Tracy,
I would love to be there, but it doesn't seem likely for me at the moment. I do, however, wish to contribute to the uses of January discussion. I'm aware of an effort to use January on gateways for edge analytics and can/could/will present experiences. I also want to describe some work I've been doing with integrating Java and Python. Specifically, integrating OSGi services and Python. As many of you are aware, Py4j allows Java<->Python programmatic interaction with efficient inter-process communication. The core of this is pass-by-reference interaction between python code calling into java and java code calling into python. OSGi services OSGi services are POJOs that expose 1 or more service interfaces (the 'contract' of the service) to consumers. OSGI services are used heavily in Eclipse and of course in IoT-relevant frameworks like Kura, Karaf, Predix and other others. OSGi services have lots of advantages over POJOs, however, such as dynamics, versioning, use in injection frameworks like Spring, DS, others. My intention was to use OSGi Remote Services and Py4j to allow both of: 1) Python code -> OSGi service; 2) OSGi services being implemented by Python code. You can think of these as the service consumer being in Python (1) and the service provider/implementer being in Python (2). OSGi Remote Services (RS) is a specification allowing transport independent remoting of OSGi services. ECF has an implementation of RS that allows distribution providers to be substituted to support multiple transports [1]. OSGi Service Bridge 1) Created a distribution provider that uses Py4j as the communications transport between Java/OSGi process and Python process. 2) Created a python 'osgiservicebridge' that provides python code the ability to interact with the OSGi service registry [3] These two together allows full Python<->Java interaction...i.e. Python providers/impl of OSGi services, and Python consumers of OSGi services. The remote service metadata is standardized (by OSGi spec) so it's not tightly bound to ECF or any remote services impl. ECF is unique, however, in that we make it easy to substitute custom/new distribution providers. We've been using this to allow OSGi services implemented in Java to be used from within Python. More importantly, though, we've been using this osgiservicebridge to allow Python classes to be exposed as OSGi service instances. There's a full code example of doing here [4]. This makes it very easy (and dynamic) to expose python-implemented functionality into any OSGi system, and without doing anything it can use (e.g.) declarative services to have python implemented proxies injected into consumer code. For example in this [5] class is the code that consumes the IHello service. It's injected via DS calling bindHello when IHello becomes available, and then the sayHello method is called in activate. On the python side (IHello impl) is this code: @protobuf_remote_service(objectClass=['org.eclipse.ecf.examples.protobuf.hello.IHello']) class HelloServiceImpl: @protobuf_remote_service_method(arg_type=HelloMsgContent) def sayHello(self,pbarg): print("sayHello called with arg="+str(pbarg)) return None The use of the two decorators @protobuf_remote_service and @protobuf_remote_service_method are *all that is necessary* to have the HelloServiceImpl class serve as a full implementation of the IHello service interface. Note also that this is using Google's protocol buffers to do method arguments and return value serialization and deserialization. We did this aprather than use the 'pass-by-reference' default behavior of Py4j because we found that it made the Java->Python method call > 20x faster when a large amount of data (in method arg) was passed between Java and Python. This is not required for use of the osgi service bridge, but it is useful when exchanging large amounts of data between Java and Python. I would like to look at doing this with January Datasets and Python numpy Datasets for Java <-> Python. It sounds like some are interested in this: i.e. exchanging large amounts of data directly between IDataset and numpy Dataset in a performant way over Py4j. I think this could be easily done with protobuf and the osgiservicebridge described above. Apologies for the long email. Scott [1] https://wiki.eclipse.org/Distribution_Providers https://wiki.eclipse.org/Tutorial:_Creating_Custom_Distribution_Providers [2] https://github.com/ECF/Py4j-RemoteServicesProvider [3] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/python/osgiservicebridge [4] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/examples On 5/3/2017 3:48 AM, Tracy Miranda wrote:
|