Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [science-iwg] Eclipse January Workshop, June 20th @EclipseCon France

Very interesting, and great to see how IoT and Science projects are integrated in this work.

 

I’m definitely looking forward to EclipseCon France with its IoT and Science activities, and to the January workshop!

 

Regards

erwin

 

From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Ian Skerrett
Sent: Monday, May 8, 2017 4:50 PM
To: 'Science Industry Working Group' <science-iwg@xxxxxxxxxxx>
Subject: Re: [science-iwg] Eclipse January Workshop, June 20th @EclipseCon France

 

FWIW, here is a presentation that was given at the recent Eclipse IoT Day that references this work.  It is a really great IoT example and use case of January plus some Eclipse IoT technology.

 

https://www.youtube.com/watch?v=tIRBifWKjd0

 

Ian

 

 

From: science-iwg-bounces@xxxxxxxxxxx [mailto:science-iwg-bounces@xxxxxxxxxxx] On Behalf Of Tracy Miranda
Sent: Monday, May 8, 2017 6:41 AM
To: Science Industry Working Group <science-iwg@xxxxxxxxxxx>
Subject: Re: [science-iwg] Eclipse January Workshop, June 20th @EclipseCon France

 

Hi Scott,

 

This is great. That is such an interesting use case of January for edge processing on gateways. Would love to find a suitable forum to have you present your experiences. 

 

Additionally thanks for the details on the Python integration, this is one of the topics to cover at the workshop and really appreciate you taking the time to detail this usage. 

 

>>It sounds like some are interested in this:  i.e. exchanging large amounts of data directly between IDataset and numpy Dataset in a performant way over Py4j.   I think this could be easily done with protobuf and the osgiservicebridge described above..

 

Yes, you have hit the nail on the head & I look forward to diving into the details.

 

Tracy

 

On Thu, May 4, 2017 at 9:28 PM, Scott Lewis <slewis@xxxxxxxxxxxxx> wrote:

Hi Tracy,

I would love to be there, but it doesn't seem likely for me at the moment.

I do, however, wish to contribute to the uses of January discussion.   I'm aware of an effort to use January on gateways for edge analytics and can/could/will present experiences.

I also want to describe some work I've been doing with integrating Java and Python.   Specifically, integrating OSGi services and Python.   As many of you are aware, Py4j allows Java<->Python programmatic interaction with efficient inter-process communication.    The core of this is pass-by-reference interaction between python code calling into java and java code calling into python.  

OSGi services

OSGi services are POJOs that expose 1 or more service interfaces (the 'contract' of the service) to consumers.   OSGI services are used heavily in Eclipse and of course in IoT-relevant frameworks like Kura, Karaf, Predix and other others.    OSGi services have lots of advantages over POJOs, however, such as dynamics, versioning, use in injection frameworks like Spring, DS, others.  

My intention was to use OSGi Remote Services and Py4j to allow both of:  1)  Python code -> OSGi service; 2) OSGi services being implemented by Python code.   You can think of these as the service consumer being in Python (1) and the service provider/implementer being in Python (2).

OSGi Remote Services (RS) is a specification allowing transport independent remoting of OSGi services.   ECF has an implementation of RS that allows distribution providers to be substituted to support multiple transports [1].

OSGi Service Bridge

1) Created a distribution provider that uses Py4j as the communications transport between Java/OSGi process and Python process.
2) Created a python 'osgiservicebridge' that provides python code the ability to interact with the OSGi service registry [3]

These two together allows full Python<->Java interaction...i.e. Python providers/impl of OSGi services, and Python consumers of OSGi services.   The remote service metadata is standardized (by OSGi spec) so it's not tightly bound to ECF or any remote services impl.   ECF is unique, however, in that we make it easy to substitute custom/new distribution providers.

We've been using this to allow OSGi services implemented in Java to be used from within Python.   More importantly, though, we've been using this osgiservicebridge to allow Python classes to be exposed as OSGi service instances.   There's a full code example of doing here [4].

This makes it very easy (and dynamic) to expose python-implemented functionality into any OSGi system, and without doing anything it can use (e.g.) declarative services to have python implemented proxies injected into consumer code.  For example in this [5] class is the code that consumes the IHello service.  It's injected via DS calling bindHello when IHello becomes available, and then the sayHello method is called in activate.

On the python side (IHello impl) is this code:

@protobuf_remote_service(objectClass=['org.eclipse.ecf.examples.protobuf.hello.IHello'])
class HelloServiceImpl:
   
    @protobuf_remote_service_method(arg_type=HelloMsgContent)
    def sayHello(self,pbarg):
        print("sayHello called with arg="+str(pbarg))
        return None

The use of the two decorators @protobuf_remote_service and @protobuf_remote_service_method are *all that is necessary* to have the HelloServiceImpl class serve as a full implementation of the IHello service interface.

Note also that this is using Google's protocol buffers to do method arguments and return value serialization and deserialization.  We did this aprather than use the 'pass-by-reference' default behavior of Py4j because we found that it made the Java->Python method call > 20x faster when a large amount of data (in method arg) was passed between Java and Python.   This is not required for use of the osgi service bridge, but it is useful when exchanging large amounts of data between Java and Python.   I would like to look at doing this with January Datasets and Python numpy Datasets for Java <-> Python.   It sounds like some are interested in this:  i.e. exchanging large amounts of data directly between IDataset and numpy Dataset in a performant way over Py4j.   I think this could be easily done with protobuf and the osgiservicebridge described above.

Apologies for the long email.

Scott

[1] https://wiki.eclipse.org/Distribution_Providers https://wiki.eclipse.org/Tutorial:_Creating_Custom_Distribution_Providers
[2] https://github.com/ECF/Py4j-RemoteServicesProvider
[3] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/python/osgiservicebridge
[4] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/examples




On 5/3/2017 3:48 AM, Tracy Miranda wrote:

Hello all,

 

We will be running an Eclipse January workshop in Toulouse France on June 20th 2017. 

 

Eclipse January is the best way to manage large datasets in Java and features ability to create canonical multidimensional datasets, metadata and many numpy-like features. It has numerous applications for data handling in scientific and IoT fields. 

 

The workshop will cover the main elements of the library: defining different types of datasets, manipulating data with slices and iterators, using metadata, simple loading and storing of data,  etc. We will also touch on how it is being used including Python integration.  

 

The workshop will be part of the Unconference day preceeding the main EclipseCon France conference.  For more details & to register please see https://www.eclipsecon.org/france2017/science

 

Any questions please let me know.

 

Regards,

Tracy

 

 

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

 


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

 


Back to the top