Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [science-iwg] Eclipse January Workshop, June 20th @EclipseCon France

Hi,

Same here. There are mandatory holidays the week before (June 15/16) and I'm responsible for the Java program and speaking at DWX the week after, so it seems a bit tight this year even for the Unconference. Probably have to see for ECE at least that's in Germany and easier for me to reach.

Regards,
Werner

On Fri, May 5, 2017 at 6:00 PM, <science-iwg-request@xxxxxxxxxxx> wrote:
Send science-iwg mailing list submissions to
        science-iwg@xxxxxxxxxxx

To subscribe or unsubscribe via the World Wide Web, visit
        https://dev.eclipse.org/mailman/listinfo/science-iwg
or, via email, send a message with subject or body 'help' to
        science-iwg-request@eclipse.org

You can reach the person managing the list at
        science-iwg-owner@xxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of science-iwg digest..."


Today's Topics:

   1. Re: Eclipse January Workshop, June 20th @EclipseCon France
      (Scott Lewis)


----------------------------------------------------------------------

Message: 1
Date: Thu, 4 May 2017 13:28:02 -0700
From: Scott Lewis <slewis@xxxxxxxxxxxxx>
To: science-iwg@xxxxxxxxxxx
Subject: Re: [science-iwg] Eclipse January Workshop, June 20th
        @EclipseCon France
Message-ID: <1c153069-4f16-a015-b1ea-34ad8a02d346@xxxxxxxxxxxxx>
Content-Type: text/plain; charset="windows-1252"; Format="flowed"

Hi Tracy,

I would love to be there, but it doesn't seem likely for me at the moment.

I do, however, wish to contribute to the uses of January discussion.
I'm aware of an effort to use January on gateways for edge analytics and
can/could/will present experiences.

I also want to describe some work I've been doing with integrating Java
and Python.   Specifically, integrating OSGi services and Python.   As
many of you are aware, Py4j allows Java<->Python programmatic
interaction with efficient inter-process communication.    The core of
this is pass-by-reference interaction between python code calling into
java and java code calling into python.

OSGi services

OSGi services are POJOs that expose 1 or more service interfaces (the
'contract' of the service) to consumers.   OSGI services are used
heavily in Eclipse and of course in IoT-relevant frameworks like Kura,
Karaf, Predix and other others.    OSGi services have lots of advantages
over POJOs, however, such as dynamics, versioning, use in injection
frameworks like Spring, DS, others.

My intention was to use OSGi Remote Services and Py4j to allow both of:
1)  Python code -> OSGi service; 2) OSGi services being implemented by
Python code.   You can think of these as the service consumer being in
Python (1) and the service provider/implementer being in Python (2).

OSGi Remote Services (RS) is a specification allowing transport
independent remoting of OSGi services.   ECF has an implementation of RS
that allows distribution providers to be substituted to support multiple
transports [1].

OSGi Service Bridge

1) Created a distribution provider that uses Py4j as the communications
transport between Java/OSGi process and Python process.
2) Created a python 'osgiservicebridge' that provides python code the
ability to interact with the OSGi service registry [3]

These two together allows full Python<->Java interaction...i.e. Python
providers/impl of OSGi services, and Python consumers of OSGi
services.   The remote service metadata is standardized (by OSGi spec)
so it's not tightly bound to ECF or any remote services impl.   ECF is
unique, however, in that we make it easy to substitute custom/new
distribution providers.

We've been using this to allow OSGi services implemented in Java to be
used from within Python.   More importantly, though, we've been using
this osgiservicebridge to allow Python classes to be exposed as OSGi
service instances.   There's a full code example of doing here [4].

This makes it very easy (and dynamic) to expose python-implemented
functionality into any OSGi system, and without doing anything it can
use (e.g.) declarative services to have python implemented proxies
injected into consumer code.  For example in this [5] class is the code
that consumes the IHello service.  It's injected via DS calling
bindHello when IHello becomes available, and then the sayHello method is
called in activate.

On the python side (IHello impl) is this code:

@protobuf_remote_service(objectClass=['org.eclipse.ecf.examples.protobuf.hello.IHello'])
class HelloServiceImpl:

     @protobuf_remote_service_method(arg_type=HelloMsgContent)
     def sayHello(self,pbarg):
         print("sayHello called with arg="+str(pbarg))
         return None

The use of the two decorators @protobuf_remote_service and
@protobuf_remote_service_method are *all that is necessary* to have the
HelloServiceImpl class serve as a full implementation of the IHello
service interface.

Note also that this is using Google's protocol buffers to do method
arguments and return value serialization and deserialization.  We did
this aprather than use the 'pass-by-reference' default behavior of Py4j
because we found that it made the Java->Python method call > 20x faster
when a large amount of data (in method arg) was passed between Java and
Python.   This is not required for use of the osgi service bridge, but
it is useful when exchanging large amounts of data between Java and
Python.   I would like to look at doing this with January Datasets and
Python numpy Datasets for Java <-> Python.   It sounds like some are
interested in this:  i.e. exchanging large amounts of data directly
between IDataset and numpy Dataset in a performant way over Py4j.   I
think this could be easily done with protobuf and the osgiservicebridge
described above.

Apologies for the long email.

Scott

[1] https://wiki.eclipse.org/Distribution_Providers
https://wiki.eclipse.org/Tutorial:_Creating_Custom_Distribution_Providers
[2] https://github.com/ECF/Py4j-RemoteServicesProvider
[3]
https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/python/osgiservicebridge
[4] https://github.com/ECF/Py4j-RemoteServicesProvider/tree/master/examples


On 5/3/2017 3:48 AM, Tracy Miranda wrote:
> Hello all,
>
> We will be running an Eclipse January workshop in Toulouse France on
> June 20th 2017.
>
> Eclipse January is the best way to manage large datasets in Java and
> features ability to create canonical multidimensional datasets,
> metadata and many numpy-like features. It has numerous applications
> for data handling in scientific and IoT fields.
>
> The workshop will cover the main elements of the library: defining
> different types of datasets, manipulating data with slices and
> iterators, using metadata, simple loading and storing of data,  etc.
> We will also touch on how it is being used including Python integration.
>
> The workshop will be part of the Unconference day preceeding the main
> EclipseCon France conference.  For more details & to register please
> see https://www.eclipsecon.org/france2017/science
>
> Any questions please let me know.
>
> Regards,
> Tracy
>
>
> _______________________________________________
> science-iwg mailing list
> science-iwg@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/science-iwg


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dev.eclipse.org/mailman/private/science-iwg/attachments/20170504/51f021a5/attachment.html>

------------------------------

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

End of science-iwg Digest, Vol 52, Issue 2
******************************************


Back to the top