Re: [science-iwg] About java-python integration

[Marek Jagielski <marek.jagielski@xxxxxxxxx>]

It was my main concern - how to transfer possibly big amount of data in both directions between Java and Python. I chose thrift (alternatively there is protobuf - https://developers.google.com/protocol-buffers/). At time of decision, thrift came already with rpc service included. It was easier to start up.

In my configuration, I don't run python script directly from Java. Java starts a separate python process. On both side there are thrift server and client. And then, Java sends a path to Python file to be executed.

With thrift, I am still not super satisfied with performance. The serialization and de-serialization is a bottleneck. With thriftpy it can help when you have cython installed. My still unconquerable goal is to process video in a real time.

I think that tools like thrift can be complementary.

Best,

Marek

2016-06-12 17:52 GMT+02:00 Erwin de Ley <erwin.de.ley@xxxxxxxxxx>:

We're in the process of migrating DAWN's Java-Python integration via RPC, into Triquetrum.
This work will be done together with Jonah Graham (Kichwa Coders) and Matt Gerring (Diamond LS), who together developed the original implementation for DAWN (i.e. the so-called AnalysisRPC module.

The goal is to ensure that this becomes a reusable integration module to run Python scripts from Java, so no lock-in on the remainder of Triquetrum. (but of course it will also be used in Triquetrum)

For more info on the AnalysisRPC approach, check e.g. :

- https://www.eclipsecon.org/france2015/sites/default/files/slides/Integrating%20Java%20and%20Python_ecf_0.pdf
- https://kichwacoders.com/2015/01/07/integrating-python-for-high-throughput-science/

The thrift option may be an interesting approach as well. The focus there seems to be on implementing/using "services", more than running arbitrary Python scripts from Java (and being able to pass data around). So this could be complementary?

Met vriendelijke groeten - Bien à vous - Kind regards
Erwin De Ley

---

Tel. +32 9 335 22 10 fax: +32 9 335 22 19 erwin.de.ley@xxxxxxxxxx

iSencia Belgium Voorhavenlaan 31 bus 11 B-9000 Gent www.isencia.be

 

  

Op 12/06/2016 om 13:32 schreef Marek Jagielski:

In my case (https://www.youtube.com/watch?v=qvY3V9Ujoa0) I enabled two ways of java-python integration. 

One is via Jython. However, in this solution it would be hard to use numpy, etc.

Second one is running a separate python process from Java and establish bidirectional communication using thrift (https://thrift.apache.org/). I used on the python side thriftpy library (https://thriftpy.readthedocs.io/en/latest/). However, you are limited to predefined data types. In some situations, I had to do objects serialization to binary type and restore them on the second side.

Best,

Marek

2016-06-11 8:23 GMT+02:00 Philip Wenig <philip.wenig@xxxxxxxxxxxxx>:

Hi Michele,

you may have a look at this slides:
https://www.eclipsecon.org/europe2015/session/integrating-python-and-java-high-throughput-science-eclipse

or the DAWNSci workbench:
http://www.dawnsci.org


Best,
Philip

Am 10.06.2016 um 17:33 schrieb Michele Gabusi:

Hi,

recently I was looking around for a handy solution to setup bidirectional communication between Java and Python. I know that this is not a trivial issue. However, looking at the past EclipseCon presentations, I noticed that jpy libraries (licensed under GPL3) have not been mentioned.

http://jpy.readthedocs.io/en/latest/intro.html
https://pypi.python.org/pypi/jpy/

Do you know any reason that might discourage to implement them?
Do anyone has any experience with these libraries?

Cheers,

Michele.


_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

--
~~~~~~~~~~~~~~~~~~~~~~~~
OpenChrom - the open source alternative for chromatography / mass spectrometry
Dr. Philip Wenig » Founder » philip.wenig@xxxxxxxxxxxxx » http://www.openchrom.net
~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg

_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg