Re: [science-iwg] triquetrum dependencies overview

[Erwin de Ley <erwin.de.ley@xxxxxxxxxx>]

Hi Jonah,

Thank you for this detailed reply.

The work we did last year for the python integration was mostly about cutting dependencies, so we could reuse the concepts (and a large part of the code base at that time) outside of DAWN (e.g. for non DAWN-actors, in a headless runtime, or a web server application etc).

I think I will look at the current code again and check if we can upgrade, e.g. the system scripts and underlying code.
Would you be willing to review our code?

thanks again
erwin

Jonah Graham schreef op 24/06/2015 om 11:45:

Hi Erwin,

I have tried to answer some of the questions below. I think there is a lot of reuse available from what has already been implemented for Python Actors within Diamond. Perhaps this cross reference will be a good starting point.

Thanks

Jonah 

On 23 June 2015 at 07:58, Erwin de Ley <erwin.de.ley@xxxxxxxxxx> wrote:

--snip--

So I would think that we would mainly target the "AnalysisRPC" approach, i.e. the Python-related requirements from Triquetrum would focus on (if I understood the scope correctly of AnalysisRPC):
- being able to launch a Python runtime in a separate process

In the DAWN case, this is handled by PythonPydevScript, which is an Actor.

 

- load system scripts in there to set-up an "rpc server"

The actor uses an instance of IAnalysisRpcPythonService to create and handle connection to the "AnalysisRPC server". We are using the AnalysisRpcPythonPyDevService as the concrete class, it is used to connect PyDev setting up to the actor, allowing you to configure Python paths/etc to the same as you have set in the workspace.

 

- load user scripts

PythonRunScriptService is used to handle that. It adds a service to the AnalysisRPC server to enable doing an RPC call where the call is a script to run. 

 

- flattening/marshalling of Java data structures and sending them over to the Python side

- execute the user script
- return result data
- unmarshall/unflatten it to Java structures again

Those 4 steps are the ones handled by AnalysisRPC core. 

 

Some more advanced topics that we didn't tackle yet in "Passerelle with AnalysisRPC", would be :
- efficient handling of large concurrent loads : I would imagine that it's not optimal to start new python processes for each script execution. 
Is there a way to reuse/share a pool of processes, or execute scripts concurrently in a single multi-threaded Python "server"?
(cfr evolution for web servers for CGI/FASTCGI/apache mods/...)

The current implementation (mentioned above) by default reuses the Python process for each actor. This means there is no process start-up cost. That is equivalent to a process pool size of 1, but the rpc server is a threaded server so you have some parallelism. In the multi-threaded case, the parallelism is limited by use of the GIL (https://en.wikipedia.org/wiki/Global_Interpreter_Lock), and therefore how the user's Python in the actor is written.

The main entry point of the Python process is:

uk.ac.diamond.scisoft.python/src/scisoftpy/rpc.py

As you can see it (via scisoftpy/python/pyrpc.py) uses a threaded version of XML-RPC server. Other implementations are possible, such as a multi-process XML-RPC server using a pool. 

 

 

- debugging : if there would be a way to have integrated python debugging on scripts running on the remote python instance, that would be a tremendous feature (cfr remote Java debugging)

That is available within the actor mentioned above. Simply check the checkbox for debugging and the Python will connect back to Dawn's integrated PyDev. The code that handles this is pyrpc.py (see the code around if debug:)

 

Of course, any inspiration is welcome for extra features that would be useful in the context of defining/running/debugging/monitoring workflows!

erwin.

Jay Jay Billings schreef op 22/06/2015 om 23:25:

Tracy,

I would really love to talk to you about this. We are just now building out our scripting stuff and I'm happy to share our requirements.

We are targeting Eclipse EASE and Jython at the moment.

Jay

On Jun 22, 2015 4:43 PM, "Jonah Graham" <jonah@xxxxxxxxxxxxxxxx> wrote:

Hello Erwin,

Actually this week we are in Eclipsecon where I will be doing a presentation on AnalysisRPC and comparing and contrasting Java to Python (and vice versa) technology - https://www.eclipsecon.org/france2015/session/integrating-python-and-java-eclipse

At the conference we will also be talking to other members of the Science Working Group who have expressed interest in the technology so we can capture requirements, scope the project appropriately and determine whether/how it can be resourced. Also, the project is part of DAWN today, but we would like to consider where it would be best to live (for example, Pydev or also alongside some of the newer Eclipse scripting projects).

So I will be in a better position to give you an indication after the conference. It will be good to have your input into the requirements capture as well.

Regards,

Tracy

~~~
Jonah Graham
Kichwa Coders Ltd.
www.kichwacoders.com
jonah@xxxxxxxxxxxxxxxx
+44 (0) 1494523014
+44 (0) 7988836817

On 22 June 2015 at 18:02, <Matt.Gerring@xxxxxxxxxxxxx> wrote:

Hello Erwin,

 

>> Matt, do you have more details on if/how/when this would become a separate eclipse project or not? If that would not yet be available, I assume we could initially work with the forked version I created for Olof (ESRF), last year?

 

Tracy and Jonah from Kichwa coders will be project leads for the AnalysisRPC project I think. I have taken the liberty of copying them in for their thoughts on the timescales. In the meantime using the python connection from AnalysisRPC which we abstracted for your work at the ESRF with Olof would be ok to use. It is a subset of all that can be done however!

 

Matt

 

From: Erwin de Ley [mailto:erwin.de.ley@xxxxxxxxxx]
Sent: 22 June 2015 16:52
To: Jay Jay Billings
Cc: Gerring, Matt (DLSLtd,RAL,LSCI); Christopher Brooks
Subject: triquetrum dependencies overview

 

Hi Jay,

I have attached an overview of the initial set of dependencies for Triquetrum, i.e. for the initial code drop and the work in the months after that (as far as I can predict this and ignoring more detailed requirements coming from ICE, SAW, ...).

I've checked on eclipse ORBIT and most things seem to be available there.
There's mainly the ptolemy stuff that's new of course. Christopher already gave details on copyrights/licenses on the forum.

In the second phase, we would get REST services to manage workflows remotely, and this is currently based on JBoss RESTEasy.
Another important topic would be Python integration via the DAWN work.

Matt, do you have more details on if/how/when this would become a separate eclipse project or not?
If that would not yet be available, I assume we could initially work with the forked version I created for Olof (ESRF), last year?

Do you think it's worth getting this done via ICE, seeing the rather limited set of potential issues?

--

Met vriendelijke groeten - Bien à vous - Kind regards

Erwin De Ley

---

Tel. +32 9 335 22 10 fax: +32 9 335 22 19 erwin.de.ley@xxxxxxxxxx

iSencia Belgium Voorhavenlaan 31 bus 11 B-9000 Gent www.isencia.be

 

 

 

 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 

--
~~~
Jonah Graham
Kichwa Coders Ltd.
www.kichwacoders.com
jonah@xxxxxxxxxxxxxxxx
+44 (0) 1494523014
+44 (0) 7988836817

--

Met vriendelijke groeten - Bien à vous - Kind regards

Erwin De Ley

---

Tel. +32 9 335 22 10 fax: +32 9 335 22 19 erwin.de.ley@xxxxxxxxxx

iSencia Belgium Voorhavenlaan 31 bus 11 B-9000 Gent www.isencia.be