Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [science-iwg] Few considerations from the yesterday meeting

Mario,

Here is the link to my slides:

https://docs.google.com/presentation/d/1vQQ0d-YrxtprEPKDTle8DyY9MFEbQKkOfUczMjmh1MI/edit?usp=sharing

I appreciate your thoughts. Here are some counter opinions.

First, you are coming at this from the perspective of a person who knows how to program "with massive doses of C, C++, Shell, TCL/TK and Python." So, what you really need from workflow tools is something that provides you a clean API and you'll code the rest. Nearly everyone on this list is the same, but what about the large number of people who are not? Must we force them to learn how to code? At the end of the day, different workflow tools offer different user experiences because they have different customers. 

Second, you should try Java. You might like it. I cut my teeth on C, but also love Java. ;-)

Finally, keep in mind too that while ICE and Triquetrum are both written in Java, they are accessible to other languages through language bindings, as Erwin says. ICE also publishes its services through a RESTful web API, can be run headlessly, or accessed through a web browser from a remote server. As Erwin suggested, you might find Jonah and Tracy's work from Kichwa Coders to integrate C python with the Eclipse Advanced Scripting Environment (EASE) very interesting. We already use EASE extensively in ICE for scripting, especially related to neutron reflectivity simulations.

Jay

On Tue, Jun 7, 2016 at 7:44 AM, Erwin de Ley <erwin.de.ley@xxxxxxxxxx> wrote:
Hi Mario,

Thank you for the very interesting summary and points-of-view.

I must first of all apologize for the somewhat chaotic intro to Triquetrum.
I was tired from traveling and was not well prepared for the kind of "sales pitch" format ;-)

To address your remarks below, in the context of Triquetrum :

1. As I explained, a large part of the internal architecture is inherited from a predecessor project called Passerelle.
This also uses Ptolemy II as its "engine", and mainly targets server clusters as deployment platform for production usage.
Workflows get launched, hundreds of thousands a day, via remote service invocations, scheduled jobs etc. (and only sometimes by user triggers)

It has been developed as OSGi bundles since 10 years or so allowing us to package it depending on the target environment needs.
One incarnation is using a plain personal GUI in Swing (derived from Ptolemy's Vergil). It can do workflows locally or connect to a server to edit,configure models, run them on the server and get progress info back.
The other extreme (which is not open source though) is a complete web-based management GUI + web-based editor etc.

These approaches share most of their modules, but have different interface/view components on top of that (a bit like the Sandia packaging approach from the meeting yesterday).

The goal for Triquetrum in this respect would be to contain all required service APIs and core implementations. But I don't think currently that the web-based interface would be rebuilt in there (unless we can get significant interest and collaboration to get that done). I would try to reach a situation where the RCP application could act as management tool and editor in a kind of client/server setup.

2. About development languages : we're natively Java-based, but integration with R, Python/Jython, _javascript_, Drools, ... is available in different ways in Ptolemy, Passerelle and related Science projects.
Python is on the list as the first "non-JVM" integration in Triquetrum. The basis could be the work we've done in the past to get that working on Passerelle workflow servers at ESRF.
That code has been prepared for reuse, this will be discussed this week with Jonah Graham (Kichwa Coders) who has deep knowledge on Java/Python integration and on the related Science projects.
FYI, the screenshots of the beamline experiment that you refer to, were completely based on using a workflow definition as a coordination of Python scripts that they already had.
And their custom GUI to control the beamlines is also Python and Qt if I'm not mistaken, so we're surrounded by it! ;-)

3. The Triquetrum slides will be my talk at EclipseCon Toulouse tomorrow. I'll make them available from our Wiki, and I guess Eclipse will also upload them on the EclipseCon site.

Regards

erwin

Op 07/06/2016 om 09:23 schreef Mario Valle:

Dear group,
thanks for the yesterday meeting! It was really informative and helpful to me.

One of my jobs here at CSCS is to find, if it exists, a workflow manager that could be adopted for applications like the Material Cloud, the NExT VRE European project, or in general for the kind of computational experiments done at CSCS. The second goal is to understand what these systems need to provide to become the generic workflow management system that could be offered as a service by CSCS.

That said, I should confess that I “kindly hate” workflow managers that do everything locally behind a GUI. I found them limited and not very user friendly. Maybe it is better to call them, as KNIME does, “analytics platforms”. For these reasons, I was pleasantly surprised to hear about the Sandia work and their split desktop / HPC workflow “console”. Also the Triquetrum from the slides (contrasted to the one downloaded from the Eclipse project) seems heading in this direction.

The second thing I discovered yesterday is that integration with HPC machines and workload managers is possible. Seems trivial, but WF managers like FireWorks have a quite clumsy way to do this. AiiDA is in a better shape, but it is inextricably linked to the material simulation part. On a different path, Taverna introduced the executable components as Web services and Triquetrum will have REST services. In my opinion, this is the right way to decouple the “console” (WF editor and management) from the execution part.

Last consideration is about how to define and make available new executable components. I grew up with massive doses of C, C++, Shell, TCL/TK and Python so I tend to stay clear of every WF tool that requires Java programming, but I welcome Python and R scripting. From the presentations seems I'm not alone in this request.

To make complete my learning I would like to ask one thing. Is it possible to access the slides presented yesterday? And to have pointers to the public tools mentioned? Forgive me if I have misunderstood or forgot something from yesterday, but with Socrates: “I know that I do not know”.

See you!
        mario



_______________________________________________
science-iwg mailing list
science-iwg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/science-iwg



--
Jay Jay Billings
Oak Ridge National Laboratory
Twitter Handle: @jayjaybillings

Back to the top