[science-iwg] SWG code for DRMAA

["Torkild U. Resheim" <torkildr@xxxxxxxxx>]

Hi all,

Erwin, Greg and I was earlier briefly discussing how to handle compute clusters within the SWG. I’m working on a project that requires my code to work on both Torque and Slurm based clusters, which obviously made me want to do some abstraction. PTP looks interesting, but I think we can and should utilize that code without change, mostly for monitoring for advanced users. What I want is something without the Perl dependencies, that can be utilized by our users without them having to think about clusters. They just want to run a workflow that will take a couple of days on an 144 CPU EC2 cluster that the application will automatically provision for the – and get a message when it’s done. Also there are breakpoints and error handling that must be taken care of.

So I think the code based on Son of Grid Engine may be part of the solution. I’ve not investigated it yet, and it may be that I must do some adjustments to that code. Has anyone looked into this? If possible, should we adopt this code so that we can further develop it?

I do have a budget and can probably spend 2-3 weeks on something we can share. If we include the provisioning mechanism I can spend more time. I need to do that regardless. Anyone up for a short meeting to discuss this? 

Best regards,
Torkild

PS: If I remember correctly the AWS code from Amazon is BSD licensed.