AW: [geclipse-dev] Transferring local input files duringjob submission -

["Tao, Jie" <jie.tao@xxxxxxxxxx>]

Hi Harald,

For the user, nothing changes. Just as you mentioned: define an JSDL and then submit. After users strat Submit, gEclipse first translates the JSDL to JDL, then delegates proxies, and finally submits the job using a WMS function.

Currently gLiite provides us two possibilities: 

1. directly call the jobSubmit function.
2. Use three steps, just like Mariusz wrote in the last mail: jobRegister, file transfer, and jobStart.

In the first case, we can not handel files. So Mariusz had the selection: copy the local files to gEclipse memory and use the second possibility to transfer the files to the CE before the executable is started.

This means our current implementation of job submission must be changed to approach 2. But: as I remeber, Powel said this way did not work (he has tried it. The codes are still in the program WMSClient.java, commented of course).

It is a good idea that we discuss this issue in the VRVS meeting.

Cheers,

Jie

-----Ursprüngliche Nachricht-----
Von: geclipse-dev-bounces@xxxxxxxxxxx [mailto:geclipse-dev-bounces@xxxxxxxxxxx] Im Auftrag von Harald Kornmayer
Gesendet: Dienstag, 13. November 2007 08:49
An: Developer mailing list
Betreff: Re: [geclipse-dev] Transferring local input files duringjob submission - changes in jdl

Hi all

For me not clear yet is the time/moment when the local file will be uploaded to the WMS (or Grid??) ? 

Does the WMS upload the localfile immediately after the jobsubmission? 
(when specified as a part of the input sandbox). 

Or does the upload start after the execution of the job happens? 

Or is the problem the way, that the user specifies the file to be local, but the WN does not have a mean to access the local file during execution ("because I switched my laptop of")? 

One solution might be a check if the "local file" are really "transferred" as a part of the InputSandBox? 

Then the question pops up, how the output will be brought back to the local disk?? 

May be we discuss this afternoon?? 

Harald 



 
Did I understand the WMS System right, when I conclude the following: 

1. User defines a JSDL file. (including the usage of local files). 
2. User submits that JSDL to the WMS. 
3. The WMS looks for resources, but did  

On Mon, 2007-11-12 at 17:50 +0100, Ariel Garcia wrote:
> Hi Mariusz,
> 
> > I have to make changes in JDL, because original JDL refer to files 
> > on user's local machine, which is not accessible from worker node.
> 
> of course the local machine cannot (and shouldn't, we don't want to 
> have a gridftp server on our geclipse!!) be accessed from RB/CE/WN
> 
> but that is why you write:
>    InputSandbox = { <SOME-FILES> };
> in your JDL, those files are uploaded from the local machine (or 
> possibly also SE servers) to the WMS/RB. From there the CE fetches the 
> files, and the WN fetches them from the CE (a gLite "optimization" ;-)
> 
> If you wouldn't specify 
>    InputSandboxDestFileName
> at all, all those files would keep their original name but land in the 
> working-dir of the job in the WN
> 
> Actually a JDL with 
>   Executable      = "/bin/cat";
>   Arguments       = "input.txt";
>   InputSandbox = { "file:///tmp/input.txt" }; works without any 
> problems even if "file:///tmp/input.txt" is on the local machine 
> (using the cli, for the API it should work too).
> 
> On the other hand, the file renaming when uploading using 
> InputSandboxDestFileName  doesn't seem to work at all for me from the 
> command line (my UI in a glite 3.0 one, perhaps it would work in 3.1?) 
> the inputsandbox file is uploaded with the original name...
> 
> > Submission will look like:
> > 1. Remove from JDL (in g-Eclipse memory) all references to local 
> > files 2. Register job on WMS 3. Send local files to sandbox, into 
> > location returned during job registration 4. Start job on WMS
> 
> Ok, i am trying to argue that  point 1. shouldn't be required
> 
> > We cannot change JSDL->JDL transformation file (because of license), 
> > so
> 
> that is fine
> 
> > I have to transform JSDL to JDL (in g-Elipse memory), change JDL 
> > (still in g-Eclipse memory) and submit it.
> > I'm not sure if after sending local files to WN sandbox, should I 
> > refer to those files in submitted JDL?
> > I guess not, because those files are in right place (sandbox).
> 
> i think you can leave in the JDL all entries which are not relevant to 
> the current submission step, they shouldn't matter!
> 
> Are you using the WMProxy API provided by glite, right?
> so what you mean (or what i'd expect) is the following method calls:
> 
> myJobID = jobRegister( myJDL, delegationId )
> getTransferProtocols()
> getSandboxDestURI( myJobID, choosenProtocol )
>   [ Upload files here ]
> jobStart( myJobID )
> 
> There the JDL is given only once and it is stored by the instance of 
> WMProxyAPI you are working with. I don't see the need to change the 
> JDL anywhere...
> 
> 
> Cheers, Ariel
> _______________________________________________
> geclipse-dev mailing list
> geclipse-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/geclipse-dev
--
==============================================
Dr. Harald Kornmayer
Principal Researcher
NEC Laboratories Europe, IT Research Division NEC Europe Ltd.
Rathausallee 10, D-53757 St. Augustin, Germany
Phone:       +49 / 2241-92 52 57
Fax:         +49 / 2241-92 52 99
E-mail:      harald.kornmayer@xxxxxxxxxxxx
 
NEC Europe Ltd., Registered Office: NEC House,
1 Victoria Road, London W3 6BL,
Registered in England 2832014
==============================================

_______________________________________________
geclipse-dev mailing list
geclipse-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/geclipse-dev