AW: [geclipse-dev] Job Submission with input DataStaging

["Stuempert, Mathias IWR" <mathias.stuempert@xxxxxxxxxx>]

Hi All,

> BTW. Mathias, I do not remember you announced a change in URL
handling.

If I would announce every change in the code you would leave the project
after a week because of the high mail traffic ;-)

> I know you changed the model, but I am not sure about details. What is
the 
> reason for introducing gecl:// ?

Let me give you a more detailed overview what the gecl
EFS-implementation is needed for.

1) The old grid connection implementation was not based on Eclipse's
IResources. That caused bunches (and I really mean BUNCHES) of problems,
please do not force me to go into details here since this would blow up
this mail.

2) Therefore the decision was to make use of
IFile/IFolder.createLink(URI uri) that creates fully integrated
IResources that may be linked to any EFS-implementation.

3) The first attempt with this approach was not very promising (Thomas
can tell you about this, too) since the current (Europa) implementation
of the createLink-method does not support lazy loading (and there is
absolutely no way to enable it directly). That means that if you try to
mount a root directory on e.g. hydra.gup.uni-linz.ac.at createLink
fetches ALL children of that directory recursively. Obviously this is a
no-go. Even for local links (i.e. C:/) this approach is really painful.
You can see this behaviour with the following test:

a) Right-click in the Navigator View and select "New Folder".
b) Open the advanced tab and select "Link to folder in the file system".
Specify a directory that contains a lot of files and folders (e.g.
root).
c) Press finish and have fun...

4) By further examining the source code of the createLink-method it
appeared that the IFileStore.childNames() method was called recursively
for the linked URI. That was the point where I had the idea to wrap all
EFS-implementations by an own implementation where the lazy loading is
implemented directly in the IFileStore. The
GEclipseFileStore.childNames() method returns an empty array if the file
store was not activated before. The activation takes place whenever you
expand a corresponding tree node in any Grid model view. After that the
file store is immediately made inactive again. That prevents Eclipse
from parsing the whole directory tree. As you can see from the current
implementation this approach works quite smoothly for any
EFS-implementation. BTW., If you try to expand such a linked file system
in the Navigator you will realize, that you can expand all folders that
were expanded in a Grid model view before. This is because Eclipse
cashes the directory structure unless you directly tell it to refresh it
(F5).

5) Now the gecl URIs are the native URIs for the gecl-EFS
implementation. The syntax is the following:

gecl://originalURI?originalQuery&geclslave=originalScheme&gecluid=a_uniq
ue_id#originalFragment

So for something like

gsiftp://hydra.gup.uni-linz.ac.at:2811/home/geclise

You end up with

gecl://hydra.gup.uni-linz.ac.at:2811/home/geclise?geclslave=gsiftp&geclu
id= hydra.gup.uni-linz.ac.at2811

Please don't ask me about the unique ID, just believe me that it is
necessary.

> For now it looks that it causes more 
> problems than benefits, as we need to convert it in many places to 
> standard URLs.

Well, for me the conversion problem is the only (big?) problem we
currently have with this implementation. And to be honest, it is rather
a bug than a problem ;-) So it is not very hard to work around this. On
the other hand we have a lot of benefits with this approach. The lazy
loading is only one. Much more important is the fact that each remote
element is now represented by a native IResource implementation. That
makes it very easy to make use of all standard actions for these
resources. Just to name a view of them: copy&paste, create new files and
folders, open files in an editor, change them and save them back ...
With the old implementation we had at least to wrap all these standard
actions. Some other actions (like saving a remote file back) were not
accessible and there was no (easy) way to make them accessible.

> I know this issue. The problem is that for every usage of
GridFileDialog
> Nicholas (in JSDL editor) and I (in JSDL wizard) have to make changes
in
> code. The better solution would be to change code in GridFileDialog.
Is 
> there something that prevents us from changing dialog's code?

No, the dialog's code could for sure be changed. But for me this is the
wrong place to change something. I would change the IGridConnection
implementation (i.e. ConnectionElement) directly in order to retrieve
the slave URI. So if you give me a few days I will try to implement
this.

I hope that helps somehow and gives you a somehow deeper insight into
the connection management. Please tell me if there are still open
questions.

Cheers, Mathias