[geclipse-dev] srm, gridftp upload problem

[Romain <reuillon@xxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi guys,

happy new year !!!

I am trying to updload at least 10000 files on diverse storage elements of EGEE (srm and gridftp).

To do that I use the geclipse implemetation of EFS (copy method) and 200 threads.

At the begining everything is fine and uploads are very fast, but after a while my application
get very slow and freeze.

Many thread have been created: 1031 active threads instead of around 200.

Most of them are waiting:

**************************************
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.globus.ftp.dc.TaskThread$Buffer.get(TaskThread.java:122)
org.globus.ftp.dc.TaskThread.getNextTask(TaskThread.java:55)
org.globus.ftp.dc.TaskThread.run(TaskThread.java:66)
java.lang.Thread.run(Thread.java:619)
**************************************

I have got also many timer threads (I suppose it is for the timout):

**************************************
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
**************************************

And my user threads are waiting to :( also I have lot of network bandwidth available:

**************************************
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
java.io.InputStreamReader.read(InputStreamReader.java:167)
java.io.BufferedReader.fill(BufferedReader.java:136)
java.io.BufferedReader.readLine(BufferedReader.java:299)
java.io.BufferedReader.readLine(BufferedReader.java:362)
org.globus.ftp.vanilla.Reply.<init>(Reply.java:71)
org.globus.ftp.vanilla.FTPControlChannel.read(FTPControlChannel.java:342)
org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:285)
org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:107)
org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:92)
eu.geclipse.efs.gridftp.internal.GridFTPConnection.connect(GridFTPConnection.java:187)
eu.geclipse.efs.gridftp.internal.GridFTPConnection.getClient(GridFTPConnection.java:476)
eu.geclipse.efs.gridftp.internal.GridFTPConnection.asyncPut(GridFTPConnection.java:716)
eu.geclipse.efs.gridftp.GridFile.openOutputStream(GridFile.java:437)
org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:218)
org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
org.eclipse.core.internal.filesystem.local.LocalFile.copy(LocalFile.java:109)
org.simexplorer.efsfile.EFSFile.copy(EFSFile.java:215)
org.simexplorer.applicationloader.internal.EFSSimulationSerializerImpl.saveJar(EFSSimulationSerializerImpl.java:145)
org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.uploadInputFile(GEclipseSimExplorerExecutionJob.java:208)
org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getInputFile(GEclipseSimExplorerExecutionJob.java:244)
org.simexplorer.geclipse.GEclipseJobBuilder.getJobDescription(GEclipseJobBuilder.java:37)
org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getDescription(GEclipseSimExplorerExecutionJob.java:265)
org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.getDescription(GEclipseEnvironmentExecutionJob.java:33)
org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.submitJob(GEclipseEnvironmentExecutionJob.java:39)
org.simexplorer.geclipse.GEclipseExecutionJob.submit(GEclipseExecutionJob.java:190)
org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.submit(GEclipseSimExplorerExecutionJob.java:305)
org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.update(GEclipseSimExplorerExecutionJob.java:68)
org.simexplorer.updater.UpdaterTask.run(UpdaterTask.java:33)
org.simexplorer.tools.PoolExecutor.run(PoolExecutor.java:22)
java.lang.Thread.run(Thread.java:619)
**************************************


Do you have any idea how to tackle this problem?

Is it an implementation error in globus ftp implementation?

Cheers,
Romain.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklks3EACgkQbUN+hKKXqguU6ACdE3Jh7BeKLML6hzWhNelWq8pD
wOwAnA3C4oyhX75M8kiR0g//clWCI9Zm
=hSYU
-----END PGP SIGNATURE-----