Re: [geclipse-dev] srm, gridftp upload problem

[Mateusz Pabis' <uranium@xxxxxxxxxxxxx>]

Hi Romain.

Happy New Year.

Yes. There is one extra thread for each connection to maintain timeouts
and keep alive activity. But this should not be a problem with freezing.
 Where do other threads come from I don't know. With this amount of
active threads you may cause a distributed denial of service attack.


> Hi guys,
> 
> happy new year !!!
> 
> I am trying to updload at least 10000 files on diverse storage elements of EGEE (srm and gridftp).
> 
> To do that I use the geclipse implemetation of EFS (copy method) and 200 threads.
> 
> At the begining everything is fine and uploads are very fast, but after a while my application
> get very slow and freeze.
> 
> Many thread have been created: 1031 active threads instead of around 200.
> 
> Most of them are waiting:
> 
> **************************************
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> org.globus.ftp.dc.TaskThread$Buffer.get(TaskThread.java:122)
> org.globus.ftp.dc.TaskThread.getNextTask(TaskThread.java:55)
> org.globus.ftp.dc.TaskThread.run(TaskThread.java:66)
> java.lang.Thread.run(Thread.java:619)
> **************************************
> 
> I have got also many timer threads (I suppose it is for the timout):
> 
> **************************************
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> **************************************
> 
> And my user threads are waiting to :( also I have lot of network bandwidth available:
> 
> **************************************
> java.net.SocketInputStream.socketRead0(Native Method)
> java.net.SocketInputStream.read(SocketInputStream.java:129)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
> org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
> org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
> org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
> sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
> java.io.InputStreamReader.read(InputStreamReader.java:167)
> java.io.BufferedReader.fill(BufferedReader.java:136)
> java.io.BufferedReader.readLine(BufferedReader.java:299)
> java.io.BufferedReader.readLine(BufferedReader.java:362)
> org.globus.ftp.vanilla.Reply.<init>(Reply.java:71)
> org.globus.ftp.vanilla.FTPControlChannel.read(FTPControlChannel.java:342)
> org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:285)
> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:107)
> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:92)
> eu.geclipse.efs.gridftp.internal.GridFTPConnection.connect(GridFTPConnection.java:187)
> eu.geclipse.efs.gridftp.internal.GridFTPConnection.getClient(GridFTPConnection.java:476)
> eu.geclipse.efs.gridftp.internal.GridFTPConnection.asyncPut(GridFTPConnection.java:716)
> eu.geclipse.efs.gridftp.GridFile.openOutputStream(GridFile.java:437)
> org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:218)
> org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
> org.eclipse.core.internal.filesystem.local.LocalFile.copy(LocalFile.java:109)
> org.simexplorer.efsfile.EFSFile.copy(EFSFile.java:215)
> org.simexplorer.applicationloader.internal.EFSSimulationSerializerImpl.saveJar(EFSSimulationSerializerImpl.java:145)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.uploadInputFile(GEclipseSimExplorerExecutionJob.java:208)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getInputFile(GEclipseSimExplorerExecutionJob.java:244)
> org.simexplorer.geclipse.GEclipseJobBuilder.getJobDescription(GEclipseJobBuilder.java:37)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getDescription(GEclipseSimExplorerExecutionJob.java:265)
> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.getDescription(GEclipseEnvironmentExecutionJob.java:33)
> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.submitJob(GEclipseEnvironmentExecutionJob.java:39)
> org.simexplorer.geclipse.GEclipseExecutionJob.submit(GEclipseExecutionJob.java:190)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.submit(GEclipseSimExplorerExecutionJob.java:305)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.update(GEclipseSimExplorerExecutionJob.java:68)
> org.simexplorer.updater.UpdaterTask.run(UpdaterTask.java:33)
> org.simexplorer.tools.PoolExecutor.run(PoolExecutor.java:22)
> java.lang.Thread.run(Thread.java:619)
> **************************************
> 
> 
> Do you have any idea how to tackle this problem?
> 
> Is it an implementation error in globus ftp implementation?
> 
> Cheers,
> Romain.
_______________________________________________
geclipse-dev mailing list
geclipse-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/geclipse-dev