Re: [geclipse-dev] srm, gridftp upload problem

[Romain <reuillon@xxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I don't think it is a (not distributed, I use only one machine) denial of service problem. The SE are still available when I use them with
a glite UI.

Also if I limit the file accesses to 20 simultaneous accesses, after a while, I get the same effect.

Do you have an explanation for most of my threads being blocked here also there is plenty of bandwidth available ?

>> java.net.SocketInputStream.socketRead0(Native Method)
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>> org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
>> org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
>> org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
>> org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
>> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
>> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
>> sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
>> java.io.InputStreamReader.read(InputStreamReader.java:167)
>> java.io.BufferedReader.fill(BufferedReader.java:136)
>> java.io.BufferedReader.readLine(BufferedReader.java:299)
>> java.io.BufferedReader.readLine(BufferedReader.java:362)
>> org.globus.ftp.vanilla.Reply.<init>(Reply.java:71)
>> org.globus.ftp.vanilla.FTPControlChannel.read(FTPControlChannel.java:342)
>> org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:285)
>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:107)
>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:92)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.connect(GridFTPConnection.java:187)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.getClient(GridFTPConnection.java:476)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.asyncPut(GridFTPConnection.java:716)
>> eu.geclipse.efs.gridftp.GridFile.openOutputStream(GridFile.java:437)
>> org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:218)
>> org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
>> org.eclipse.core.internal.filesystem.local.LocalFile.copy(LocalFile.java:109)

Cheers,
Romain

Mateusz Pabis' a écrit :
> Hi Romain.
> 
> Happy New Year.
> 
> Yes. There is one extra thread for each connection to maintain timeouts
> and keep alive activity. But this should not be a problem with freezing.
>  Where do other threads come from I don't know. With this amount of
> active threads you may cause a distributed denial of service attack.
> 
> 
>> Hi guys,
>>
>> happy new year !!!
>>
>> I am trying to updload at least 10000 files on diverse storage elements of EGEE (srm and gridftp).
>>
>> To do that I use the geclipse implemetation of EFS (copy method) and 200 threads.
>>
>> At the begining everything is fine and uploads are very fast, but after a while my application
>> get very slow and freeze.
>>
>> Many thread have been created: 1031 active threads instead of around 200.
>>
>> Most of them are waiting:
>>
>> **************************************
>> java.lang.Object.wait(Native Method)
>> java.lang.Object.wait(Object.java:485)
>> org.globus.ftp.dc.TaskThread$Buffer.get(TaskThread.java:122)
>> org.globus.ftp.dc.TaskThread.getNextTask(TaskThread.java:55)
>> org.globus.ftp.dc.TaskThread.run(TaskThread.java:66)
>> java.lang.Thread.run(Thread.java:619)
>> **************************************
>>
>> I have got also many timer threads (I suppose it is for the timout):
>>
>> **************************************
>> java.lang.Object.wait(Native Method)
>> java.util.TimerThread.mainLoop(Timer.java:509)
>> java.util.TimerThread.run(Timer.java:462)
>> **************************************
>>
>> And my user threads are waiting to :( also I have lot of network bandwidth available:
>>
>> **************************************
>> java.net.SocketInputStream.socketRead0(Native Method)
>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>> org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
>> org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
>> org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
>> org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
>> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
>> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
>> sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
>> java.io.InputStreamReader.read(InputStreamReader.java:167)
>> java.io.BufferedReader.fill(BufferedReader.java:136)
>> java.io.BufferedReader.readLine(BufferedReader.java:299)
>> java.io.BufferedReader.readLine(BufferedReader.java:362)
>> org.globus.ftp.vanilla.Reply.<init>(Reply.java:71)
>> org.globus.ftp.vanilla.FTPControlChannel.read(FTPControlChannel.java:342)
>> org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:285)
>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:107)
>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:92)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.connect(GridFTPConnection.java:187)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.getClient(GridFTPConnection.java:476)
>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.asyncPut(GridFTPConnection.java:716)
>> eu.geclipse.efs.gridftp.GridFile.openOutputStream(GridFile.java:437)
>> org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:218)
>> org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
>> org.eclipse.core.internal.filesystem.local.LocalFile.copy(LocalFile.java:109)
>> org.simexplorer.efsfile.EFSFile.copy(EFSFile.java:215)
>> org.simexplorer.applicationloader.internal.EFSSimulationSerializerImpl.saveJar(EFSSimulationSerializerImpl.java:145)
>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.uploadInputFile(GEclipseSimExplorerExecutionJob.java:208)
>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getInputFile(GEclipseSimExplorerExecutionJob.java:244)
>> org.simexplorer.geclipse.GEclipseJobBuilder.getJobDescription(GEclipseJobBuilder.java:37)
>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getDescription(GEclipseSimExplorerExecutionJob.java:265)
>> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.getDescription(GEclipseEnvironmentExecutionJob.java:33)
>> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.submitJob(GEclipseEnvironmentExecutionJob.java:39)
>> org.simexplorer.geclipse.GEclipseExecutionJob.submit(GEclipseExecutionJob.java:190)
>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.submit(GEclipseSimExplorerExecutionJob.java:305)
>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.update(GEclipseSimExplorerExecutionJob.java:68)
>> org.simexplorer.updater.UpdaterTask.run(UpdaterTask.java:33)
>> org.simexplorer.tools.PoolExecutor.run(PoolExecutor.java:22)
>> java.lang.Thread.run(Thread.java:619)
>> **************************************
>>
>>
>> Do you have any idea how to tackle this problem?
>>
>> Is it an implementation error in globus ftp implementation?
>>
>> Cheers,
>> Romain.
> _______________________________________________
> geclipse-dev mailing list
> geclipse-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/geclipse-dev
> 
> _______________________________________________
> geclipse-dev mailing list
> geclipse-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/geclipse-dev

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklkvqgACgkQbUN+hKKXqgs4wwCgx0hDK5f+07AWfI05Xiq5UxDX
MC0AniGzJR/JyvdTfdvBsNcI+56Aa8EY
=VqA8
-----END PGP SIGNATURE-----