Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geclipse-dev] srm, gridftp upload problem 
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I think I have understood what goes wrong :

Here is the code of the method transferStreams of the class FileStore:

synchronized (buffer) {
	while (true) {
		int bytesRead = -1;
		try {
			bytesRead = source.read(buffer);
		} catch (IOException e) {
			String msg = NLS.bind(Messages.failedReadDuringWrite, path);
			Policy.error(EFS.ERROR_READ, msg, e);
		}
		if (bytesRead == -1)
			break;
		try {
			destination.write(buffer, 0, bytesRead);
		} catch (IOException e) {
		String msg = NLS.bind(Messages.couldNotWrite, path);
		Policy.error(EFS.ERROR_WRITE, msg, e);
	}
	monitor.worked(1);
}

The buffer object is final and static !!

private static final byte[] buffer = new byte[8192];

This means that only one file transfer appends at time on an instance of eclipse no matter what is the destination
file system.

In my case I have one very slow transfer happening, preventing all others from doing there jobs.. it sucks !

Romain


Romain a écrit :
> Hi Mateusz,
> 
> actually, you where right about the denial of service problem, with a better load and failure management (using more often reliable SE),
> I manage to avoid most of the thread spending long times in that state :
> 
>>> java.net.SocketInputStream.socketRead0(Native Method)
>>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>> org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
>>> org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
>>> org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
>>> org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
> 
> So my freezing problem was not because of that, but after a longer run all my threads are freezes here :
> 
> org.eclipse.core.filesystem.provider.FileStore.transferStreams(FileStore.java:70)
> org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:219)
> org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
> org.eclipse.core.internal.filesystem.FileCache.cache(FileCache.java:89)
> org.eclipse.core.filesystem.provider.FileStore.toLocalFile(FileStore.java:468)
> org.simexplorer.efsfile.EFSFile.cache(EFSFile.java:186)
> org.simexplorer.applicationloader.internal.EFSSimulationSerializerImpl.loadJar(EFSSimulationSerializerImpl.java:45)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.tryToGetOutput(GEclipseSimExplorerExecutionJob.java:134)
> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.update(GEclipseSimExplorerExecutionJob.java:97)
> org.simexplorer.updater.UpdaterTask.run(UpdaterTask.java:33)
> org.simexplorer.tools.PoolExecutor.run(PoolExecutor.java:22)
> java.lang.Thread.run(Thread.java:619)
> 
> 
> It an eclipse function !!!
> 
> By chance, do anyone have any idea what's going on?
> 
> Rom
> 
> Mateusz Pabis' a écrit :
>> Hi Romain.
> 
>> Happy New Year.
> 
>> Yes. There is one extra thread for each connection to maintain timeouts
>> and keep alive activity. But this should not be a problem with freezing.
>>  Where do other threads come from I don't know. With this amount of
>> active threads you may cause a distributed denial of service attack.
> 
> 
>>> Hi guys,
>>>
>>> happy new year !!!
>>>
>>> I am trying to updload at least 10000 files on diverse storage elements of EGEE (srm and gridftp).
>>>
>>> To do that I use the geclipse implemetation of EFS (copy method) and 200 threads.
>>>
>>> At the begining everything is fine and uploads are very fast, but after a while my application
>>> get very slow and freeze.
>>>
>>> Many thread have been created: 1031 active threads instead of around 200.
>>>
>>> Most of them are waiting:
>>>
>>> **************************************
>>> java.lang.Object.wait(Native Method)
>>> java.lang.Object.wait(Object.java:485)
>>> org.globus.ftp.dc.TaskThread$Buffer.get(TaskThread.java:122)
>>> org.globus.ftp.dc.TaskThread.getNextTask(TaskThread.java:55)
>>> org.globus.ftp.dc.TaskThread.run(TaskThread.java:66)
>>> java.lang.Thread.run(Thread.java:619)
>>> **************************************
>>>
>>> I have got also many timer threads (I suppose it is for the timout):
>>>
>>> **************************************
>>> java.lang.Object.wait(Native Method)
>>> java.util.TimerThread.mainLoop(Timer.java:509)
>>> java.util.TimerThread.run(Timer.java:462)
>>> **************************************
>>>
>>> And my user threads are waiting to :( also I have lot of network bandwidth available:
>>>
>>> **************************************
>>> java.net.SocketInputStream.socketRead0(Native Method)
>>> java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>>> org.globus.ftp.extended.GridFTPInputStream.readLine(GridFTPInputStream.java:35)
>>> org.globus.ftp.extended.GridFTPInputStream.readMsg(GridFTPInputStream.java:103)
>>> org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:86)
>>> org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:60)
>>> sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
>>> sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
>>> sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
>>> java.io.InputStreamReader.read(InputStreamReader.java:167)
>>> java.io.BufferedReader.fill(BufferedReader.java:136)
>>> java.io.BufferedReader.readLine(BufferedReader.java:299)
>>> java.io.BufferedReader.readLine(BufferedReader.java:362)
>>> org.globus.ftp.vanilla.Reply.<init>(Reply.java:71)
>>> org.globus.ftp.vanilla.FTPControlChannel.read(FTPControlChannel.java:342)
>>> org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:285)
>>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:107)
>>> org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:92)
>>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.connect(GridFTPConnection.java:187)
>>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.getClient(GridFTPConnection.java:476)
>>> eu.geclipse.efs.gridftp.internal.GridFTPConnection.asyncPut(GridFTPConnection.java:716)
>>> eu.geclipse.efs.gridftp.GridFile.openOutputStream(GridFile.java:437)
>>> org.eclipse.core.filesystem.provider.FileStore.copyFile(FileStore.java:218)
>>> org.eclipse.core.filesystem.provider.FileStore.copy(FileStore.java:140)
>>> org.eclipse.core.internal.filesystem.local.LocalFile.copy(LocalFile.java:109)
>>> org.simexplorer.efsfile.EFSFile.copy(EFSFile.java:215)
>>> org.simexplorer.applicationloader.internal.EFSSimulationSerializerImpl.saveJar(EFSSimulationSerializerImpl.java:145)
>>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.uploadInputFile(GEclipseSimExplorerExecutionJob.java:208)
>>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getInputFile(GEclipseSimExplorerExecutionJob.java:244)
>>> org.simexplorer.geclipse.GEclipseJobBuilder.getJobDescription(GEclipseJobBuilder.java:37)
>>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.getDescription(GEclipseSimExplorerExecutionJob.java:265)
>>> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.getDescription(GEclipseEnvironmentExecutionJob.java:33)
>>> org.simexplorer.geclipse.GEclipseEnvironmentExecutionJob.submitJob(GEclipseEnvironmentExecutionJob.java:39)
>>> org.simexplorer.geclipse.GEclipseExecutionJob.submit(GEclipseExecutionJob.java:190)
>>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.submit(GEclipseSimExplorerExecutionJob.java:305)
>>> org.simexplorer.geclipse.GEclipseSimExplorerExecutionJob.update(GEclipseSimExplorerExecutionJob.java:68)
>>> org.simexplorer.updater.UpdaterTask.run(UpdaterTask.java:33)
>>> org.simexplorer.tools.PoolExecutor.run(PoolExecutor.java:22)
>>> java.lang.Thread.run(Thread.java:619)
>>> **************************************
>>>
>>>
>>> Do you have any idea how to tackle this problem?
>>>
>>> Is it an implementation error in globus ftp implementation?
>>>
>>> Cheers,
>>> Romain.
>> _______________________________________________
>> geclipse-dev mailing list
>> geclipse-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/geclipse-dev
> 
>> _______________________________________________
>> geclipse-dev mailing list
>> geclipse-dev@xxxxxxxxxxx
>> https://dev.eclipse.org/mailman/listinfo/geclipse-dev
> 
_______________________________________________
geclipse-dev mailing list
geclipse-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/geclipse-dev

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkllrNAACgkQbUN+hKKXqgsLaQCfbrh4OkzPNDByGxKJhOyQD2Gl
9q0AoMq4rvL98LI3MOIF8agotvRowQlt
=Tasm
-----END PGP SIGNATURE-----

Back to the top