Community
Participate
Working Groups
I have a project of about 500 files residing on a network drive (samba). The output folder is also there. Too often, eclipse decides it should "scrub output folder". It takes too much time (sometimes more than few minues). I can usually do the same operation (delete the output folder) within few seconds. It usually happens after VCM operation, or changing the build path.
What build are you using ? What OS ?
Moving to Platform/Core, we simply use the IFile#delete implementation.
It happens to me on windows NT and XP, with the last stable build (14Feb). Also my colleagues complain that the compilation is slow and that "scrubbing output folder" sometimes happens on normal file save (that should only trigger compilation). I suspect that the file access is somehow not optimized. If you need, I can provide the exact samba server version and configuration
Ok, we discovered some samba configuration deficiency and now it works faster, so the subject can probably change to simply "scrubbing output folder is slow". Nevertheless, I think that deleting the whole directory is faster than deleting it file by file and this method should be used if possible. Some additional info: It seems that when "scrubbing output folder is printed", what really happens is removing all files from the output folder (as expected) and also copying all non-source code resources to the output folder!
I checked and found that deleting files in the command window (win2000) is about twice faster than doing it from java. Maybe you want to consider using a native function to delete a subtree. This would be only one (expensive) JNI call instead of hundrends/thousands. P.S. This is my test code (tested on 2000 files) File dir = new File(dirName); File[] list = dir.listFiles(); //Arrays.sort(list); long l1 = System.currentTimeMillis(); for (int i = 0; i < list.length; i++) { File file = list[i]; file.delete(); } long l2 = System.currentTimeMillis(); System.out.println("Time: " + (l2-l1));
*** Bug 12119 has been marked as a duplicate of this bug. ***
Just to describe the problem we had: the samba loglevel was set to 3. Setting it to 1 definetely made things work faster.
Are you still seeing poor performance in the latest builds or has it improved?
I tried it with the latest build, but no change. It will take Eclipse to scrub the output 55 seconds, when Explorer will delete it in 15 seconds.
It is better, but still quite slow. I suspect eclipse accesses many (very many) files during "scrubbing" + full compilation. Genady
I think this bug/feaure should be given higher priority. Since the build on network drives is slow, it encourages people to do everything on the local drives which are not being backed up. This practice will sooner or later lead to data loss.
There is current (high-priority) investigation on allowing users to change the structure of their project. Specifically one of the results will (hopefully) allow the bin directory to be (for instance) on a local drive and the rest of the project data on a server.
That's true, but in some cases it is not applicable. For example if a user wants to work on the same eclipse workspace on different computers, he has no choice. I don't say it's a high priority request, but I think it should be considered for 2.1.
I have the same issue with a Clearcase network drive. An alternative solution is to allow the use of a build directory that doesn't belong to the project space, so the build classes will be stored on the local drive, which is acceptable as they don't need backup.
is this bug being investigated by the new performance team ?
I believe #32146 can be considered as a duplicate of this bug?
I meant bug #32146.. Here is another observation (I believe that at least some of the file access operations are unnecessary). Note that all these are OS operations, which means each of them requires java to do a native call, and then do a system call. Combination of these two is very slow. I did a simple test of compiling (saving) one file, in a project that has only two files. Now, I have used a file access monitor program, and it shows that compiling the file results in about ~10,000 OS-level file access operations: Here is the list (remember - it is a result of compiling just one! file): Number of accesses ---- File name --------- 8526 C:\jdk-1.4.1\jre\lib\rt.jar 416 C:\jdk-1.4.1\jre\lib\jsse.jar 177 C:\jdk-1.4.1\jre\lib\ext\localedata.jar 99 C:\jdk-1.4.1\jre\lib\sunrsasign.jar 78 C:\jdk-1.4.1\jre\lib\ext\sunjce_provider.jar 68 ...\workspace\rmi-test4\RemoteClass.java 61 ...\workspace\.metadata\.plugins\org.eclipse.jdt.core\1291816999.index 54 ...\workspace\rmi-test4\ 53 C:\jdk-1.4.1\jre\lib\jce.jar 46 ...\workspace\.metadata\.plugins\org.eclipse.jdt.core\1291816999.indexTempVA 30 C:\ 24 ...\workspace\ 24 C:\eclipse\ 20 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\.index. log 20 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\.index 18 ...\workspace\.metadata\.plugins\ 18 ...\workspace\.metadata\ 13 ...\workspace\rmi-test4\RemoteClass.class 12 ...\workspace\rmi-test4 10 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\5c 9 ...\workspace\.metadata\.plugins\org.eclipse.jdt.core\ 9 ..\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\5c\00b43f f40282001716c5bbf66f15be0b 9 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\ 9 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\ 6 C:\jdk-1.4.1\jre\lib\ 6 C:\jdk-1.4.1\jre\ 6 C:\jdk-1.4.1\ 6 ...\workspace\.metadata\.plugins\org.eclipse.jdt.core\savedIndexNames.txt 6 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.history\5c\ 5 C:\jdk-1.4.1\jre\lib\charsets.jar 3 C:\jdk-1.4.1\jre\lib\ext\ldapsec.jar 3 C:\jdk-1.4.1\jre\lib\ext\dnsns.jar 3 ...\workspace\rmi-test4\MyRemote.class 3 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.projects\rmi- test4\org.eclipse.jdt.core 2 C:\Recycler\NProtect\00011130.IND 2 C:\Recycler\NProtect\00011126.LOG 1 ...\workspace\.metadata\.plugins\org.eclipse.core.resources\.projects\rmi- test4\org.eclipse.jdt.core\state.dat Genady
*** Bug 77245 has been marked as a duplicate of this bug. ***
So what is the status of this issue? There has been no activity on this bug entry for almost 18 months. I just created a new entry for general slowness of deletion (not Samba-releated) and it was marked as a duplicate of this one. That's fine, except that this one seems to be lost in the void. Is there any real active research or dev going on here? It is a real pain to have to go out of Eclipse every time I want to clean my large build directories.
Has anybody tried profiling this with YourKit or equivalent? I haven't, but just by watching the package explorer redraw and flash during a length deletion my theory is that Eclipse is trying to do some refreshes and builds and cvs re-decoration *during* the delete operation, which makes it take many times longer than it should. I don't think this should be marked as an enhancement request. It's a performance bug. It'd be a good candidate for adding to the performance test suite too (not sure if that needs another bug entry or not).
I only solved this problem by working with local workspace. The original problem of network workspaces still exists. In another bug report/feature request I suggested renaming the output directory and deleting it in another thread. Maybe it should be reconsidered.
*** Bug 81405 has been marked as a duplicate of this bug. ***
*** Bug 32146 has been marked as a duplicate of this bug. ***
After investigation we decided not to pursue this for 3.1. Most operating systems don't seem to have a recursive deletion native. Thus, writing our own native to delete N files will still require N OS calls. This would only eliminate the cost of the JNI calls, which in the case of slow (network) file systems, is not the principle bottleneck. I'll mark this as "LATER" since it is a valid request, but it's just not clear we can do much better than what java.io.File#delete provides us with today.
First let me say that I really appreciate the explanation of the status of this bug - it is quite refreshing compared to the usual "Not going to address at this time." that most bugzilla entries seem to get. Having said that, I want to point out, in the context of the explanation, that this is a problem even with local drives, not just with slow file systems. My point is that it seems part of the justification for leaving this unaddressed for now was because the bottleneck was the remote file system IO, not the Java->native jump. I would argue that for local file systems the repeated Java->native jumps IS a significant portion of the overall time. Thus there would be a noticable benefit to writing a native recursive (or N-files) delete function.
Just thought of another idea; although it is a little bit of a hack it might be an easier-to-implement interim solution. I think that for most OSes, moving/renaming a directory is a small operation compared to recursive delete. So Eclipse could, when a large directory tree is being deleted, actually just relocate/rename it to a hidden "to-be-purged" location in the workspace. An accompanying background job could clean stuff up out of that "to-be-purged" location, in the background so it doesn't interfer with the user's activity.
An interesting idea. However, I think this will only help with the local disk case. java.io.File#renameTo is very fast when moving a directory to another location on the same disk, but will almost certainly be slower than deletion if it forces content to be copied to another disk or across a network. My question for you: in the local disk case, now that deletion runs in a background thread, how common is it that the deletion speed in Eclipse holds you up? Do you frequently delete very large amounts of data? As a quick test, I deleted a project (org.eclipse.jdt.ui), with 2000 files, 280 folders, for a total of 14 MB. Deletion from the resource navigator took just over 5 seconds. Are you running on a laptop or other machine with slow disk?
I also think it's a good idea - see comment #21 (refers to bug 37417) (In reply to comment #26) > Just thought of another idea; although it is a little bit of a hack it might be > an easier-to-implement interim solution. > I think that for most OSes, moving/renaming a directory is a small operation > compared to recursive delete. So Eclipse could, when a large directory tree is > being deleted, actually just relocate/rename it to a hidden "to-be-purged" > location in the workspace. An accompanying background job could clean stuff up > out of that "to-be-purged" location, in the background so it doesn't interfer > with the user's activity.
(In reply to comment #27) > My question for you: in the local disk case, now that deletion runs in a > background thread, how common is it that the deletion speed in Eclipse holds you > up? Do you frequently delete very large amounts of data? As a quick test, I > deleted a project (org.eclipse.jdt.ui), with 2000 files, 280 folders, for a > total of 14 MB. Deletion from the resource navigator took just over 5 seconds. > Are you running on a laptop or other machine with slow disk? Are you referring to 3.1 when you say that deletion runs in the background? I'm not on 3.1 yet, but that might be enough to entice me... As I said in Bug # 77245, I experimented with a folder that contains 2400 files in 64 subdirectories, totaling 72MB. Windows File Explorer took on average 5-7 seconds to delete that folder. Eclipse Resource Navigator took more than 100 seconds. Not an otherwise slow disk (normal desktop, most likely a 7200 RPM IDE drive).
> Are you referring to 3.1 when you say that deletion runs in the background? I'm > not on 3.1 yet, but that might be enough to entice me... Yes, this is only in Eclipse 3.1 M5 or greater.
On linux GTK and eclipse 3.1 M6 I see deletion of a folder with 2000 files *very slow* and freezing the UI.
Adam, how are you deleting the folder, from the Package explorer, Navigator, or elsewhere? Deletion is usually executed as a background job, so it should never block the UI no matter how long it takes. Some deletions in the Package Explorer execute as refactorings, which may take longer and don't run in the background.
*** Bug 102727 has been marked as a duplicate of this bug. ***
[LATER->WONTFIX] The "LATER" bugzilla resolution is being removed so reopening to mark as WONTFIX.
