Community
Participate
Working Groups
+++ This bug was initially created as a clone of Bug #190803 +++ * connected to a Solaris dstore-unix server (szg-anar, Running). * Expand "Root" filter * Expand "/folk" fokder --> "Pending" message is shown --> After about 2:30 minutes it returns but shows an empty folder. No exception messages are shown or logged, neither on the client nor on the server. Doing the same on the same host through an SSH Only connection returns a result of 3361 folder children within about 3 seconds. It looks like the dstore daemon tries to refresh the children of "/folk" two levels deep which takes too long. Is there any kind of timeout? Client is RSE 2.0RC2 on WinXP / eclipse 3.3RC3 / Sun 1.6.0_01. Server is RSE 2.0RC2 on Solaris / Sun 1.4.2_05.
Dave - how can we add debugging information to understand what's going on here?
I added some trace statement inside the DStore server code to track time. And I created a folder with around 40000 subfolders, and the following is the time recorded: Scenario 1: Expand the folder (dummy) for the first time: . UniversalFileSystemMiner#handleQueryAll - time elapsed: 90,668 ms - internalQueryAll - 23143 ms -- createDataElement() - 8,084 ms -- clsfy.start() - 11,984 ms - statusDone() - 67,002 ms -- mainly is in DataStore#refresh() method called by element isdone So in this scenario, the bottleneck is the statusDone() method. Scenario 2: Refresh the folder (dummy): UniversalFileSystemMiner#handleQueryAll - time elapsed: 1539,577 ms - internalQueryAll - 1469,480 ms -- createDataElement() - 1448,837 ms -- clsfy.start() - 11,933 ms - statusDone() - 69,870 ms So in scenario #2, the bottleneck is the createDataElement() method, which caused this refresh operation run for more than 25 minutes. I could see the reason why createDataElement() running for such a long time in this scenario is the nested loop inside this method: // Check if the current Objects in the DataStore are valid... exist // on the remote host try { for (int i = 0; i < currentObjList.length; ++i) { found = false; DataElement previousElement = (DataElement) currentObjList[i]; for (int j = 0; j < list.length && !found; ++j) { . . .
For Scenario #1, the bottleneck is the XMLgenerator#generate() method call for dataelement dummy(the big folder). Please see the attachment for the sceen capture where I halted the thread.
Created attachment 70698 [details] screen capture for where the bottleneck code is
Created attachment 71085 [details] Fixes to improve the performance of query for large folder I made the following fixes to improve the performance (mainly for the refresh case): . Move the init() method from the constructor of FileClassifer to its start() method . Merge the nested loop and the loop update/create dataelement into one loop through the result of listFile operation. This fix applied for both regular file/folder and virtual file/folder inside a archive file. Legal Message: I, Xuan Chen, declare that I developed attached code from scratch, without referencing any 3rd party materials except material licensed under the EPL. I am authorized by my employer, IBM Canada Ltd. to make this contribution under the EPL.
Please review the patch.
The patch is large - but I get the general idea of what's going on. I approve.
I've applied the patch for this.
Looks ok to me.
I found a problem with the patch for handling the virtual file case. I used : String oldValue = deObj.getAttribute(DE.A_VALUE); String newValue = rootPath + ArchiveHandlerManager.VIRTUAL_SEPARATOR + virtualPath; if (!oldValue.startsWith(newValue)) ^^^^^^^^^^ { deObj.setAttribute(DE.A_VALUE, newValue); //$NON-NLS-1$ } It is not correct if I rename a name from "dummy1" to "dummy" I will provide a patch to fix this problem. I need to change to equals() instead of startsWith(). Sorry about it.
Created attachment 71268 [details] update the problem with rename in virtual file Legal Message: I, Xuan Chen, declare that I developed attached code from scratch, without referencing any 3rd party materials except material licensed under the EPL. I am authorized by my employer, IBM Canada Ltd. to make this contribution under the EPL.
I've committed the update.
[target cleanup] 2.0 RC3 was the original target milestone for this bug
