Bug 178551 - [index] Deadlock when doing Type Hierarchy while updating a large workspace
Summary: [index] Deadlock when doing Type Hierarchy while updating a large workspace
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.3   Edit
Hardware: PC Windows XP
: P3 major (vote)
Target Milestone: 3.3 M7   Edit
Assignee: Frederic Fusier CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 181459 181475 187861 (view as bug list)
Depends on:
Blocks:
 
Reported: 2007-03-21 10:27 EDT by Martin Oberhuber CLA
Modified: 2007-05-19 06:36 EDT (History)
8 users (show)

See Also:


Attachments
Full thread dump of the deadlock (13.86 KB, text/plain)
2007-03-21 10:28 EDT, Martin Oberhuber CLA
no flags Details
Portions of .log during the time the deadlock happened (72.87 KB, text/plain)
2007-03-21 13:33 EDT, Martin Oberhuber CLA
no flags Details
Proposed patch (1.06 KB, patch)
2007-04-05 08:55 EDT, Frederic Fusier CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Oberhuber CLA 2007-03-21 10:27:53 EDT
Just switched to I20070320-0010 on a large PDE workspace (about 60 plugins).
I'm running on Windows XP SP1 with Sun 1.5.0_11 JVM.

Selected all plugins, did Team > Update to initiate a CVS update of the entire workspace. Place update into background. I'm pretty sure that the update did actually affect mostly binary files and only very few .java files.

While the update was still going on, I selected a class that has a large type hierarchy (about 10 interfaces implemented) in the editor, and pressed F4. Cursor became a sandclock and would not wake up again. UI is unusable except for the "Cancel" button in the lower window trim, but pressing it has no effect.

Java Stacktrace of the deadlock will be attached. Looks like the problem is a nested event loop with a ModalContext Thread which is locked by the indexer. Therefore assigning to JDT-UI (need to get rid of the nested event loop, I guess)
Comment 1 Martin Oberhuber CLA 2007-03-21 10:28:48 EDT
Created attachment 61547 [details]
Full thread dump of the deadlock
Comment 2 Martin Oberhuber CLA 2007-03-21 10:30:39 EDT
PS eclipse.exe runs at 50% CPU on my hyperthreaded machine, so it's taking 100% of one virtual core.
Comment 3 Dani Megert CLA 2007-03-21 12:56:56 EDT
The dump doesn't look like a deadlock but the UI being blocked by the indexer still doing work.
Comment 4 Martin Oberhuber CLA 2007-03-21 13:13:11 EDT
I waited for more than 30 minutes. Never had an indexer take that long.
Comment 5 Martin Aeschlimann CLA 2007-03-21 13:18:24 EDT
I also can't see anything special in this dump. It seems the update thread has already ended and the indexer is just working.
I'm expecting here that the type hierarchy shows the progress bar in the status bar and it is possible to cancel.

I've never seen an indexer taking 30 minutes. Yes, that's strange.

Can't do much without more information. Setting to remind.
Comment 6 Dani Megert CLA 2007-03-21 13:20:48 EDT
Maybe the indexer ran into a loop?
Comment 7 Frederic Fusier CLA 2007-03-21 13:24:34 EDT
Can you take several thread dumps and see if the stack trace of the indexer job is different in these several dumps or always at the same place (ie. DiskIndex.readStreamChars(DiskIndex.java:853)?
Thanks
Comment 8 Martin Oberhuber CLA 2007-03-21 13:33:09 EDT
Created attachment 61582 [details]
Portions of .log during the time the deadlock happened

Attached are portions of my .log file for the time when the deadlock happened.
I could not find anything that looked connected to the deadlock.

I did indeed see the progress bar operate in the window trim. It's cancel button reacted to mouse-over and I could press it, but the Workbench did not react.

All UI was totally frozen except for the progress bar and cancel button.
Not a single menu item would react in any way.
Comment 9 Martin Oberhuber CLA 2007-03-21 13:35:49 EDT
I'm afraid I can't since this was a one-time thing which I don't think I can reproduce. Since I needed my workspace I had to kill the workbench... next time, I'm going to do more than one thread dump. Sorry.
Comment 10 Dani Megert CLA 2007-03-21 13:36:14 EDT
That's expected from looking at the stack trace as the main thread is waiting for the TH.
Comment 11 Frederic Fusier CLA 2007-04-04 11:48:30 EDT
Reopen as we got this issue while starting the monster workspace used for bug 146577...
Comment 12 Frederic Fusier CLA 2007-04-04 13:30:34 EDT
Problem is effectively in DiskIndex.readStreamChars(FileInputStream) method.
In the case of monster workspace, while reading a disk index file, we got following numbers while entering the while (i<length) loop:
this.bufferEnd = 2048
this.bufferIndex = 2046
So, charsInBuffer is equals to i as ((this.bufferEnd - this.bufferIndex) / 3) = 0!
Then we drop in an infinite loop.. :-(

It sounds like the test of the second loop is invalid:
	while (i < charsInBuffer) {
charsInBuffer is only an estimated size to see if the rest of the buffer can contain all the word remaining characters, it should not be used to test to loop inside the buffer. It appears that in this case we loop infinitely but on other cases, we surely miss several characters at the end of the buffer.

I guess we pass all Search tests due to the fact that our disk index files are small (we do not use the real complete rt.jar file in our tests)
Comment 13 Frederic Fusier CLA 2007-04-04 13:39:44 EDT
Forget my previous comment.... readStreamBuffer(FileInputStream) is shifting the unread characters at the end of the buffer to the beginning => this should be OK...
I'll continue to investigate this failure tomorrow morning using monster workspace on David's box where the infinite loop was always reproducible.
Comment 14 Frederic Fusier CLA 2007-04-05 05:27:14 EDT
I also reproduced it using the monster workspace available on our Numbat drive in Workspaces root folder.

Here's the scenario:
1) Unzip the monster workspace
2) Start an eclipse session on this wksp
3) Wait for the end of Java Tooling Initialization
4) Open type
5) Enter 'a' => after a while of searching indexes, CPU heats at 100% and never 
   stops. It was not possible to quit eclipse, you have to kill the session!

DiskIndex loops on readStreamChars because in this case, the stream parameter is null when call stack is:
 - DiskIndex.readStreamChars(FileInputStream) line: 851	
 - DiskIndex.readChunk(String[], FileInputStream, int, int) line: 681	
 - DiskIndex.readDocumentName(int) line: 725	
 - EntryResult.getDocumentNames(Index) line: 53	
 - ...
Then, readStreamBuffer(FileInputStream) is never called, so charsInBuffer never changes and loop runs infinitely...
Comment 15 Frederic Fusier CLA 2007-04-05 08:55:44 EDT
Created attachment 63027 [details]
Proposed patch
Comment 16 Frederic Fusier CLA 2007-04-05 08:56:09 EDT
Released for 3.3 M7 in HEAD stream.
Comment 17 Frederic Fusier CLA 2007-04-07 04:52:35 EDT
*** Bug 181475 has been marked as a duplicate of this bug. ***
Comment 18 Karen Butzke CLA 2007-04-13 16:17:20 EDT
*** Bug 181459 has been marked as a duplicate of this bug. ***
Comment 19 Frederic Fusier CLA 2007-04-27 02:13:11 EDT
*** Bug 181459 has been marked as a duplicate of this bug. ***
Comment 20 Jerome Lanneluc CLA 2007-04-27 10:39:46 EDT
Verified for 3.3M7 for I20070427-0010
Comment 21 Jerome Lanneluc CLA 2007-05-19 06:36:28 EDT
*** Bug 187861 has been marked as a duplicate of this bug. ***