Bug 413730 - DependencyGraphImpl deadlocks if workspace lock has been acquired on another thread
Summary: DependencyGraphImpl deadlocks if workspace lock has been acquired on another ...
Status: RESOLVED FIXED
Alias: None
Product: WTP Common Tools
Classification: WebTools
Component: wst.common (show other bugs)
Version: 3.4.2   Edit
Hardware: PC Windows 7
: P3 blocker (vote)
Target Milestone: 3.4.2 P   Edit
Assignee: Chuck Bridgham CLA
QA Contact: Carl Anderson CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 416088
  Show dependency tree
 
Reported: 2013-07-25 09:53 EDT by Remy Suen CLA
Modified: 2013-09-11 15:07 EDT (History)
6 users (show)

See Also:
ccc: review+


Attachments
Zip file that reproduces the problem in question. (16.49 KB, application/zip)
2013-07-25 09:53 EDT, Remy Suen CLA
no flags Details
Proposed patch (8.63 KB, patch)
2013-08-02 14:13 EDT, Chuck Bridgham CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Remy Suen CLA 2013-07-25 09:53:59 EDT
Created attachment 233796 [details]
Zip file that reproduces the problem in question.

I'm on 3.8.2 and have installed WTP plug-ins from the Juno update site.

-----------------

1. Unzip the attached zip file.
2. Import deadlock-bug into your outer Eclipse.
3. Start your inner Eclipse and import bug-setup into that other workspace.
4. Refresh the many folders of bug-setup to ensure that everything is in synch with what's on disk.
5. Close your inner Eclipse.
6. Open WorkbenchURIConverterImpl and find the createPlatformResourceInputStream(*) method.
7. Put a breakpoint in the isSynchronized(*) call contained within the if statement.
8. Start your inner Eclipse. It should hit the breakpoint that we just added.

Thread [Worker-2] (Suspended (breakpoint at line 497 in WorkbenchURIConverterImpl))	
	owns: ModuleStructuralModel  (id=79)	
	ComponentCoreURIConverter(WorkbenchURIConverterImpl).createPlatformResourceInputStream(String) line: 497	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI) line: 435	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI, Map<?,?>) line: 451	
	WTPModulesResource(ResourceImpl).load(Map<?,?>) line: 1256	
	WTPModulesResource(CompatibilityXMIResourceImpl).load(Map) line: 272	
	WTPModulesResource(TranslatorResourceImpl).load(Map) line: 423	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoad(Resource) line: 259	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).demandLoad(Resource) line: 811	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoadHelper(Resource) line: 274	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).getResource(URI, boolean) line: 1064	
	WorkbenchResourceHelper.getOrCreateResource(URI, ResourceSet) line: 380	
	ModuleStructuralModel(EditModel).getResource(URI) line: 685	
	ModuleStructuralModel.getPrimaryResource() line: 332	
	ModuleStructuralModel.prepareProjectModulesIfNecessary() line: 240	
	ModuleStructuralModel.getPrimaryRootObject() line: 119	
	StructureEdit.getComponentModelRoot() line: 471	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	DependencyGraphImpl$GraphUpdateJob$1.run() line: 494	
	SafeRunner.run(ISafeRunnable) line: 42	
	DependencyGraphImpl$GraphUpdateJob.run(IProgressMonitor) line: 462	
	Worker.run() line: 54

9. Open the 'Progress' view.
10. Ensure all other startup-related operations have completed.
11. Put a breakpoint in StructureEdit's getComponentModelRoot() method.
12. Click the 'Initiate Deadlock' button in the workbench window's toolbar.
13. A breakpoint should be hit.

Thread [Worker-3] (Suspended (breakpoint at line 461 in StructureEdit))	
	owns: DeadlockOperation  (id=138)	
	StructureEdit.getComponentModelRoot() line: 461	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	J2EEElementChangedListener.processJavaProject(IJavaProject, int, int, IJavaElementDelta) line: 101	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 87	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 80	
	J2EEElementChangedListener.elementChanged(ElementChangedEvent) line: 67	
	DeltaProcessor$4.run() line: 1682	
	SafeRunner.run(ISafeRunnable) line: 42	
	DeltaProcessor.notifyListeners(IJavaElementDelta, int, IElementChangedListener[], int[], int) line: 1672	
	DeltaProcessor.firePostChangeDelta(IJavaElementDelta, IElementChangedListener[], int[], int) line: 1506	
	DeltaProcessor.fire(IJavaElementDelta, int) line: 1482	
	DeltaProcessor.resourceChanged(IResourceChangeEvent) line: 2094	
	DeltaProcessingState.resourceChanged(IResourceChangeEvent) line: 470	
	NotificationManager$1.run() line: 291	
	SafeRunner.run(ISafeRunnable) line: 42	
	NotificationManager.notify(ResourceChangeListenerList$ListenerEntry[], IResourceChangeEvent, boolean) line: 285	
	NotificationManager.broadcastChanges(ElementTree, ResourceChangeEvent, boolean) line: 149	
	Workspace.broadcastPostChange() line: 395	
	Workspace.endOperation(ISchedulingRule, boolean, IProgressMonitor) line: 1530	
	Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2353	
	DeadlockOperation(WorkspaceModifyOperation).run(IProgressMonitor) line: 118	
	DeadlockAction$1.run(IProgressMonitor) line: 31	
	Worker.run() line: 54

14. Go back to the first thread that's suspended in WorkbenchURIConverterImpl.
15. F6 through the if statement. It should detect that the file is out-of-sync and try to refresh. The refresh will hang.

Thread [Worker-2] (Suspended)	
	owns: ModuleStructuralModel  (id=79)	
	waiting for: Object  (id=162)	
	Object.wait(long) line: not available [native method]	
	Object.wait() line: 503	
	ThreadJob.waitForRun(ThreadJob, IProgressMonitor, InternalJob, Thread) line: 270	
	ThreadJob.joinRun(ThreadJob, IProgressMonitor) line: 197	
	ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) line: 92	
	JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 286	
	WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 118	
	Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) line: 2282	
	File(Resource).refreshLocal(int, IProgressMonitor) line: 1691	
	File.refreshLocal(int, IProgressMonitor) line: 333	
	ComponentCoreURIConverter(WorkbenchURIConverterImpl).createPlatformResourceInputStream(String) line: 501	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI) line: 435	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI, Map<?,?>) line: 451	
	WTPModulesResource(ResourceImpl).load(Map<?,?>) line: 1256	
	WTPModulesResource(CompatibilityXMIResourceImpl).load(Map) line: 272	
	WTPModulesResource(TranslatorResourceImpl).load(Map) line: 423	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoad(Resource) line: 259	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).demandLoad(Resource) line: 811	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoadHelper(Resource) line: 274	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).getResource(URI, boolean) line: 1064	
	WorkbenchResourceHelper.getOrCreateResource(URI, ResourceSet) line: 380	
	ModuleStructuralModel(EditModel).getResource(URI) line: 685	
	ModuleStructuralModel.getPrimaryResource() line: 332	
	ModuleStructuralModel.prepareProjectModulesIfNecessary() line: 240	
	ModuleStructuralModel.getPrimaryRootObject() line: 119	
	StructureEdit.getComponentModelRoot() line: 471	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	DependencyGraphImpl$GraphUpdateJob$1.run() line: 494	
	SafeRunner.run(ISafeRunnable) line: 42	
	DependencyGraphImpl$GraphUpdateJob.run(IProgressMonitor) line: 462	
	Worker.run() line: 54

16. Go back to the second thread from DeadlockOperation.
17. F6 through the code. It should stall when trying to acquire the lock.

Thread [Worker-3] (Suspended)	
	owns: DeadlockOperation  (id=138)	
	waiting for: Semaphore  (id=169)	
	Object.wait(long) line: not available [native method]	
	Semaphore.acquire(long) line: 39	
	OrderedLock.doAcquire(Semaphore, long) line: 176	
	OrderedLock.acquire(long) line: 110	
	OrderedLock.acquire() line: 84	
	StructureEdit.getComponentModelRoot() line: 467	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	J2EEElementChangedListener.processJavaProject(IJavaProject, int, int, IJavaElementDelta) line: 101	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 87	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 80	
	J2EEElementChangedListener.elementChanged(ElementChangedEvent) line: 67	
	DeltaProcessor$4.run() line: 1682	
	SafeRunner.run(ISafeRunnable) line: 42	
	DeltaProcessor.notifyListeners(IJavaElementDelta, int, IElementChangedListener[], int[], int) line: 1672	
	DeltaProcessor.firePostChangeDelta(IJavaElementDelta, IElementChangedListener[], int[], int) line: 1506	
	DeltaProcessor.fire(IJavaElementDelta, int) line: 1482	
	DeltaProcessor.resourceChanged(IResourceChangeEvent) line: 2094	
	DeltaProcessingState.resourceChanged(IResourceChangeEvent) line: 470	
	NotificationManager$1.run() line: 291	
	SafeRunner.run(ISafeRunnable) line: 42	
	NotificationManager.notify(ResourceChangeListenerList$ListenerEntry[], IResourceChangeEvent, boolean) line: 285	
	NotificationManager.broadcastChanges(ElementTree, ResourceChangeEvent, boolean) line: 149	
	Workspace.broadcastPostChange() line: 395	
	Workspace.endOperation(ISchedulingRule, boolean, IProgressMonitor) line: 1530	
	Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2353	
	DeadlockOperation(WorkspaceModifyOperation).run(IProgressMonitor) line: 118	
	DeadlockAction$1.run(IProgressMonitor) line: 31	
	Worker.run() line: 54

18. The deadlock is now in place. You can see that the 'Progress' view is stalled.

Basically, thread A acquires a lock on StructureEdit and then thread B acquires a workspace lock. Now thread A tries to refresh the file but it can't because thread B has a workspace lock. Meanwhile, thread B wants to talk to StructureEdit but it can't because thread A's got a lock on it. It seems to be that any asynchronous operation on thread B that locks the workspace (or just that one file really) as thread A is trying to refresh the file will cause a deadlock.
Comment 1 John Arthorne CLA 2013-07-31 13:21:12 EDT
This is the classic resource change event deadlock. Someone is trying to acquire a lock from within a resource change event callback, that is also sometimes held while modifying resources. From the contract of resource change events, this could only be resolved at the platform level by allowing resource changes to occur in other threads during resource change events, which would be a breaking change for listeners (I tried this many years ago and it was not successful).  There are two basic ways to fix this:

 - Don't lock StructureEdit during resource change events. The classic solution here is to pull the data you need out of the resource change event, and perform your model updates asynchronously (e.g., in another job).

 - Don't hold StructureEdit lock while modifying resources. This is probably the better option. Holding this lock while modifying resources exposes you to various forms of deadlock because arbitrary third party code can be executed within the context of a resource change.
Comment 2 Chuck Bridgham CLA 2013-08-02 14:13:44 EDT
Created attachment 234066 [details]
Proposed patch

Proposed patch that moves all required processing using Component models inside a Job.
Comment 3 Carl Anderson CLA 2013-09-11 15:07:14 EDT
Committed/released to R3_4_2_patches