Bug 416088 - DependencyGraphImpl deadlocks if workspace lock has been acquired on another thread
Summary: DependencyGraphImpl deadlocks if workspace lock has been acquired on another ...
Status: RESOLVED FIXED
Alias: None
Product: WTP Java EE Tools
Classification: WebTools
Component: jst.j2ee (show other bugs)
Version: 3.5.1   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 3.5.1   Edit
Assignee: Chuck Bridgham CLA
QA Contact: Chuck Bridgham CLA
URL:
Whiteboard: PMC_approved
Keywords:
Depends on: 413730
Blocks:
  Show dependency tree
 
Reported: 2013-08-28 15:05 EDT by Chuck Bridgham CLA
Modified: 2013-09-05 11:01 EDT (History)
9 users (show)

See Also:
cbridgha: pmc_approved? (david_williams)
raghunathan.srinivasan: pmc_approved+
cbridgha: pmc_approved? (naci.dai)
neil.hauge: pmc_approved+
cbridgha: pmc_approved? (cbridgha)
cbridgha: pmc_approved? (kaloyan)

Attachments
Patch (8.63 KB, patch)
2013-08-28 15:47 EDT, Chuck Bridgham CLA 		no flags Details | Diff
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Chuck Bridgham CLA 2013-08-28 15:05:09 EDT 
+++ This bug was initially created as a clone of Bug #413730 +++

I'm on 3.8.2 and have installed WTP plug-ins from the Juno update site.

-----------------

1. Unzip the attached zip file.
2. Import deadlock-bug into your outer Eclipse.
3. Start your inner Eclipse and import bug-setup into that other workspace.
4. Refresh the many folders of bug-setup to ensure that everything is in synch with what's on disk.
5. Close your inner Eclipse.
6. Open WorkbenchURIConverterImpl and find the createPlatformResourceInputStream(*) method.
7. Put a breakpoint in the isSynchronized(*) call contained within the if statement.
8. Start your inner Eclipse. It should hit the breakpoint that we just added.

Thread [Worker-2] (Suspended (breakpoint at line 497 in WorkbenchURIConverterImpl))	
	owns: ModuleStructuralModel  (id=79)	
	ComponentCoreURIConverter(WorkbenchURIConverterImpl).createPlatformResourceInputStream(String) line: 497	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI) line: 435	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI, Map<?,?>) line: 451	
	WTPModulesResource(ResourceImpl).load(Map<?,?>) line: 1256	
	WTPModulesResource(CompatibilityXMIResourceImpl).load(Map) line: 272	
	WTPModulesResource(TranslatorResourceImpl).load(Map) line: 423	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoad(Resource) line: 259	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).demandLoad(Resource) line: 811	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoadHelper(Resource) line: 274	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).getResource(URI, boolean) line: 1064	
	WorkbenchResourceHelper.getOrCreateResource(URI, ResourceSet) line: 380	
	ModuleStructuralModel(EditModel).getResource(URI) line: 685	
	ModuleStructuralModel.getPrimaryResource() line: 332	
	ModuleStructuralModel.prepareProjectModulesIfNecessary() line: 240	
	ModuleStructuralModel.getPrimaryRootObject() line: 119	
	StructureEdit.getComponentModelRoot() line: 471	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	DependencyGraphImpl$GraphUpdateJob$1.run() line: 494	
	SafeRunner.run(ISafeRunnable) line: 42	
	DependencyGraphImpl$GraphUpdateJob.run(IProgressMonitor) line: 462	
	Worker.run() line: 54

9. Open the 'Progress' view.
10. Ensure all other startup-related operations have completed.
11. Put a breakpoint in StructureEdit's getComponentModelRoot() method.
12. Click the 'Initiate Deadlock' button in the workbench window's toolbar.
13. A breakpoint should be hit.

Thread [Worker-3] (Suspended (breakpoint at line 461 in StructureEdit))	
	owns: DeadlockOperation  (id=138)	
	StructureEdit.getComponentModelRoot() line: 461	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	J2EEElementChangedListener.processJavaProject(IJavaProject, int, int, IJavaElementDelta) line: 101	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 87	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 80	
	J2EEElementChangedListener.elementChanged(ElementChangedEvent) line: 67	
	DeltaProcessor$4.run() line: 1682	
	SafeRunner.run(ISafeRunnable) line: 42	
	DeltaProcessor.notifyListeners(IJavaElementDelta, int, IElementChangedListener[], int[], int) line: 1672	
	DeltaProcessor.firePostChangeDelta(IJavaElementDelta, IElementChangedListener[], int[], int) line: 1506	
	DeltaProcessor.fire(IJavaElementDelta, int) line: 1482	
	DeltaProcessor.resourceChanged(IResourceChangeEvent) line: 2094	
	DeltaProcessingState.resourceChanged(IResourceChangeEvent) line: 470	
	NotificationManager$1.run() line: 291	
	SafeRunner.run(ISafeRunnable) line: 42	
	NotificationManager.notify(ResourceChangeListenerList$ListenerEntry[], IResourceChangeEvent, boolean) line: 285	
	NotificationManager.broadcastChanges(ElementTree, ResourceChangeEvent, boolean) line: 149	
	Workspace.broadcastPostChange() line: 395	
	Workspace.endOperation(ISchedulingRule, boolean, IProgressMonitor) line: 1530	
	Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2353	
	DeadlockOperation(WorkspaceModifyOperation).run(IProgressMonitor) line: 118	
	DeadlockAction$1.run(IProgressMonitor) line: 31	
	Worker.run() line: 54

14. Go back to the first thread that's suspended in WorkbenchURIConverterImpl.
15. F6 through the if statement. It should detect that the file is out-of-sync and try to refresh. The refresh will hang.

Thread [Worker-2] (Suspended)	
	owns: ModuleStructuralModel  (id=79)	
	waiting for: Object  (id=162)	
	Object.wait(long) line: not available [native method]	
	Object.wait() line: 503	
	ThreadJob.waitForRun(ThreadJob, IProgressMonitor, InternalJob, Thread) line: 270	
	ThreadJob.joinRun(ThreadJob, IProgressMonitor) line: 197	
	ImplicitJobs.begin(ISchedulingRule, IProgressMonitor, boolean) line: 92	
	JobManager.beginRule(ISchedulingRule, IProgressMonitor) line: 286	
	WorkManager.checkIn(ISchedulingRule, IProgressMonitor) line: 118	
	Workspace.prepareOperation(ISchedulingRule, IProgressMonitor) line: 2282	
	File(Resource).refreshLocal(int, IProgressMonitor) line: 1691	
	File.refreshLocal(int, IProgressMonitor) line: 333	
	ComponentCoreURIConverter(WorkbenchURIConverterImpl).createPlatformResourceInputStream(String) line: 501	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI) line: 435	
	ComponentCoreURIConverter(URIConverterImpl).createInputStream(URI, Map<?,?>) line: 451	
	WTPModulesResource(ResourceImpl).load(Map<?,?>) line: 1256	
	WTPModulesResource(CompatibilityXMIResourceImpl).load(Map) line: 272	
	WTPModulesResource(TranslatorResourceImpl).load(Map) line: 423	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoad(Resource) line: 259	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).demandLoad(Resource) line: 811	
	ProjectResourceSetEditImpl(ResourceSetImpl).demandLoadHelper(Resource) line: 274	
	ProjectResourceSetEditImpl(ProjectResourceSetImpl).getResource(URI, boolean) line: 1064	
	WorkbenchResourceHelper.getOrCreateResource(URI, ResourceSet) line: 380	
	ModuleStructuralModel(EditModel).getResource(URI) line: 685	
	ModuleStructuralModel.getPrimaryResource() line: 332	
	ModuleStructuralModel.prepareProjectModulesIfNecessary() line: 240	
	ModuleStructuralModel.getPrimaryRootObject() line: 119	
	StructureEdit.getComponentModelRoot() line: 471	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	DependencyGraphImpl$GraphUpdateJob$1.run() line: 494	
	SafeRunner.run(ISafeRunnable) line: 42	
	DependencyGraphImpl$GraphUpdateJob.run(IProgressMonitor) line: 462	
	Worker.run() line: 54

16. Go back to the second thread from DeadlockOperation.
17. F6 through the code. It should stall when trying to acquire the lock.

Thread [Worker-3] (Suspended)	
	owns: DeadlockOperation  (id=138)	
	waiting for: Semaphore  (id=169)	
	Object.wait(long) line: not available [native method]	
	Semaphore.acquire(long) line: 39	
	OrderedLock.doAcquire(Semaphore, long) line: 176	
	OrderedLock.acquire(long) line: 110	
	OrderedLock.acquire() line: 84	
	StructureEdit.getComponentModelRoot() line: 467	
	StructureEdit.getWorkbenchModules() line: 506	
	StructureEdit.getComponent() line: 949	
	J2EEModuleVirtualComponent(VirtualComponent).createResource() line: 124	
	J2EEModuleVirtualComponent(VirtualComponent).initializeResource() line: 113	
	J2EEModuleVirtualComponent(VirtualComponent).<init>(IProject, IPath) line: 148	
	J2EEModuleVirtualComponent.<init>(IProject, IPath) line: 92	
	J2EEModuleVirtualComponent.createComponent(IProject) line: 96	
	ComponentImplManager.createComponent(IProject, boolean) line: 215	
	ComponentImplManager.createComponent(IProject) line: 203	
	ComponentCore.createComponent(IProject) line: 64	
	J2EEElementChangedListener.processJavaProject(IJavaProject, int, int, IJavaElementDelta) line: 101	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 87	
	J2EEElementChangedListener.processJavaElementDelta(IJavaElementDelta) line: 80	
	J2EEElementChangedListener.elementChanged(ElementChangedEvent) line: 67	
	DeltaProcessor$4.run() line: 1682	
	SafeRunner.run(ISafeRunnable) line: 42	
	DeltaProcessor.notifyListeners(IJavaElementDelta, int, IElementChangedListener[], int[], int) line: 1672	
	DeltaProcessor.firePostChangeDelta(IJavaElementDelta, IElementChangedListener[], int[], int) line: 1506	
	DeltaProcessor.fire(IJavaElementDelta, int) line: 1482	
	DeltaProcessor.resourceChanged(IResourceChangeEvent) line: 2094	
	DeltaProcessingState.resourceChanged(IResourceChangeEvent) line: 470	
	NotificationManager$1.run() line: 291	
	SafeRunner.run(ISafeRunnable) line: 42	
	NotificationManager.notify(ResourceChangeListenerList$ListenerEntry[], IResourceChangeEvent, boolean) line: 285	
	NotificationManager.broadcastChanges(ElementTree, ResourceChangeEvent, boolean) line: 149	
	Workspace.broadcastPostChange() line: 395	
	Workspace.endOperation(ISchedulingRule, boolean, IProgressMonitor) line: 1530	
	Workspace.run(IWorkspaceRunnable, ISchedulingRule, int, IProgressMonitor) line: 2353	
	DeadlockOperation(WorkspaceModifyOperation).run(IProgressMonitor) line: 118	
	DeadlockAction$1.run(IProgressMonitor) line: 31	
	Worker.run() line: 54

18. The deadlock is now in place. You can see that the 'Progress' view is stalled.

Basically, thread A acquires a lock on StructureEdit and then thread B acquires a workspace lock. Now thread A tries to refresh the file but it can't because thread B has a workspace lock. Meanwhile, thread B wants to talk to StructureEdit but it can't because thread A's got a lock on it. It seems to be that any asynchronous operation on thread B that locks the workspace (or just that one file really) as thread A is trying to refresh the file will cause a deadlock.
Comment 1 Chuck Bridgham CLA 2013-08-28 15:47:50 EDT 
Created attachment 234883 [details]
Patch
Comment 2 Chuck Bridgham CLA 2013-08-28 15:55:08 EDT 
This is a clone for a bug and fix that went into 342 patches... Now adding to 351

Explain why you believe this is a stop-ship defect. Or, if it is a "hotbug" (requested by an adopter) please document it as such.
This deadlock occurs because this resource change listener class calls out to the Component model, which in turn, creates a workspace lock, and never is allowed to release...


Is there a work-around? If so, why do you believe the work-around is insufficient?
No


How has the fix been tested? Is there a test case attached to the bugzilla record? Has a JUnit Test been added?
no - but has been tested manually, and junit bucket run.


Give a brief technical overview. Who has reviewed this fix?
The patch moves all required processing using Component models inside a Job to be called asynchronously using a workspace root rule.


What is the risk associated with this fix?
No