Community
Participate
Working Groups
From: Remy Suen/Ottawa/IBM@IBMCA To: Thomas Pitzen/Rochester/IBM@IBMUS Cc: Carl Anderson/Raleigh/IBM@IBMUS, Chuck Bridgham/Raleigh/IBM@IBMUS, Jeffrey E Care/Raleigh/IBM@IBMUS, Katie Imming/Rochester/IBM@IBMUS Date: 09/11/2013 02:21 PM Subject: Re: Fw: [Bug 416088] DependencyGraphImpl deadlocks if workspace lock has been acquired on another thread Thanks for providing the thread dump so quickly, Tom! So we've got SCM's accept going on like before in thread A below. We have the workspace locked and it's waiting on StructureEdit. Thread A: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:196(Compiled Code)) at org/eclipse/core/internal/jobs/Semaphore.acquire(Semaphore.java:39(Compiled Code)) at org/eclipse/core/internal/jobs/OrderedLock.doAcquire(OrderedLock.java:176(Compiled Code)) at org/eclipse/core/internal/jobs/OrderedLock.acquire(OrderedLock.java:110(Compiled Code)) at org/eclipse/core/internal/jobs/OrderedLock.acquire(OrderedLock.java:84(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getComponentModelRoot(StructureEdit.java:467(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getWorkbenchModules(StructureEdit.java:506(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getComponent(StructureEdit.java:949(Compiled Code)) at org/eclipse/jst/j2ee/internal/common/classpath/J2EEComponentClasspathUpdater.findNode(J2EEComponentClasspathUpdater.java:477(Compiled Code)) at org/eclipse/jst/j2ee/internal/common/classpath/J2EEComponentClasspathUpdater.findNode(J2EEComponentClasspathUpdater.java:483(Compiled Code)) at org/eclipse/jst/j2ee/internal/common/classpath/J2EEComponentClasspathUpdater.findNode(J2EEComponentClasspathUpdater.java:483(Compiled Code)) at org/eclipse/jst/j2ee/internal/common/classpath/J2EEComponentClasspathUpdater.resourceChanged(J2EEComponentClasspathUpdater.java:434(Compiled Code)) at org/eclipse/core/internal/events/NotificationManager$1.run(NotificationManager.java:291(Compiled Code)) at org/eclipse/core/runtime/SafeRunner.run(SafeRunner.java:42(Compiled Code)) at org/eclipse/core/internal/events/NotificationManager.notify(NotificationManager.java:285(Compiled Code)) at org/eclipse/core/internal/events/NotificationManager.broadcastChanges(NotificationManager.java:149(Compiled Code)) at org/eclipse/core/internal/resources/Workspace.broadcastPostChange(Workspace.java:395(Compiled Code)) at org/eclipse/core/internal/resources/Workspace.endOperation(Workspace.java:1530(Compiled Code)) at org/eclipse/core/internal/resources/Workspace.run(Workspace.java:2353(Compiled Code)) at com/ibm/team/filesystem/rcp/core/internal/resources/ResourceSharingManager.runWithinFileSystemLock(ResourceSharingManager.java:88) at com/ibm/team/filesystem/client/internal/operations/FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:164) at com/ibm/team/filesystem/client/internal/operations/FileSystemOperation.runWithinFileSystemLock(FileSystemOperation.java:144) at com/ibm/team/filesystem/client/internal/operations/WorkspaceUpdateOperation.execute(WorkspaceUpdateOperation.java:714) at com/ibm/team/filesystem/client/internal/operations/FileSystemOperation.run(FileSystemOperation.java:89) at com/ibm/team/filesystem/rcp/core/internal/changes/model/WorkspaceUpdateUtil.accept(WorkspaceUpdateUtil.java:68) at com/ibm/team/filesystem/ui/changes/actions/AcceptAction$2.repositoryRun(AcceptAction.java:262) at com/ibm/team/repository/rcp/ui/operations/RepositoryOperation.run(RepositoryOperation.java:43) at com/ibm/team/repository/rcp/ui/operations/JobRunner.run(JobRunner.java:240) at com/ibm/team/repository/rcp/ui/operations/JobRunner$1.run(JobRunner.java:63) at org/eclipse/core/internal/jobs/Worker.run(Worker.java:54(Compiled Code)) Elsewhere I can see DependencyGraphImpl at work and it's in the refresh code of WorkbenchURIConverterImpl from before. The file can't refresh because of the workspace lock acquired from thread A above. We can see that the thread below is inside StructureEdit so I think it's safe to assume that this thread B has the lock on StructureEdit. Thread B: at java/lang/Object.wait(Native Method) at java/lang/Object.wait(Object.java:167(Compiled Code)) at org/eclipse/core/internal/jobs/ThreadJob.waitForRun(ThreadJob.java:270(Compiled Code)) at org/eclipse/core/internal/jobs/ThreadJob.joinRun(ThreadJob.java:197(Compiled Code)) at org/eclipse/core/internal/jobs/ImplicitJobs.begin(ImplicitJobs.java:92(Compiled Code)) at org/eclipse/core/internal/jobs/JobManager.beginRule(JobManager.java:286(Compiled Code)) at org/eclipse/core/internal/resources/WorkManager.checkIn(WorkManager.java:118(Compiled Code)) at org/eclipse/core/internal/resources/Workspace.prepareOperation(Workspace.java:2282(Compiled Code)) at org/eclipse/core/internal/resources/Resource.refreshLocal(Resource.java:1691(Compiled Code)) at org/eclipse/core/internal/resources/File.refreshLocal(File.java:333) at org/eclipse/jem/util/emf/workbench/WorkbenchURIConverterImpl.createPlatformResourceInputStream(WorkbenchURIConverterImpl.java:501) at org/eclipse/emf/ecore/resource/impl/URIConverterImpl.createInputStream(URIConverterImpl.java:435) at org/eclipse/emf/ecore/resource/impl/URIConverterImpl.createInputStream(URIConverterImpl.java:451) at org/eclipse/emf/ecore/resource/impl/ResourceImpl.load(ResourceImpl.java:1256) at org/eclipse/wst/common/internal/emf/resource/CompatibilityXMIResourceImpl.load(CompatibilityXMIResourceImpl.java:272) at org/eclipse/wst/common/internal/emf/resource/TranslatorResourceImpl.load(TranslatorResourceImpl.java:423) at org/eclipse/wst/common/componentcore/internal/ModuleStructuralModel.getAndLoadLocalResource(ModuleStructuralModel.java:377(Compiled Code)) at org/eclipse/wst/common/internal/emfworkbench/integration/EditModel.getResource(EditModel.java:683(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/ModuleStructuralModel.getPrimaryResource(ModuleStructuralModel.java:332(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/ModuleStructuralModel.prepareProjectModulesIfNecessary(ModuleStructuralModel.java:240(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/ModuleStructuralModel.getPrimaryRootObject(ModuleStructuralModel.java:119(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getComponentModelRoot(StructureEdit.java:471(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getWorkbenchModules(StructureEdit.java:506(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/StructureEdit.getComponent(StructureEdit.java:949(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/resources/VirtualComponent.createResource(VirtualComponent.java:124) at org/eclipse/wst/common/componentcore/internal/resources/VirtualComponent.initializeResource(VirtualComponent.java:113) at org/eclipse/wst/common/componentcore/internal/resources/VirtualComponent.<init>(VirtualComponent.java:148) at org/eclipse/jst/j2ee/componentcore/util/EARVirtualComponent.<init>(EARVirtualComponent.java:68) at org/eclipse/jst/j2ee/componentcore/util/EARVirtualComponent.createComponent(EARVirtualComponent.java:76) at org/eclipse/wst/common/componentcore/internal/util/ComponentImplManager.createComponent(ComponentImplManager.java:215(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/util/ComponentImplManager.createComponent(ComponentImplManager.java:203(Compiled Code)) at org/eclipse/wst/common/componentcore/ComponentCore.createComponent(ComponentCore.java:64(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/builder/DependencyGraphImpl$GraphUpdateJob$1.run(DependencyGraphImpl.java:527) at org/eclipse/core/runtime/SafeRunner.run(SafeRunner.java:42(Compiled Code)) at org/eclipse/wst/common/componentcore/internal/builder/DependencyGraphImpl$GraphUpdateJob.run(DependencyGraphImpl.java:462) at org/eclipse/core/internal/jobs/Worker.run(Worker.java:54(Compiled Code)) This is similar to https://bugs.eclipse.org/bugs/show_bug.cgi?id=413730
I'm guessing it's not practical to lock the file that StructureEdit needs before acquiring the lock on StructureEdit?
Created attachment 236042 [details] Patch adding new job to prevent deadlock
Added patch that moves the processing for J2EEComponentClasspathUpdater.findNode() into a job with workspace root as the rule, so will prevent deadlocks in this instance.
Committed to R3_4_2_patches
I'm not sure that this fix solved the problems here. I'm trying to figure out how to reproduce this and what causes it, but I often get my workspace into a state where Eclipse is constantly running this new "Find Node Job". When I start with a new workspace, I am fine. After working a while, eclipse will get into a state where it constantly spawns new "Find Node Job" and basically prevents any other task from completing. Restarting the workspace won't fix it. My only recourse is to delete the server and recreate it. I apologize I don't have more relevant information at this time on this. I'm only vaguely familiar with the code here and found this bug because of the "Find Node Job" string in the patch. My best guess is that the chain of events is resource changes -> Find Node Job runs -> finds changes -> copies changes -> resource changes (rinse, wash, repeat).
