Bug 469555 - Deadlock during IndexDiffCacheEntry$6.resourceChanged() call
Summary: Deadlock during IndexDiffCacheEntry$6.resourceChanged() call
Status: RESOLVED FIXED
Alias: None
Product: EGit
Classification: Technology
Component: Core (show other bugs)
Version: 4.0   Edit
Hardware: All All
: P3 normal (vote)
Target Milestone: 4.1   Edit
Assignee: Andrey Loskutov CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-06-07 02:56 EDT by Andrey Loskutov CLA
Modified: 2015-07-03 19:00 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Loskutov CLA 2015-06-07 02:56:35 EDT
This one is similar to bug 468270: but in different place.

I've observed sporadic deadlock on startup of Eclipse 4.5 RC4 (4.5.0.I20150603-2000) with latest egit 4.1.0.201506030602 installed.

eclipse.buildId=4.5.0.I20150603-2000
java.version=1.8.0_45
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments:  -showLocation -cssTheme none
Command-line arguments:  -data file:/home/andrey/workspace-platform/ -os linux -ws gtk -arch x86_64 -showLocation -cssTheme none

org.eclipse.core.jobs
Error
Sat Jun 06 19:55:26 CEST 2015
Deadlock detected. All locks owned by thread Worker-95 will be suspended.

java.lang.IllegalStateException
	at org.eclipse.core.internal.jobs.DeadlockDetector.reportDeadlock(DeadlockDetector.java:599)
	at org.eclipse.core.internal.jobs.DeadlockDetector.lockWaitStart(DeadlockDetector.java:403)
	at org.eclipse.core.internal.jobs.LockManager.addLockWaitThread(LockManager.java:154)
	at org.eclipse.core.internal.jobs.OrderedLock.doAcquire(OrderedLock.java:168)
	at org.eclipse.core.internal.jobs.OrderedLock.acquire(OrderedLock.java:106)
	at org.eclipse.core.internal.jobs.OrderedLock.acquire(OrderedLock.java:82)
	at org.eclipse.team.core.RepositoryProvider.mapExistingProvider(RepositoryProvider.java:217)
	at org.eclipse.team.core.RepositoryProvider.getProvider(RepositoryProvider.java:578)
	at org.eclipse.egit.core.project.RepositoryMapping.getMapping(RepositoryMapping.java:320)
	at org.eclipse.egit.core.internal.indexdiff.GitResourceDeltaVisitor.visit(GitResourceDeltaVisitor.java:79)
	at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:63)
	at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:74)
	at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:47)
	at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$6.resourceChanged(IndexDiffCacheEntry.java:556)
	at org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:299)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
	at org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:289)
	at org.eclipse.core.internal.events.NotificationManager.broadcastChanges(NotificationManager.java:152)
	at org.eclipse.core.internal.resources.Workspace.broadcastPostChange(Workspace.java:373)
	at org.eclipse.core.internal.resources.Workspace.endOperation(Workspace.java:1470)
	at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:46)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

org.eclipse.core.jobs
Error
Sat Jun 06 19:55:26 CEST 2015
Thread Worker-94 has locks: R/, P/org.eclipse.e4.ui.tests, P/org.eclipse.jface.tests.databinding, OrderedLock (1) and is waiting for lock OrderedLock (0)

org.eclipse.core.jobs
Error
Sat Jun 06 19:55:26 CEST 2015
Thread Worker-95 has locks: OrderedLock (0) and is waiting for lock OrderedLock (1)

org.eclipse.egit.core
Error
Sat Jun 06 19:55:23 CEST 2015
Remove repository mapping of Git mapped resource for which project or mapped git repository has disappeared: 'RepositoryMapping[<empty> -> '../../.git', absolute path: '/home/andrey/.git' ]'

java.io.FileNotFoundException: 
	at org.eclipse.egit.core.project.GitProjectData.logAndUnmapGoneMappedResource(GitProjectData.java:533)
	at org.eclipse.egit.core.project.GitProjectData.map(GitProjectData.java:499)
	at org.eclipse.egit.core.project.GitProjectData.remapAll(GitProjectData.java:473)
	at org.eclipse.egit.core.project.GitProjectData.load(GitProjectData.java:466)
	at org.eclipse.egit.core.project.GitProjectData.get(GitProjectData.java:206)
	at org.eclipse.egit.core.GitProvider.getData(GitProvider.java:86)
	at org.eclipse.egit.core.project.RepositoryMapping.getMapping(RepositoryMapping.java:326)
	at org.eclipse.egit.ui.Activator$ResourceRefreshJob.triggerRefresh(Activator.java:466)
	at org.eclipse.egit.ui.Activator$1.windowActivated(Activator.java:317)
	at org.eclipse.ui.internal.Workbench$11.run(Workbench.java:1039)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
	at org.eclipse.ui.internal.Workbench.fireWindowActivated(Workbench.java:1036)
	at org.eclipse.ui.internal.WorkbenchWindow$15.shellActivated(WorkbenchWindow.java:2284)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:88)
	at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
	at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4481)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1327)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1351)
	at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1332)
	at org.eclipse.swt.widgets.Shell.filterProc(Shell.java:770)
	at org.eclipse.swt.widgets.Display.filterProc(Display.java:1566)
	at org.eclipse.swt.internal.gtk.OS._g_main_context_iteration(Native Method)
	at org.eclipse.swt.internal.gtk.OS.g_main_context_iteration(OS.java:2422)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3428)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1127)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1018)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156)
	at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:654)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:598)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
	at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:139)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1488)
Comment 1 Andrey Loskutov CLA 2015-06-07 12:11:02 EDT
Analysis:
For whatever reason GitProjectData decided to logAndUnmapGoneMappedResource() on startup.

I have no clue what this RepositoryMapping[<empty> -> '../../.git', absolute path: '/home/andrey/.git' ] could be because I do NOT have .git in my home, but this is another bug.

So logAndUnmapGoneMappedResource() schedules UnmapJob for some project which simply calls RepositoryProvider.unmap(project). unmap() acquires first workspace rule for project, then mappingLock, and finally tries to call project.touch(null). This one requires another lock via workspace.prepareOperation() -> WorkManager.checkIn() -> lock.acquire() but this time the last lock is already taken by previously started workspace operation.

This previously started operation (which we don't know what exact it did) is not finished yet, since it is dispatching resource change notification to listeners. One of this listeners is IndexDiffCacheEntry.resourceChangeListener which wants to know if a RepositoryMapping is known for the project in the resource delta. So it calls RepositoryMapping.getMapping(IProject) -> RepositoryProvider.getProvider(IProject, String) -> RepositoryProvider.mapExistingProvider() -> mappingLock.acquire() -> KABOOM.

The case is almost identical to the bug 468270, so I've created bug 469566 to request non-locking RepositoryProvider.getProvider() team API.

Of course we can't wait for the new team API and need support older Eclipse versions.

So basically all uses of RepositoryProvider.getProvider(...) in egit code must be re-visited to check if they can deadlock. 

Most likely we want our own version of RepositoryProvider.getProvider implementation which never locks anything.
Comment 2 Andrey Loskutov CLA 2015-06-07 18:47:34 EDT
Fix: https://git.eclipse.org/r/49601

The fix need to be tested.
Comment 3 Andrey Loskutov CLA 2015-06-08 17:30:38 EDT
(In reply to Andrey Loskutov from comment #2)
> Fix: https://git.eclipse.org/r/49601
> 
> The fix need to be tested.

Not yet ready yet!

Now without CVS/Subversion etc plugins there is no code anymore which would actually instantiate GitProvider - because it happened always indirectly as side effect of RepositoryProvider.getProvider() calls. OMG. So one MUST make sure that this code IS executed but please ONLY in specific threads and NOT in resource change listeners... This is extremely poor designed. Ideally I would expect that on workspace startup / projects initialization the configured team providers are initialized, but this seem to be not the case and we have lazy loading nightmare.