| [dali-dev] comments on bug 243547 (today's deadlock) |
|
Here's my take on bug 243547.
(Which was just closed, since the SSE-related deadlock seems to have been fixed. But Gary Karasiuk will probably open a new bug with the new deadlock, whose relevant stacktraces I've extracted from the dump and attached.) Any comments, corrections, or insights are welcome. Both threads are running the Dali validator. I think(?) Worker-2 has locked the workspace while the Dali validator deletes the Dali-related markers. This causes the workspace to notify Dali that a resource has changed. The Dali listener is waiting for a lock on the Dali model (GenericJpaModel) so it can forward the notification to the appropriate JPA project. Unfortunately, GenericJpaModel is already locked by Worker-5 (below). In Worker-5, validation has triggered the (lazy) creation of a JPA project, locking the Dali model (GenericJpaModel) until the JPA project is built. This creation also triggers the loading of the DTP plug-in, which then loads and validates the DTP drivers. It looks like DTP has detected an error in one of the drivers and needs to lock the workspace so it can add the appropriate problem markers. Unfortunately, the workspace is already locked by Worker-2 (above). The way we are doing things currently (which I've never really warmed up to :-) ) doesn't really allow us to remove the validation markers asynchronously. We could investigate changing that.... Likewise, it might be difficult for DTP to add its problem markers asynchronously; but might merit investigation.... Rough notes: Worker-2 JpaValidator calls WorkbenchReporter.removeAllMessages(IValidator) calls Workspace.deleteMarkers(IMarker[]) which notifies listeners that a resource has changed waiting to call GenericJpaModel.synchronizeFiles(IProject, IResourceDelta) which is locked by Worker-5 somewhere acquires a lock... Worker-5 JpaValidator calls JptCorePlugin to get the appropriate JPA project locks and calls GenericJpaModel to get the JPA project the JPA project is (lazily) built triggers the loading of JptDbPlugin builds the DTPConnectionProfileRepository triggers the DTP InternalProfileManager to load the profiles the properties of a DTP driver are invalid so... DriverValidator tries to add a problem marker WorkManager waits to lock the "primary workspace lock" |
3XMTHREADINFO "Worker-2" TID:0x3210B200, j9thread_t:0x385EC750, state:B, prio=5 3XMTHREADINFO1 (native thread ID:0x378, native priority:0x5, native policy:UNKNOWN) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.synchronizeFiles(JpaModelManager.java:295) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.synchronizeFiles(JpaModelManager.java:275) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.synchronizeFiles(JpaModelManager.java:286) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.synchronizeFiles(JpaModelManager.java:272) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.resourcePostChange(JpaModelManager.java:264) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.resourceChanged(JpaModelManager.java:237) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager$ResourceChangeListener.resourceChanged(JpaModelManager.java:487) 4XESTACKTRACE at org/eclipse/core/internal/events/NotificationManager$2.run(NotificationManager.java:288) 4XESTACKTRACE at org/eclipse/core/runtime/SafeRunner.run(SafeRunner.java:37(Compiled Code)) 4XESTACKTRACE at org/eclipse/core/internal/events/NotificationManager.notify(NotificationManager.java:282) 4XESTACKTRACE at org/eclipse/core/internal/events/NotificationManager.broadcastChanges(NotificationManager.java:148) 4XESTACKTRACE at org/eclipse/core/internal/resources/Workspace.broadcastPostChange(Workspace.java:313) 4XESTACKTRACE at org/eclipse/core/internal/resources/Workspace.endOperation(Workspace.java:1022) 4XESTACKTRACE at org/eclipse/core/internal/resources/Workspace.deleteMarkers(Workspace.java:961) 4XESTACKTRACE at org/eclipse/wst/validation/internal/TaskListUtility.removeTaskSubset(TaskListUtility.java:428) 4XESTACKTRACE at org/eclipse/wst/validation/internal/TaskListUtility.removeAllTasks(TaskListUtility.java:406) 4XESTACKTRACE at org/eclipse/wst/validation/internal/TaskListUtility.removeAllTasks(TaskListUtility.java:402) 4XESTACKTRACE at org/eclipse/wst/validation/internal/operations/WorkbenchReporter.removeAllMessages(WorkbenchReporter.java:164) 4XESTACKTRACE at org/eclipse/wst/validation/internal/operations/WorkbenchReporter.removeAllMessages(WorkbenchReporter.java:544) 4XESTACKTRACE at org/eclipse/wst/validation/internal/operations/WorkbenchReporter.removeAllMessages(WorkbenchReporter.java:509) 4XESTACKTRACE at org/eclipse/jpt/core/internal/validation/JpaValidator.validate(JpaValidator.java:46) 4XESTACKTRACE at org/eclipse/jpt/core/internal/validation/JpaValidator.validateInJob(JpaValidator.java:77) 4XESTACKTRACE at org/eclipse/wst/validation/internal/operations/ValidatorJob.run(ValidatorJob.java:78) 4XESTACKTRACE at org/eclipse/core/internal/jobs/Worker.run(Worker.java:55) 3XMTHREADINFO "Worker-5" TID:0x32119D00, j9thread_t:0x39E62C38, state:CW, prio=1 3XMTHREADINFO1 (native thread ID:0x9FC, native priority:0x1, native policy:UNKNOWN) 4XESTACKTRACE at java/lang/Object.wait(Native Method) 4XESTACKTRACE at java/lang/Object.wait(Object.java:196) 4XESTACKTRACE at org/eclipse/core/internal/jobs/Semaphore.acquire(Semaphore.java:38) 4XESTACKTRACE at org/eclipse/core/internal/jobs/OrderedLock.doAcquire(OrderedLock.java:169) 4XESTACKTRACE at org/eclipse/core/internal/jobs/OrderedLock.acquire(OrderedLock.java:105(Compiled Code)) 4XESTACKTRACE at org/eclipse/core/internal/jobs/OrderedLock.acquire(OrderedLock.java:82(Compiled Code)) 4XESTACKTRACE at org/eclipse/core/internal/resources/WorkManager.checkIn(WorkManager.java:118) 4XESTACKTRACE at org/eclipse/core/internal/resources/Workspace.prepareOperation(Workspace.java:1747) 4XESTACKTRACE at org/eclipse/core/internal/resources/Resource.createMarker(Resource.java:641) 4XESTACKTRACE at org/eclipse/datatools/connectivity/drivers/DriverValidator.addProblemMarker(DriverValidator.java:230) 4XESTACKTRACE at org/eclipse/datatools/connectivity/drivers/DriverValidator.isValid(DriverValidator.java:96) 4XESTACKTRACE at org/eclipse/datatools/connectivity/drivers/DriverValidator.isValid(DriverValidator.java:79) 4XESTACKTRACE at org/eclipse/datatools/connectivity/internal/InternalProfileManager.getDriverInstance(InternalProfileManager.java:1334) 4XESTACKTRACE at org/eclipse/datatools/connectivity/internal/InternalProfileManager.enableLocalDatabase(InternalProfileManager.java:1252) 4XESTACKTRACE at org/eclipse/datatools/connectivity/internal/InternalProfileManager.loadLocalRegisteredDatabases(InternalProfileManager.java:1211) 4XESTACKTRACE at org/eclipse/datatools/connectivity/internal/InternalProfileManager.loadProfiles(InternalProfileManager.java:928) 4XESTACKTRACE at org/eclipse/datatools/connectivity/internal/InternalProfileManager.getProfiles(InternalProfileManager.java:139) 4XESTACKTRACE at org/eclipse/datatools/connectivity/ProfileManager.getProfiles(ProfileManager.java:46) 4XESTACKTRACE at org/eclipse/jpt/db/internal/DTPConnectionProfileRepository.start(DTPConnectionProfileRepository.java:63) 4XESTACKTRACE at org/eclipse/jpt/db/JptDbPlugin.start(JptDbPlugin.java:53) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleContextImpl$2.run(BundleContextImpl.java:1009) 4XESTACKTRACE at java/security/AccessController.doPrivileged(AccessController.java:251) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleContextImpl.startActivator(BundleContextImpl.java:1003) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleContextImpl.start(BundleContextImpl.java:984) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleHost.startWorker(BundleHost.java:346) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/AbstractBundle.start(AbstractBundle.java:265) 4XESTACKTRACE at org/eclipse/osgi/framework/util/SecureAction.start(SecureAction.java:400) 4XESTACKTRACE at org/eclipse/core/runtime/internal/adaptor/EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:111(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/baseadaptor/loader/ClasspathManager.findLocalClass(ClasspathManager.java:427(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.findLocalClass(DefaultClassLoader.java:193(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleLoader.findLocalClass(BundleLoader.java:368(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/SingleSourcePackage.loadClass(SingleSourcePackage.java:33(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleLoader.findClassInternal(BundleLoader.java:441(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleLoader.findClass(BundleLoader.java:397(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/framework/internal/core/BundleLoader.findClass(BundleLoader.java:385(Compiled Code)) 4XESTACKTRACE at org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader.loadClass(DefaultClassLoader.java:87(Compiled Code)) 4XESTACKTRACE at java/lang/ClassLoader.loadClass(ClassLoader.java:609(Compiled Code)) 4XESTACKTRACE at java/lang/J9VMInternals.verifyImpl(Native Method) 4XESTACKTRACE at java/lang/J9VMInternals.verify(J9VMInternals.java:72(Compiled Code)) 4XESTACKTRACE at java/lang/J9VMInternals.initialize(J9VMInternals.java:134(Compiled Code)) 4XESTACKTRACE at org/eclipse/jpt/core/internal/platform/GenericJpaFactory.buildJpaDataSource(GenericJpaFactory.java:226) 4XESTACKTRACE at org/eclipse/jpt/core/internal/GenericJpaProject.<init>(GenericJpaProject.java:141) 4XESTACKTRACE at org/eclipse/jpt/core/internal/platform/GenericJpaFactory.buildJpaProject(GenericJpaFactory.java:222) 4XESTACKTRACE at org/eclipse/jpt/core/internal/GenericJpaModel$DefaultJpaProjectHolder.buildJpaProject(GenericJpaModel.java:453) 4XESTACKTRACE at org/eclipse/jpt/core/internal/GenericJpaModel$DefaultJpaProjectHolder.jpaProject(GenericJpaModel.java:445) 4XESTACKTRACE at org/eclipse/jpt/core/internal/GenericJpaModel.getJpaProject(GenericJpaModel.java:84) 4XESTACKTRACE at org/eclipse/jpt/core/internal/JpaModelManager.getJpaProject(JpaModelManager.java:178) 4XESTACKTRACE at org/eclipse/jpt/core/JptCorePlugin.getJpaProject(JptCorePlugin.java:157) 4XESTACKTRACE at org/eclipse/jpt/core/internal/validation/JpaValidator.validationMessages(JpaValidator.java:55) 4XESTACKTRACE at org/eclipse/jpt/core/internal/validation/JpaValidator.validate(JpaValidator.java:48) 4XESTACKTRACE at org/eclipse/jpt/core/internal/validation/JpaValidator.validateInJob(JpaValidator.java:77) 4XESTACKTRACE at org/eclipse/wst/validation/internal/operations/ValidatorJob.run(ValidatorJob.java:78) 4XESTACKTRACE at org/eclipse/core/internal/jobs/Worker.run(Worker.java:55)