Bug 510552 - Deadlock on eclipse startup when adding listeners to 2 servers
Summary: Deadlock on eclipse startup when adding listeners to 2 servers
Status: RESOLVED FIXED
Alias: None
Product: WTP ServerTools
Classification: WebTools
Component: jst.server (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 3.9 M7   Edit
Assignee: Elson Yuen CLA
QA Contact: Elson Yuen CLA
URL:
Whiteboard:
Keywords: plan
Depends on:
Blocks:
 
Reported: 2017-01-17 04:58 EST by Beirti O 'Nunain CLA
Modified: 2017-10-11 16:46 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Beirti O 'Nunain CLA 2017-01-17 04:58:42 EST
Setup:

- Windows 7
- JDK 1.8.0_102
- Eclipse JEE Neon 2
- JBoss Developer Studio 10.2 plugins from Marketplace

Problem:

I have created 2 servers (JBoss EAP 6.1+) which are very similar but refer to 2 different projects in my workspace. Whenever I start Eclipse, the progress bar shows the following jobs:
- Initializing Server Tools
- Registering Listeners
- Updating Maven Dependencies (Blocked: The user operation is waiting for background work to complete)
- Initializing Java Tooling (touching resource xxx)
- Initializing JavaScript Tooling (Configuring Javascript Lanugage (ECMA-262) )
- InitRemoteEditJob (Waiting)

I left my workspace open overnight and these tasks are still there. If I try to interact with the workspace in any meaningful way, the Eclipse process hangs and I need to hard kill it. I also cannot cancel any of the background jobs.

I attached a yourkit profiler to it and a deadlock was immediately detected. (Thread dump below)

Workaround: Create a new workspace. I'll add a comment to this bug if I encounter the problem again.

Yourkit Thread Dump:

Java-level deadlocks detected

This means that some threads are blocked waiting to enter a synchronization block or
waiting to reenter a synchronization block after an Object.wait() call, where each thread
owns one monitor while trying to obtain another monitor already held by another thread.

Deadlock:

Thread-8 is waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c which is held by Worker-3
Worker-3 is waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f which is held by Thread-8

Deadlock:

Worker-3 is waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f which is held by Thread-8
Thread-8 is waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c which is held by Worker-3

Deadlock:

Thread-8 is waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c which is held by Worker-3
Worker-3 is waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f which is held by Thread-8

Deadlock:

Thread-8 is waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c which is held by Worker-3
Worker-3 is waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f which is held by Thread-8

Deadlock:

Thread-8 is waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c which is held by Worker-3
Worker-3 is waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f which is held by Thread-8

Thread stacks

Thread-8 [BLOCKED; waiting to lock org.eclipse.wst.server.core.internal.Server@5fd81b2c]
 org.eclipse.wst.server.core.internal.Server.getServerNotificationManager(Server.java:1034)
 org.eclipse.wst.server.core.internal.Server.removeServerListener(Server.java:697)
 org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager.protectAddManagerAsListeners(UnitedServerListenerManager.java:152)
 org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager.initializeManager(UnitedServerListenerManager.java:81)
 org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager.access$1(UnitedServerListenerManager.java:73)
 org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager$1.run(UnitedServerListenerManager.java:68)

Worker-3 [BLOCKED; waiting to lock org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager@31c5a13f]
 org.jboss.ide.eclipse.as.core.server.UnitedServerListenerManager.addListener(UnitedServerListenerManager.java:109)
 org.jboss.ide.eclipse.as.core.JBossServerCorePlugin.start(JBossServerCorePlugin.java:74)
 org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:774)
 org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1)
 java.security.AccessController.doPrivileged(native method)
 org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:767)
 org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:724)
 org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:932)
 org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309)
 org.eclipse.osgi.container.Module.doStart(Module.java:581)
 org.eclipse.osgi.container.Module.start(Module.java:449)
 org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:470)
 org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:107)
 org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:529)
 org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:325)
 org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:345)
 org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:423)
 org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:372)
 org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:364)
 org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161)
 java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:564)
 org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:174)
 org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:905)
 org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
 org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:55)
 org.eclipse.wst.server.core.internal.ServerType.createServerDelegate(ServerType.java:91)
 org.eclipse.wst.server.core.internal.Server.getDelegate(Server.java:506)
 org.eclipse.wst.server.core.internal.Server.getChildModules(Server.java:2634)
 org.eclipse.wst.server.core.internal.Server.visitModule(Server.java:3058)
 org.eclipse.wst.server.core.internal.Server.visit(Server.java:3039)
 org.eclipse.wst.server.core.internal.Server.getAllModules(Server.java:1542)
 org.eclipse.wst.server.ui.internal.cnf.ServersView2$3.run(ServersView2.java:189)
 org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)
Comment 1 Beirti O 'Nunain CLA 2017-01-17 05:02:53 EST
- Running eclipse with '-clean' did not resolve the issue.
Comment 2 Elson Yuen CLA 2017-01-17 10:18:42 EST
Based on the thread dump, it looks like the deadlock is related to the server implementation of specific vendor and it doesn't looks like the deadlock is caused by the WTP Server Tools framework.  Please contact JBoss directly who provides the JBoss specific plugin on this issue.
Comment 3 Rob Stryker CLA 2017-01-18 13:46:35 EST
It appears to me that while three of the threads mention JBoss, the core of the deadlock is that:

1) An adopter plugin is being 'started' because the servers view wants to get child modules for the server, and

2) The start() method of the adopter plugin tries to add or remove listeners on the locked server, and won't return until its listeners are added

So basically, any plugin wishing to add listeners to existing servers need to recognize that their plugin may be started in the context of a locked server, and behave accordingly. 

I think this is a significant enough issue that it should be documented somewhere, as it's not unreasonable that the start() method of a plugin might decide to add listeners to the server framework or individual server instances.
Comment 4 Elson Yuen CLA 2017-01-18 14:55:48 EST
Rob, that's a good point. I am reopening this item to add more info to the javadoc of addServerListener() to warn API user that this should not be done during the startup of the server implementation plugin.
Comment 5 Beirti O 'Nunain CLA 2017-01-26 05:18:51 EST
Workaround to save the workspace:

- Open [workspace]/.metadata/.plugins/org.eclipse.wst.server.core/servers.xml

Remove *all* servers from here.
Kill and start eclipse again.
Comment 6 Eclipse Genie CLA 2017-05-12 16:09:20 EDT
New Gerrit change created: https://git.eclipse.org/r/96989
Comment 8 Elson Yuen CLA 2017-05-12 16:12:16 EDT
Added warning to javadoc. Code dropped to master.
Comment 9 Eclipse Genie CLA 2017-10-11 16:46:40 EDT
New Gerrit change created: https://git.eclipse.org/r/109353