[NEXUS-5709] capabilities plugin can cause deadlock, prevent access to capabilities UI - Sonatype JIRA

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.2.1
Fix Version/s: 2.3
Component/s: Capabilities
Labels:
None
Environment:
capabilities plugin 2.1.1

Description

This issue is to track a reported deadlock seen by customers running Nexus 2.2.1. This deadlock is avoided by using nexus-capabilities-plugin 2.2 which is available in Nexus 2.3.0-04 release. All customers affected by this issue should upgrade Nexus to avoid the problem.

Symptoms

Nexus hangs
Nexus hangs during shutdown
Capabilities UI is non-responsive
thread contention in thread dumps which include capabilities plugin activity
threads waiting to lock org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler
smart proxy activity ( mostly because it uses capabilities in a common way to trigger blocked threads

Example Threads marking the condition

"nxevthost-1-thread-7259" prio=10 tid=0x00002aaafc8ef800 nid=0x1426 in Object.wait() [0x0000000045a94000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.springframework.jms.listener.DefaultMessageListenerContainer.doShutdown(DefaultMessageListenerContainer.java:538)
	- locked <0x0000000683bd9e00> (a java.lang.Object)
	at org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:231)
	at org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:172)
	at org.apache.camel.component.jms.JmsConsumer.doStop(JmsConsumer.java:135)
	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
	at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:109)
	at org.apache.camel.impl.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:247)
	at org.apache.camel.impl.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:221)
	at org.apache.camel.impl.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:146)
	at org.apache.camel.impl.DefaultShutdownStrategy.shutdown(DefaultShutdownStrategy.java:93)
	at org.apache.camel.impl.DefaultShutdownStrategy.shutdown(DefaultShutdownStrategy.java:85)
	at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:1516)
	- locked <0x0000000683ba8dc8> (a org.apache.camel.impl.DefaultCamelContext)
	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.doStop(EventSubscriberImpl.java:113)
	at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport$Handler.doStop(LifecycleSupport.java:69)
	at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext$MainMap_Stopping.stopped(LifecycleHandlerContext.java:354)
	at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext.stopped(LifecycleHandlerContext.java:73)
	- locked <0x00000006827a07f8> (a org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext)
	at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport.stop(LifecycleSupport.java:130)
	- locked <0x00000006827a07e8> (a org.sonatype.sisu.goodies.common.Mutex)
	at org.sonatype.sisu.goodies.lifecycle.Stopper.stop(Stopper.java:40)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.maybeStop(EventSubscriberImpl.java:308)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.unsubscribe(EventSubscriberImpl.java:378)
	- locked <0x00000006827a0798> (a com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.access$1(EventSubscriberImpl.java:360)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl$SubscriptionImpl.cancel(EventSubscriberImpl.java:214)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.SubscribeCapability.onPassivate(SubscribeCapability.java:77)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.SubscribeCapability.onPassivate(SubscribeCapability.java:1)
	at com.sonatype.nexus.plugins.smartproxy.internal.capability.CapabilitySupport.onPassivate(CapabilitySupport.java:190)
	at org.sonatype.nexus.plugins.capabilities.internal.DefaultCapabilityReference$ActiveState.passivate(DefaultCapabilityReference.java:734)
	at org.sonatype.nexus.plugins.capabilities.internal.DefaultCapabilityReference.passivate(DefaultCapabilityReference.java:243)
	at org.sonatype.nexus.plugins.capabilities.internal.ActivationConditionHandler.handle(ActivationConditionHandler.java:77)
	at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
	- locked <0x0000000680807658> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
	at org.sonatype.nexus.plugins.capabilities.support.condition.ConditionSupport.setSatisfied(ConditionSupport.java:123)
	at org.sonatype.nexus.plugins.capabilities.support.condition.CompositeConditionSupport.handle(CompositeConditionSupport.java:84)
	at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
	- locked <0x0000000680807878> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
	at org.sonatype.nexus.plugins.capabilities.support.condition.ConditionSupport.setSatisfied(ConditionSupport.java:123)
	at org.sonatype.nexus.plugins.capabilities.internal.condition.RepositoryConditionSupport.setSatisfied(RepositoryConditionSupport.java:83)
	at org.sonatype.nexus.plugins.capabilities.internal.condition.RepositoryProxyModeCondition.handle(RepositoryProxyModeCondition.java:65)
	at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
	- locked <0x00000006807da460> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
	at org.sonatype.nexus.plugins.capabilities.internal.NexusEventsForwarderEventInspector.inspect(NexusEventsForwarderEventInspector.java:68)
	at org.sonatype.nexus.events.DefaultEventInspectorHost.processEvent(DefaultEventInspectorHost.java:128)
	at org.sonatype.nexus.events.DefaultEventInspectorHost.onEvent(DefaultEventInspectorHost.java:101)
	at org.sonatype.plexus.appevents.AbstractSimpleEventMulticaster.notifyEventListeners(AbstractSimpleEventMulticaster.java:53)
	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.setProxyMode(AbstractProxyRepository.java:547)
	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.autoBlockProxying(AbstractProxyRepository.java:658)
	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem0(AbstractProxyRepository.java:1247)
	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem(AbstractProxyRepository.java:1114)
	at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.doRetrieveItem(AbstractMavenRepository.java:393)
	at org.sonatype.nexus.proxy.maven.maven2.M2Repository.doRetrieveItem(M2Repository.java:431)
	at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:830)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.fetchItem(ItemHandlerSupport.java:144)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.doFetchItem(ItemHandlerSupport.java:159)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.access$0(ItemHandlerSupport.java:157)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEvent.perform(ItemHandlerSupport.java:202)
	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEventInspector.inspect(ItemHandlerSupport.java:222)
	at org.sonatype.nexus.events.DefaultEventInspectorHost$EventInspectorHandler.run(DefaultEventInspectorHost.java:184)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000006807da770> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	- <0x00000006841a6220> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	- <0x0000000684218af0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

In the case above, many threads blocked - waiting to lock <0x00000006807da460> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)

Related Commit that fixes these problems

https://github.com/sonatype/nexus-capabilities-plugin/commit/8895c6c4126d849f0954d36495e748115307c93f
( available in Nexus 2.3.0-04 )

Solution

must force restart during deadlock
upgrade to at least Nexus 2.3.0-04

Attachments

Activity

People

Assignee:: Peter Lynch

Reporter:: Peter Lynch

Last Updated By:: Peter Lynch

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 05/02/13 12:55 PM

Updated:: 05/02/13 01:06 PM

Resolved:: 05/02/13 12:56 PM