Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-5709

capabilities plugin can cause deadlock, prevent access to capabilities UI

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.1
    • Fix Version/s: 2.3
    • Component/s: Capabilities
    • Labels:
      None
    • Environment:
      capabilities plugin 2.1.1

      Description

      This issue is to track a reported deadlock seen by customers running Nexus 2.2.1. This deadlock is avoided by using nexus-capabilities-plugin 2.2 which is available in Nexus 2.3.0-04 release. All customers affected by this issue should upgrade Nexus to avoid the problem.

      Symptoms

      • Nexus hangs
      • Nexus hangs during shutdown
      • Capabilities UI is non-responsive
      • thread contention in thread dumps which include capabilities plugin activity
      • threads waiting to lock org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler
      • smart proxy activity ( mostly because it uses capabilities in a common way to trigger blocked threads

      Example Threads marking the condition

      "nxevthost-1-thread-7259" prio=10 tid=0x00002aaafc8ef800 nid=0x1426 in Object.wait() [0x0000000045a94000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Object.wait(Object.java:485)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer.doShutdown(DefaultMessageListenerContainer.java:538)
      	- locked <0x0000000683bd9e00> (a java.lang.Object)
      	at org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:231)
      	at org.springframework.jms.listener.AbstractJmsListeningContainer.destroy(AbstractJmsListeningContainer.java:172)
      	at org.apache.camel.component.jms.JmsConsumer.doStop(JmsConsumer.java:135)
      	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
      	at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:109)
      	at org.apache.camel.impl.DefaultShutdownStrategy.shutdownNow(DefaultShutdownStrategy.java:247)
      	at org.apache.camel.impl.DefaultShutdownStrategy.shutdownRoutesNow(DefaultShutdownStrategy.java:221)
      	at org.apache.camel.impl.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:146)
      	at org.apache.camel.impl.DefaultShutdownStrategy.shutdown(DefaultShutdownStrategy.java:93)
      	at org.apache.camel.impl.DefaultShutdownStrategy.shutdown(DefaultShutdownStrategy.java:85)
      	at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:1516)
      	- locked <0x0000000683ba8dc8> (a org.apache.camel.impl.DefaultCamelContext)
      	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:91)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.doStop(EventSubscriberImpl.java:113)
      	at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport$Handler.doStop(LifecycleSupport.java:69)
      	at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext$MainMap_Stopping.stopped(LifecycleHandlerContext.java:354)
      	at org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext.stopped(LifecycleHandlerContext.java:73)
      	- locked <0x00000006827a07f8> (a org.sonatype.sisu.goodies.lifecycle.LifecycleHandlerContext)
      	at org.sonatype.sisu.goodies.lifecycle.LifecycleSupport.stop(LifecycleSupport.java:130)
      	- locked <0x00000006827a07e8> (a org.sonatype.sisu.goodies.common.Mutex)
      	at org.sonatype.sisu.goodies.lifecycle.Stopper.stop(Stopper.java:40)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.maybeStop(EventSubscriberImpl.java:308)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.unsubscribe(EventSubscriberImpl.java:378)
      	- locked <0x00000006827a0798> (a com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl.access$1(EventSubscriberImpl.java:360)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.EventSubscriberImpl$SubscriptionImpl.cancel(EventSubscriberImpl.java:214)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.SubscribeCapability.onPassivate(SubscribeCapability.java:77)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.SubscribeCapability.onPassivate(SubscribeCapability.java:1)
      	at com.sonatype.nexus.plugins.smartproxy.internal.capability.CapabilitySupport.onPassivate(CapabilitySupport.java:190)
      	at org.sonatype.nexus.plugins.capabilities.internal.DefaultCapabilityReference$ActiveState.passivate(DefaultCapabilityReference.java:734)
      	at org.sonatype.nexus.plugins.capabilities.internal.DefaultCapabilityReference.passivate(DefaultCapabilityReference.java:243)
      	at org.sonatype.nexus.plugins.capabilities.internal.ActivationConditionHandler.handle(ActivationConditionHandler.java:77)
      	at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
      	- locked <0x0000000680807658> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
      	at org.sonatype.nexus.plugins.capabilities.support.condition.ConditionSupport.setSatisfied(ConditionSupport.java:123)
      	at org.sonatype.nexus.plugins.capabilities.support.condition.CompositeConditionSupport.handle(CompositeConditionSupport.java:84)
      	at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
      	- locked <0x0000000680807878> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
      	at org.sonatype.nexus.plugins.capabilities.support.condition.ConditionSupport.setSatisfied(ConditionSupport.java:123)
      	at org.sonatype.nexus.plugins.capabilities.internal.condition.RepositoryConditionSupport.setSatisfied(RepositoryConditionSupport.java:83)
      	at org.sonatype.nexus.plugins.capabilities.internal.condition.RepositoryProxyModeCondition.handle(RepositoryProxyModeCondition.java:65)
      	at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:68)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:45)
      	- locked <0x00000006807da460> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:315)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus$1.dispatchQueuedEvents(DefaultEventBus.java:72)
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:266)
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:99)
      	at org.sonatype.nexus.plugins.capabilities.internal.NexusEventsForwarderEventInspector.inspect(NexusEventsForwarderEventInspector.java:68)
      	at org.sonatype.nexus.events.DefaultEventInspectorHost.processEvent(DefaultEventInspectorHost.java:128)
      	at org.sonatype.nexus.events.DefaultEventInspectorHost.onEvent(DefaultEventInspectorHost.java:101)
      	at org.sonatype.plexus.appevents.AbstractSimpleEventMulticaster.notifyEventListeners(AbstractSimpleEventMulticaster.java:53)
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.setProxyMode(AbstractProxyRepository.java:547)
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.autoBlockProxying(AbstractProxyRepository.java:658)
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem0(AbstractProxyRepository.java:1247)
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem(AbstractProxyRepository.java:1114)
      	at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.doRetrieveItem(AbstractMavenRepository.java:393)
      	at org.sonatype.nexus.proxy.maven.maven2.M2Repository.doRetrieveItem(M2Repository.java:431)
      	at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:830)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.fetchItem(ItemHandlerSupport.java:144)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.doFetchItem(ItemHandlerSupport.java:159)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.access$0(ItemHandlerSupport.java:157)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEvent.perform(ItemHandlerSupport.java:202)
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEventInspector.inspect(ItemHandlerSupport.java:222)
      	at org.sonatype.nexus.events.DefaultEventInspectorHost$EventInspectorHandler.run(DefaultEventInspectorHost.java:184)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
         Locked ownable synchronizers:
      	- <0x00000006807da770> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	- <0x00000006841a6220> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	- <0x0000000684218af0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      

      In the case above, many threads blocked - waiting to lock <0x00000006807da460> (a org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler)

      Related Commit that fixes these problems

      https://github.com/sonatype/nexus-capabilities-plugin/commit/8895c6c4126d849f0954d36495e748115307c93f
      ( available in Nexus 2.3.0-04 )

      Solution

      • must force restart during deadlock
      • upgrade to at least Nexus 2.3.0-04

        Attachments

          Activity

            People

            Assignee:
            plynch Peter Lynch
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                tigCommentSecurity.panel-title