Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-9439

deadlock between mergeropo and staging promotion

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.11.4
    • Fix Version/s: 2.12.1
    • Component/s: Staging, Yum
    • Labels:
      None
    • Story Points:
      0.5
    • Sprint:
      Sprint 63 - Föhn

      Description

      A GET request comes into a group repository which triggers a yum metadata merge. It can't proceed because it can't get a lock to schedule the task:

      qtp1444610073-5921 id=5921 state=WAITING

      • waiting on <0x631ba521> (a java.util.concurrent.FutureTask$Sync)
      • locked <0x631ba521> (a java.util.concurrent.FutureTask$Sync)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248)
        at java.util.concurrent.FutureTask.get(FutureTask.java:111)
        at org.sonatype.scheduling.DefaultScheduledTask.get(DefaultScheduledTask.java:260)
        at org.sonatype.nexus.yum.internal.YumGroupImpl.getYumRepository(YumGroupImpl.java:91)
        at org.sonatype.nexus.yum.internal.MergeMetadataRequestStrategy.onHandle(MergeMetadataRequestStrategy.java:90)
        at org.sonatype.nexus.proxy.repository.AbstractRepository.checkRequestStrategies(AbstractRepository.java:1201)
        at org.sonatype.nexus.proxy.repository.AbstractRepository.checkConditions(AbstractRepository.java:1193)
        at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:590)
        at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155)
        at org.sonatype.nexus.content.internal.ContentServlet.doGet(ContentServlet.java:376)
        at org.sonatype.nexus.content.internal.ContentServlet.service(ContentServlet.java:342)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:288)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:278)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at org.sonatype.nexus.web.internal.NexusGuiceFilter$MultiFilterChain.doFilter(NexusGuiceFilter.java:82)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at org.sonatype.nexus.web.internal.NexusGuiceFilter$MultiFilterChain.doFilter(NexusGuiceFilter.java:82)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:89)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
        at org.sonatype.nexus.web.internal.SecurityFilter.executeChain(SecurityFilter.java:90)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

      It can't get a lock to do this because another thread has the default configuration lock because it is promoting a staging repository. This thread can't proceed because it is trying to acquire a lock held by the first thread:

      pxpool-1-thread-4 id=124 state=WAITING

      • waiting on <0x13ccef4e> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      • locked <0x13ccef4e> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        owned by qtp1444610073-5921 id=5921
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at org.sonatype.nexus.yum.internal.YumGroupImpl.markDirty(YumGroupImpl.java:109)
        at org.sonatype.nexus.yum.internal.EventsRouter.on(EventsRouter.java:96)
        at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
        at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
        at org.sonatype.nexus.proxy.repository.AbstractGroupRepository.prepareForSave(AbstractGroupRepository.java:125)
        at org.sonatype.nexus.configuration.AbstractConfigurable.onEvent(AbstractConfigurable.java:105)
        at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:49)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
        at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
        at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.applyConfiguration(DefaultNexusConfiguration.java:339)
      • locked org.sonatype.nexus.configuration.application.DefaultNexusConfiguration@1a90baa2
        at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.saveConfiguration(DefaultNexusConfiguration.java:363)
      • locked org.sonatype.nexus.configuration.application.DefaultNexusConfiguration@1a90baa2
        at com.sonatype.nexus.staging.internal.DefaultStagingRepositoryManager.setRepositoryGroupStaged(DefaultStagingRepositoryManager.java:265)
        at com.sonatype.nexus.staging.internal.task.RepositoryPromoteTask$PromoteOperation.perform(RepositoryPromoteTask.java:200)
        at com.sonatype.nexus.staging.internal.task.OperationTaskSupport.executeOperations(OperationTaskSupport.java:434)
        at com.sonatype.nexus.staging.internal.task.OperationTaskSupport.doCall(OperationTaskSupport.java:415)
        at com.sonatype.nexus.staging.internal.task.RepositoryPromoteTask.doCall(RepositoryPromoteTask.java:365)
        at com.sonatype.nexus.staging.internal.task.RepositoryPromoteTask.doCall(RepositoryPromoteTask.java:1)
        at com.sonatype.nexus.staging.internal.task.TaskSupport.call(TaskSupport.java:37)
        at com.sonatype.nexus.staging.internal.task.StagingTaskSupport.call(StagingTaskSupport.java:133)
        at com.sonatype.nexus.staging.internal.task.StagingBackgroundTask.execute(StagingBackgroundTask.java:67)
        at com.sonatype.nexus.staging.internal.task.NexusTaskSupport.doRun(NexusTaskSupport.java:52)
        at com.sonatype.nexus.staging.internal.task.NexusTaskSupport.doRun(NexusTaskSupport.java:1)
        at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:163)
        at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:418)
        at org.sonatype.nexus.threads.MDCAwareCallable.call(MDCAwareCallable.java:44)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title