Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-8195

Deadlock between yum merge metadata task and yum generate metadata task

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.11.2
    • Fix Version/s: 2.11.3, 3.0.0-m4
    • Component/s: Yum
    • Labels:
    • Story Points:
      0.5
    • Sprint:
      Sprint 37, Sprint 38, Sprint 39

      Description

      Concurrently running yum generate and merge metadata tasks can deadlock.

      The generate yum metadata tasks tries to get locks on group metadata:

      pxpool-1-thread-11 id=181 state=WAITING

      • waiting on <0x629c279d> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      • locked <0x629c279d> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        owned by qtp2025967620-804 id=804
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
        at org.sonatype.nexus.yum.internal.YumGroupImpl.markDirty(YumGroupImpl.java:109)
        at org.sonatype.nexus.yum.internal.task.GenerateMetadataTask.regenerateMetadataForGroups(GenerateMetadataTask.java:267)
        at org.sonatype.nexus.yum.internal.task.GenerateMetadataTask.doRun(GenerateMetadataTask.java:183)
        at org.sonatype.nexus.yum.internal.task.GenerateMetadataTask.doRun(GenerateMetadataTask.java:69)
        at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:151)
        at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:418)
        at org.sonatype.nexus.threads.MDCAwareCallable.call(MDCAwareCallable.java:44)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

      And the merge metadata tasks tries to get UID locks on the group repositories children, who (presumably) are not giving it to them because they're running generate metadata:

      pxpool-1-thread-4 id=157 state=WAITING

      • waiting on <0x4a6f1c78> (a java.util.concurrent.Semaphore$FairSync)
      • locked <0x4a6f1c78> (a java.util.concurrent.Semaphore$FairSync)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:500)
        at org.sonatype.sisu.locks.LocalResourceLock.acquire(LocalResourceLockFactory.java:83)
        at org.sonatype.sisu.locks.AbstractSemaphoreResourceLock.lockShared(AbstractSemaphoreResourceLock.java:50)
        at org.sonatype.nexus.proxy.item.SisuLockResource.lockShared(SisuLockResource.java:38)
        at org.sonatype.nexus.proxy.item.DefaultRepositoryItemUidLock.lock(DefaultRepositoryItemUidLock.java:35)
        at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:757)
        at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:592)
        at org.sonatype.nexus.yum.internal.task.MergeMetadataTask.getBaseDirsOfMemberRepositories(MergeMetadataTask.java:142)
        at org.sonatype.nexus.yum.internal.task.MergeMetadataTask.doRun(MergeMetadataTask.java:102)
        at org.sonatype.nexus.yum.internal.task.MergeMetadataTask.doRun(MergeMetadataTask.java:61)
        at org.sonatype.nexus.scheduling.AbstractNexusTask.call(AbstractNexusTask.java:151)
        at org.sonatype.scheduling.DefaultScheduledTask.call(DefaultScheduledTask.java:418)
        at org.sonatype.nexus.threads.MDCAwareCallable.call(MDCAwareCallable.java:44)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

            People

            Assignee:
            rseddon Rich Seddon
            Reporter:
            rseddon Rich Seddon
            Last Updated By:
            Peter Lynch Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title