Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-7774

deadlock trying to read .meta/obr.xml after running task to merge metadata

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7, 2.9.2
    • Fix Version/s: 2.11.1, 3.0.0-m3
    • Component/s: OBR
    • Labels:
    • Story Points:
      0
    • Sprint:
      Sprint 32

      Description

      It is possible more than one thread can attempt to merge obr group metadata at the same time.

      This causes a virtual deadlock situation and prevents other threads trying to read the metadata.

      Example thread stacks that can be seen on dumps when in this situation:

      2014-07-18 02:27:17 WARN  [esh-1-thread-35509] <unknown-user> org.sonatype.nexus.obr.shadow.ObrShadowRepository - Could not sync shadow ObrShadowRepository(id=obr-dev-tmp-releases) for event RepositoryItemEventCacheUpdate(sender="Dev Temporary Release" [id=dev-tmp-release-proxy], dev-tmp-release-proxy:/com/com/service/obligor-common/1.0.0.0/obligor-common-1.0.0.0.jar)
      org.sonatype.nexus.proxy.StorageException: A storage exception occured!
      	at org.sonatype.nexus.obr.util.ObrUtils.updateObr(ObrUtils.java:337) ~[na:na]
      	at org.sonatype.nexus.obr.shadow.ObrShadowRepository.updateLink(ObrShadowRepository.java:146) ~[na:na]
      	at org.sonatype.nexus.obr.shadow.ObrShadowRepository.createLink(ObrShadowRepository.java:125) ~[na:na]
      	at org.sonatype.nexus.proxy.repository.AbstractShadowRepository.onRepositoryItemEvent(AbstractShadowRepository.java:146) ~[nexus-core-2.7.0-06.jar:2.7.0-06]
      	at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[na:na]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_45]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_45]
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78) [goodies-eventbus-1.7.4.jar:1.7.4]
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doCacheItem(AbstractProxyRepository.java:867) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.maven.maven2.M2Repository.doCacheItem(M2Repository.java:232) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveRemoteItem(AbstractProxyRepository.java:1346) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem0(AbstractProxyRepository.java:1091) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractProxyRepository.doRetrieveItem(AbstractProxyRepository.java:974) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.doRetrieveArtifactItem(AbstractMavenRepository.java:426) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.maven.AbstractMavenRepository.doRetrieveItem(AbstractMavenRepository.java:405) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.maven.maven2.M2Repository.doRetrieveItem(M2Repository.java:396) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:749) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.fetchItem(ItemHandlerSupport.java:191) [nexus-smartproxy-plugin-2.7.0-06.jar:2.7.0-06]
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.doFetchItem(ItemHandlerSupport.java:206) [nexus-smartproxy-plugin-2.7.0-06.jar:2.7.0-06]
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport.access$000(ItemHandlerSupport.java:49) [nexus-smartproxy-plugin-2.7.0-06.jar:2.7.0-06]
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEvent.perform(ItemHandlerSupport.java:255) [nexus-smartproxy-plugin-2.7.0-06.jar:2.7.0-06]
      	at com.sonatype.nexus.plugins.smartproxy.event.internal.handler.ItemHandlerSupport$PreemptiveFetchItemEventInspector.inspect(ItemHandlerSupport.java:275) [nexus-smartproxy-plugin-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.events.EventSubscriberHost$AsynchronousEventInspectorAdapter$1.run(EventSubscriberHost.java:253) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.threads.MDCAwareRunnable.run(MDCAwareRunnable.java:42) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [shiro-core-1.2.2.jar:1.2.2]
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [shiro-core-1.2.2.jar:1.2.2]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
      	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
      Caused by: org.sonatype.nexus.proxy.StorageException: A storage exception occured!
      	at org.sonatype.nexus.obr.util.ObrUtils.retrieveObrItem(ObrUtils.java:192) ~[na:na]
      	at org.sonatype.nexus.obr.util.ObrUtils.updateObr(ObrUtils.java:315) ~[na:na]
      	... 33 common frames omitted
      Caused by: org.sonatype.nexus.proxy.AccessDeniedException: Access denied on repository ID='obr-dev-tmp-releases', path='/.meta/obr.xml', action='read'!
      	at org.sonatype.nexus.proxy.access.DefaultAccessManager.decide(DefaultAccessManager.java:49) ~[nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractRepository.checkConditions(AbstractRepository.java:1171) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:590) [nexus-core-2.7.0-06.jar:2.7.0-06]
      	at org.sonatype.nexus.obr.util.ObrUtils.retrieveObrItem(ObrUtils.java:180) ~[na:na]
      	... 34 common frames omitted
      

      Example blocked thread:

      qtp373859501-284195 id=284195 state=WAITING
          - waiting on <0x245d3463> (a java.util.concurrent.Semaphore$FairSync)
          - locked <0x245d3463> (a java.util.concurrent.Semaphore$FairSync)
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
          at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:500)
          at org.sonatype.sisu.locks.LocalResourceLock.acquire(LocalResourceLockFactory.java:83)
          at org.sonatype.sisu.locks.AbstractSemaphoreResourceLock.lockExclusive(AbstractSemaphoreResourceLock.java:60)
          at org.sonatype.nexus.proxy.item.SisuLockResource.lockExclusively(SisuLockResource.java:45)
          at org.sonatype.nexus.proxy.item.DefaultRepositoryItemUidLock.lock(DefaultRepositoryItemUidLock.java:39)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:980)
          at org.sonatype.nexus.obr.metadata.DefaultObrResourceWriter.close(DefaultObrResourceWriter.java:140)
          at org.sonatype.nexus.obr.util.ObrUtils.close(ObrUtils.java:396)
          at org.sonatype.nexus.obr.group.ObrGroupRepository.mergeObrMetadata(ObrGroupRepository.java:240)
          at org.sonatype.nexus.obr.group.ObrGroupRepository.doRetrieveItem(ObrGroupRepository.java:129)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:749)
          at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:592)
          at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155)
          at org.sonatype.nexus.web.content.NexusContentServlet.doGet(NexusContentServlet.java:359)
          at org.sonatype.nexus.web.content.NexusContentServlet.service(NexusContentServlet.java:331)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      

      Competing threads:

      at sun.misc.Unsafe.park(Native Method) 
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) 
      at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:500) 
      at org.sonatype.sisu.locks.LocalResourceLock.acquire(LocalResourceLockFactory.java:83) 
      at org.sonatype.sisu.locks.AbstractSemaphoreResourceLock.lockExclusive(AbstractSemaphoreResourceLock.java:69) 
      at org.sonatype.nexus.proxy.item.SisuLockResource.lockExclusively(SisuLockResource.java:45) 
      at org.sonatype.nexus.proxy.item.DefaultRepositoryItemUidLock.lock(DefaultRepositoryItemUidLock.java:39) 
      at org.sonatype.nexus.proxy.storage.local.fs.DefaultFSPeer.storeItem(DefaultFSPeer.java:157) 
      at org.sonatype.nexus.proxy.storage.local.fs.DefaultFSLocalRepositoryStorage.storeItem(DefaultFSLocalRepositoryStorage.java:331) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:996) 
      at org.sonatype.nexus.obr.metadata.DefaultObrResourceWriter.close(DefaultObrResourceWriter.java:140) 
      at org.sonatype.nexus.obr.util.ObrUtils.close(ObrUtils.java:396) 
      at org.sonatype.nexus.obr.group.ObrGroupRepository.mergeObrMetadata(ObrGroupRepository.java:240) 
      at org.sonatype.nexus.obr.group.ObrGroupRepository.doRetrieveItem(ObrGroupRepository.java:129) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:749) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:592) 
      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155) 
      at org.sonatype.nexus.web.content.NexusContentServlet.doGet(NexusContentServlet.java:359) 
      at org.sonatype.nexus.web.content.NexusContentServlet.service(NexusContentServlet.java:331) 
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      
      at sun.misc.Unsafe.park(Native Method) 
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) 
      at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:500) 
      at org.sonatype.sisu.locks.LocalResourceLock.acquire(LocalResourceLockFactory.java:83) 
      at org.sonatype.sisu.locks.AbstractSemaphoreResourceLock.lockExclusive(AbstractSemaphoreResourceLock.java:60) 
      at org.sonatype.nexus.proxy.item.SisuLockResource.lockExclusively(SisuLockResource.java:45) 
      at org.sonatype.nexus.proxy.item.DefaultRepositoryItemUidLock.lock(DefaultRepositoryItemUidLock.java:39) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.storeItem(AbstractRepository.java:980) 
      at org.sonatype.nexus.obr.metadata.DefaultObrResourceWriter.close(DefaultObrResourceWriter.java:140) 
      at org.sonatype.nexus.obr.util.ObrUtils.close(ObrUtils.java:396) 
      at org.sonatype.nexus.obr.group.ObrGroupRepository.mergeObrMetadata(ObrGroupRepository.java:240) 
      at org.sonatype.nexus.obr.group.ObrGroupRepository.doRetrieveItem(ObrGroupRepository.java:129) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:749) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:592) 
      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155) 
      at org.sonatype.nexus.web.content.NexusContentServlet.doGet(NexusContentServlet.java:359) 
      at org.sonatype.nexus.web.content.NexusContentServlet.service(NexusContentServlet.java:331) 
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      

      Waiting threads:

      at sun.misc.Unsafe.park(Native Method) 
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964) 
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282) 
      at java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:500) 
      at org.sonatype.sisu.locks.LocalResourceLock.acquire(LocalResourceLockFactory.java:83) 
      at org.sonatype.sisu.locks.AbstractSemaphoreResourceLock.lockShared(AbstractSemaphoreResourceLock.java:50) 
      at org.sonatype.nexus.proxy.item.SisuLockResource.lockShared(SisuLockResource.java:39) 
      at org.sonatype.nexus.proxy.item.DefaultRepositoryItemUidLock.lock(DefaultRepositoryItemUidLock.java:36) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:746) 
      at org.sonatype.nexus.proxy.repository.AbstractRepository.retrieveItem(AbstractRepository.java:592) 
      at org.sonatype.nexus.proxy.router.DefaultRepositoryRouter.retrieveItem(DefaultRepositoryRouter.java:155) 
      at org.sonatype.nexus.web.content.NexusContentServlet.doGet(NexusContentServlet.java:359) 
      at org.sonatype.nexus.web.content.NexusContentServlet.service(NexusContentServlet.java:331) 
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
      

        Attachments

          Activity

            People

            Assignee:
            plynch Peter Lynch
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                tigCommentSecurity.panel-title