Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-12995

Group repository request during OWALPageChangesPortion.readData may block other threads

    XMLWordPrintable

    Details

      Description

      When there are large numbers of Maven repositories in a Nexus 3 instance (~500+) processing of GET requests to group repositories becomes unusably slow.

      Thread dump shows that this is not a deadlock, but it causes request threads to be serialized.

      For example, this running thread:

      qtp1995359069-302 id=302 state=RUNNABLE
      at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:285)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageChangesPortion.readData(OWALPageChangesPortion.java:227)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALPageChangesPortion.getIntValue(OWALPageChangesPortion.java:85)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.getIntValue(ODurablePage.java:116)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPage.doDefragmentation(OClusterPage.java:434)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPage.appendRecord(OClusterPage.java:89)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.addEntry(OPaginatedCluster.java:2005)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createSinglePageRecord(OPaginatedCluster.java:600)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createDataRecord(OPaginatedCluster.java:486)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.createRecord(OPaginatedCluster.java:451)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:3032)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:3652)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1439)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:539)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:104)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2840)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2809)
      at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:171)
      at sun.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
      at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      at com.sun.proxy.$Proxy238.commit(Unknown Source)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
      at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl$$EnhancerByGuice$$a9e791b8.doPut(<generated>)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:200)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetUtils.putWithHashes(MavenFacetUtils.java:213)
      at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.cache(MavenGroupFacet.java:158)
      at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.mergeAndCache(MavenGroupFacet.java:135)
      at org.sonatype.nexus.repository.maven.internal.group.MergingGroupHandler.doGet(MergingGroupHandler.java:98)
      at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:79)
      

       
      Is blocking 2 other threads with this stack:

      qtp1995359069-325 id=325 state=WAITING
      - waiting on <0x3e092391> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      - locked <0x3e092391> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      owned by qtp1995359069-302 id=302
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:169)
      at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:101)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:519)
      at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.iterateEntriesBetween(OSBTree.java:765)
      at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.iterateEntriesBetween(OSBTreeIndexEngine.java:185)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doIterateIndexEntriesBetween(OAbstractPaginatedStorage.java:2064)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.iterateIndexEntriesBetween(OAbstractPaginatedStorage.java:2052)
      at com.orientechnologies.orient.core.index.OIndexMultiValues.iterateEntriesBetween(OIndexMultiValues.java:275)
      at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.iterateEntriesBetween(OIndexAbstractDelegate.java:104)
      at com.orientechnologies.orient.core.index.OIndexTxAwareMultiValue.iterateEntriesBetween(OIndexTxAwareMultiValue.java:339)
      at com.orientechnologies.orient.core.sql.operator.OQueryOperatorEquals.executeIndexQuery(OQueryOperatorEquals.java:173)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2178)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:997)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:209)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:526)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:508)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:484)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:2615)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:2561)
      at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
      at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.findByProperty(MetadataNodeEntityAdapter.java:148)
      at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssetWithProperty(StorageTxImpl.java:307)
      at sun.reflect.GeneratedMethodAccessor340.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
      at com.sun.proxy.$Proxy238.findAssetWithProperty(Unknown Source)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetUtils.findAsset(MavenFacetUtils.java:132)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.get(MavenFacetImpl.java:174)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl$$EnhancerByGuice$$a9e791b8.CGLIB$get$10(<generated>)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl$$EnhancerByGuice$$a9e791b8$$FastClassByGuice$$b1b9dd3a.invoke(<generated>)
      at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl$$EnhancerByGuice$$a9e791b8.get(<generated>)
      at org.sonatype.nexus.repository.maven.internal.proxy.MavenProxyFacet.getCachedContent(MavenProxyFacet.java:58)
      at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:166)
      at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:49)

      Full thread dump is attached.
       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              wwannemacher Wes Wannemacher
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title