Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-25294

DockerFacetUtils.findAssetByContentDigest NXRM 3.0.0-M7 backwards compatibility code is non-performant under load

    XMLWordPrintable

    Details

      Description

      Under load conditions where

      • component db asset table is large
      • high number of docker proxy repos
      • high number of docker repository requests

      Thread dumps can contain significant numbers of calls to org.sonatype.nexus.repository.docker.internal.orient.DockerFacetUtils.findAssetByContentDigest

      Example thread stack:

      "qtp2012588533-789 <command>sql.select from asset where bucket = :bucket and attributes[docker][content_digest] = :propValue</command>" #789 prio=5 os_prio=0 tid=0x00007f3aac031000 nid=0x246e waiting on condition [0x00007f3a5d478000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000006d5a1a138> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
      	at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213)
      	at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266)
      	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355)
      	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294)
      	at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145)
      	at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPositionMap.get(OClusterPositionMap.java:435)
      	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:734)
      	at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
      	at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
      	at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.findByProperty(MetadataNodeEntityAdapter.java:149)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssetWithProperty(StorageTxImpl.java:348)
      	at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
      	at com.sun.proxy.$Proxy235.findAssetWithProperty(Unknown Source)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerFacetUtils.findAssetByContentDigest(DockerFacetUtils.java:184)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getManifestByDigest(DockerProxyFacetImpl.java:970)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e.CGLIB$getManifestByDigest$11(<generated>)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e$$FastClassByGuice$$e18134a6.invoke(<generated>)
      	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
      	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
      	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e.getManifestByDigest(<generated>)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getManifest(DockerProxyFacetImpl.java:936)
      	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getCachedContent(DockerProxyFacetImpl.java:324)
      	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.maybeGetCachedContent(ProxyFacetSupport.java:369)
      	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:229)
      

      This method call DockerFacetUtils.findAssetByContentDigest makes a database query sql.select from asset where bucket = :bucket and attributes[docker][content_digest] = :propValue that performs a full table scan of the Component database asset table due to an OrientDB bug. This slow query can block other threads making database queries to the point where all database connections available in the pool are exhausted for all db queries. Threads can continue to back up in NXRM until the total number of threads peak at the limit of the Jetty thread pool size and then the application will become unstable and non-responsive.

      The method in question has only been in place to support behaviour in version 3.0.0-M7 of NXRM.

      Expected

      Remove the old Nexus 3.0.0-m7 compatibility method DockerFacetUtils.findAssetByContentDigest as it is not performant and version 3.0.0-M7 is no longer supported.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              aishchenko Artem Ishchenko
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Michael Prescott Michael Prescott
              Team:
              NXRM - Trinity
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title