Under load conditions where
- component db asset table is large
- high number of docker proxy repos
- high number of docker repository requests
Thread dumps can contain significant numbers of calls to org.sonatype.nexus.repository.docker.internal.orient.DockerFacetUtils.findAssetByContentDigest
Example thread stack:
"qtp2012588533-789 <command>sql.select from asset where bucket = :bucket and attributes[docker][content_digest] = :propValue</command>" #789 prio=5 os_prio=0 tid=0x00007f3aac031000 nid=0x246e waiting on condition [0x00007f3a5d478000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006d5a1a138> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145) at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPositionMap.get(OClusterPositionMap.java:435) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:734) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.findByProperty(MetadataNodeEntityAdapter.java:149) at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssetWithProperty(StorageTxImpl.java:348) at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy235.findAssetWithProperty(Unknown Source) at org.sonatype.nexus.repository.docker.internal.orient.DockerFacetUtils.findAssetByContentDigest(DockerFacetUtils.java:184) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getManifestByDigest(DockerProxyFacetImpl.java:970) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e.CGLIB$getManifestByDigest$11(<generated>) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e$$FastClassByGuice$$e18134a6.invoke(<generated>) at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57) at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl$$EnhancerByGuice$$adff9b8e.getManifestByDigest(<generated>) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getManifest(DockerProxyFacetImpl.java:936) at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.getCachedContent(DockerProxyFacetImpl.java:324) at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.maybeGetCachedContent(ProxyFacetSupport.java:369) at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:229)
This method call DockerFacetUtils.findAssetByContentDigest makes a database query sql.select from asset where bucket = :bucket and attributes[docker][content_digest] = :propValue that performs a full table scan of the Component database asset table due to an OrientDB bug. This slow query can block other threads making database queries to the point where all database connections available in the pool are exhausted for all db queries. Threads can continue to back up in NXRM until the total number of threads peak at the limit of the Jetty thread pool size and then the application will become unstable and non-responsive.
The method in question has only been in place to support behaviour in version 3.0.0-M7 of NXRM.
Expected
Remove the old Nexus 3.0.0-m7 compatibility method DockerFacetUtils.findAssetByContentDigest as it is not performant and version 3.0.0-M7 is no longer supported.
- is related to
-
NEXUS-26379 DockerFacetUtils.findAssetByContentDigest() causes poor performance under load
-
- Closed
-
- relates
-
-
- Closed
-
