Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Cannot Reproduce
-
Affects Version/s: 3.13.0
-
Fix Version/s: None
-
Component/s: NuGet
-
Labels:
-
Story Points:0
Description
Concurrent processing of Nuget gallery types of requests using "FindPackagesById()" queries may become increasingly slow, potentially failing builds.
These types of queries have at least two impacts.
- They lease all of the available HTTP outbound connections per route ( proxy repository remote host, currently 20 per route as of 3.13.0), impacting any other inbound requests that will also require an outbound HTTP request for that same proxy repository route in order to complete work.
- They potentially steal all of the available component database connections available for the entire repository instance ( currently 25 as of 3.13.0 ), impacting other non-related requests that also need a database connection to perform work.
The problem is exasperated when
- the parallel processing feature of a NuGet or MSBuild client is used so that concurrent requests are sent to the same Nexus instance from the same build
- a Nuget build is configured with more than one source gallery instead of just a single Nexus group repository or single proxy repository
- the nexus host is a virtual machine or one with limited I/O resources such as one being run in a cloud service - the network I/O capacity could have considerable impact on triggering this issue
"GET /repository/nuget-group/FindPackagesById()?id='Microsoft.AspNetCore.Hosting' HTTP/1.0"
Examining Nexus thread dumps taken at the time of concurrent requests may reveal two types of bottlenecks.
Bottleneck 1: HTTPClient Connection Pool Leases
A high number of waiting threads trying to lease an outbound HTTP connection from the Nuget proxy repository outbound HTTP connection pool. The 20 available connections are already leased by running threads.
qtp993807585-20058 id=20058 state=TIMED_WAITING - waiting on <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkUntil(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(Unknown Source) at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:378) at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246) at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.lambda$0(FilteredHttpClientSupport.java:56) at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport$$Lambda$406/1505474857.call(Unknown Source) at org.sonatype.nexus.repository.httpclient.internal.BlockingHttpClient.filter(BlockingHttpClient.java:116) at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.doExecute(FilteredHttpClientSupport.java:56) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.getPayload(NugetFeedFetcher.java:160) at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:77) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49) at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
Bottleneck 2: Database Connection Pool Exhaustion and Internal Record Locking
qtp993807585-20080 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20080 state=WAITING - waiting on <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) - locked <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source) at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source) at com.orientechnologies.common.collection.closabledictionary.OClosableEntry.acquireStateLock(OClosableEntry.java:84) at com.orientechnologies.common.collection.closabledictionary.OClosableLinkedContainer.acquire(OClosableLinkedContainer.java:292) at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.getFilledUpTo(OWOWCache.java:959) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.getFilledUpTo(ODurableComponent.java:132) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:762) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455) at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy225.findComponents(Unknown Source) at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310) at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.CGLIB$putMetadata$25(<generated>) at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b$$FastClassByGuice$$c1b28c7a.invoke(<generated>) at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.putMetadata(<generated>) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader$1.consume(NugetRemoteGalleryFacetSupport.java:396) at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.ended(FeedSplicer.java:109) at com.sonatype.nexus.repository.nuget.odata.XmlSplicer.consume(XmlSplicer.java:81) at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.consumePage(FeedSplicer.java:58) at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.parseFeed(NugetFeedFetcher.java:103) at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:86) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49) at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
qtp993807585-20116 <command>sql.select from asset where (component.ci_name = :p0) and (bucket=#13:0 or bucket=#13:25) ORDER BY attributes.nuget.download_count DESC, component.ci_name ASC, component.version ASC LIMIT 40</command> id=20116 state=RUNNABLE at java.util.WeakHashMap.hash(Unknown Source) at java.util.WeakHashMap.get(Unknown Source) at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44) at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:187) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:162) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:291) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1725) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:329) at com.orientechnologies.orient.core.sql.method.misc.OSQLMethodField.execute(OSQLMethodField.java:62) at com.orientechnologies.orient.core.sql.method.OSQLMethodRuntime.execute(OSQLMethodRuntime.java:139) at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemAbstract.transformValue(OSQLFilterItemAbstract.java:140) at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemField.getValue(OSQLFilterItemField.java:138) at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:379) at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88) at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384) at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88) at com.orientechnologies.orient.core.sql.filter.OSQLFilter.evaluate(OSQLFilter.java:105) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.evaluateRecord(OCommandExecutorSQLResultsetAbstract.java:414) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.filter(OCommandExecutorSQLResultsetAbstract.java:404) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:609) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:522) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170) at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:371) at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:377) at sun.reflect.GeneratedMethodAccessor268.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy225.findAssets(Unknown Source) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findAssetsForQuery(NugetLocalGalleryFacetSupport.java:244) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.queryFeed(NugetLocalGalleryFacetSupport.java:211) at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.CGLIB$queryFeed$27(<generated>) at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9$$FastClassByGuice$$8717b62f.invoke(<generated>) at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.queryFeed(<generated>) at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:211) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49) at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46) at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
qtp993807585-20176 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20176 state=RUNNABLE at java.util.WeakHashMap.expungeStaleEntries(Unknown Source) at java.util.WeakHashMap.getTable(Unknown Source) at java.util.WeakHashMap.get(Unknown Source) at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44) at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455) at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy225.findComponents(Unknown Source) at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725) at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310)
qtp993807585-20180 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20180 state=WAITING - waiting on <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) - locked <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(Unknown Source) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:766) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202) at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449) at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
Limited Workaround
One should attempt to limit the concurrent requests to nuget-proxy repositories from the same build command.
For example if nuget restore or nuget install is being used, one may pass the -DisableParallelProcessing argument:
nuget restore -DisableParallelProcessing
For dotnet CLI the argument is
--disable-parallel
Reference: https://github.com/dotnet/docs/blob/master/docs/core/tools/dotnet-restore.md
For nuget CLI the argument is
-DisableParallelProcessing
Reference: https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore
Related: https://github.com/NuGet/Home/issues/1556
https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore#options
https://docs.microsoft.com/en-us/nuget/tools/cli-ref-install#options
Also only configure your builds to access a single nuget group per Nexus instance used.
Attachments
Issue Links
- is related to
-
NEXUS-12983 NuGet FindPackagesById queries may perform slowly possibly leading to general non-responsiveness
-
- Closed
-
-
NEXUS-17501 Caching of NuGet metadata causes thread serialization, query slowdowns under load
-
- Closed
-