[NEXUS-18083] potential slow processing of concurrent nuget requests using FindPackagesById() - Sonatype JIRA

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 3.13.0
Fix Version/s: None
Component/s: NuGet
Labels:

Story Points:
0

Description

Concurrent processing of Nuget gallery types of requests using "FindPackagesById()" queries may become increasingly slow, potentially failing builds.

These types of queries have at least two impacts.

They lease all of the available HTTP outbound connections per route ( proxy repository remote host, currently 20 per route as of 3.13.0), impacting any other inbound requests that will also require an outbound HTTP request for that same proxy repository route in order to complete work.
They potentially steal all of the available component database connections available for the entire repository instance ( currently 25 as of 3.13.0 ), impacting other non-related requests that also need a database connection to perform work.

The problem is exasperated when

the parallel processing feature of a NuGet or MSBuild client is used so that concurrent requests are sent to the same Nexus instance from the same build
a Nuget build is configured with more than one source gallery instead of just a single Nexus group repository or single proxy repository
the nexus host is a virtual machine or one with limited I/O resources such as one being run in a cloud service - the network I/O capacity could have considerable impact on triggering this issue

Example Request Format

"GET /repository/nuget-group/FindPackagesById()?id='Microsoft.AspNetCore.Hosting' HTTP/1.0"

Examining Nexus thread dumps taken at the time of concurrent requests may reveal two types of bottlenecks.

Bottleneck 1: HTTPClient Connection Pool Leases

A high number of waiting threads trying to lease an outbound HTTP connection from the Nuget proxy repository outbound HTTP connection pool. The 20 available connections are already leased by running threads.

Example waiting thread on outbound Nuget HTTP connections

qtp993807585-20058 id=20058 state=TIMED_WAITING
    - waiting on <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkUntil(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(Unknown Source)
    at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:378)
    at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
    at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
    at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.lambda$0(FilteredHttpClientSupport.java:56)
    at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport$$Lambda$406/1505474857.call(Unknown Source)
    at org.sonatype.nexus.repository.httpclient.internal.BlockingHttpClient.filter(BlockingHttpClient.java:116)
    at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.doExecute(FilteredHttpClientSupport.java:56)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.getPayload(NugetFeedFetcher.java:160)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:77)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
    at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)

Bottleneck 2: Database Connection Pool Exhaustion and Internal Record Locking

Nuget Gallery Database WAITING Thread on finding component

qtp993807585-20080 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20080 state=WAITING
    - waiting on <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    - locked <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
    at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
    at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
    at com.orientechnologies.common.collection.closabledictionary.OClosableEntry.acquireStateLock(OClosableEntry.java:84)
    at com.orientechnologies.common.collection.closabledictionary.OClosableLinkedContainer.acquire(OClosableLinkedContainer.java:292)
    at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.getFilledUpTo(OWOWCache.java:959)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.getFilledUpTo(ODurableComponent.java:132)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:762)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
    at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
    at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
    at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
    at com.sun.proxy.$Proxy225.findComponents(Unknown Source)
    at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310)
    at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.CGLIB$putMetadata$25(<generated>)
    at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b$$FastClassByGuice$$c1b28c7a.invoke(<generated>)
    at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
    at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
    at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
    at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
    at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.putMetadata(<generated>)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader$1.consume(NugetRemoteGalleryFacetSupport.java:396)
    at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.ended(FeedSplicer.java:109)
    at com.sonatype.nexus.repository.nuget.odata.XmlSplicer.consume(XmlSplicer.java:81)
    at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.consumePage(FeedSplicer.java:58)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.parseFeed(NugetFeedFetcher.java:103)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:86)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
    at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)

Nuget Gallery Database WAITING Orderby Thread

qtp993807585-20116 <command>sql.select from asset where (component.ci_name = :p0) and (bucket=#13:0 or bucket=#13:25) ORDER BY attributes.nuget.download_count DESC, component.ci_name ASC, component.version ASC LIMIT 40</command> id=20116 state=RUNNABLE
    at java.util.WeakHashMap.hash(Unknown Source)
    at java.util.WeakHashMap.get(Unknown Source)
    at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44)
    at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:187)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:162)
    at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:291)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1725)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
    at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:329)
    at com.orientechnologies.orient.core.sql.method.misc.OSQLMethodField.execute(OSQLMethodField.java:62)
    at com.orientechnologies.orient.core.sql.method.OSQLMethodRuntime.execute(OSQLMethodRuntime.java:139)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemAbstract.transformValue(OSQLFilterItemAbstract.java:140)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemField.getValue(OSQLFilterItemField.java:138)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:379)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
    at com.orientechnologies.orient.core.sql.filter.OSQLFilter.evaluate(OSQLFilter.java:105)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.evaluateRecord(OCommandExecutorSQLResultsetAbstract.java:414)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.filter(OCommandExecutorSQLResultsetAbstract.java:404)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:609)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:522)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:371)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:377)
    at sun.reflect.GeneratedMethodAccessor268.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
    at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
    at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
    at com.sun.proxy.$Proxy225.findAssets(Unknown Source)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findAssetsForQuery(NugetLocalGalleryFacetSupport.java:244)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.queryFeed(NugetLocalGalleryFacetSupport.java:211)
    at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.CGLIB$queryFeed$27(<generated>)
    at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9$$FastClassByGuice$$8717b62f.invoke(<generated>)
    at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
    at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
    at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
    at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
    at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
    at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.queryFeed(<generated>)
    at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:211)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
    at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
    at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)

RUNNABLE db query expunging weak references

qtp993807585-20176 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20176 state=RUNNABLE
    at java.util.WeakHashMap.expungeStaleEntries(Unknown Source)
    at java.util.WeakHashMap.getTable(Unknown Source)
    at java.util.WeakHashMap.get(Unknown Source)
    at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44)
    at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
    at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
    at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
    at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
    at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
    at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
    at com.sun.proxy.$Proxy225.findComponents(Unknown Source)
    at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725)
    at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310)

Acquiring exclusive lock

qtp993807585-20180 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20180 state=WAITING
    - waiting on <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    - locked <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(Unknown Source)
    at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213)
    at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266)
    at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355)
    at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:766)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742)
    at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
    at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
    at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
    at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
    at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
    at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)

Limited Workaround

One should attempt to limit the concurrent requests to nuget-proxy repositories from the same build command.

For example if nuget restore or nuget install is being used, one may pass the -DisableParallelProcessing argument:

nuget restore -DisableParallelProcessing

For dotnet CLI the argument is

--disable-parallel

Reference: https://github.com/dotnet/docs/blob/master/docs/core/tools/dotnet-restore.md

For nuget CLI the argument is

-DisableParallelProcessing

Reference: https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore

Related: https://github.com/NuGet/Home/issues/1556
https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore#options
https://docs.microsoft.com/en-us/nuget/tools/cli-ref-install#options

Also only configure your builds to access a single nuget group per Nexus instance used.

Attachments

Issue Links

is related to

NEXUS-12983 NuGet FindPackagesById queries may perform slowly possibly leading to general non-responsiveness

Closed

NEXUS-17501 Caching of NuGet metadata causes thread serialization, query slowdowns under load

Closed

Activity

People

Assignee:: Joseph Stephens

Reporter:: Peter Lynch

Last Updated By:: Joe Tom

Team:: NXRM - Cypher

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 09/20/18 04:30 PM

Updated:: 08/05/21 11:41 PM

Resolved:: 12/12/18 11:26 AM

Date of First Response:: 29/Oct/18 4:49 PM