Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18083

potential slow processing of concurrent nuget requests using FindPackagesById()

    XMLWordPrintable

    Details

    • Story Points:
      0

      Description

      Concurrent processing of Nuget gallery types of requests using "FindPackagesById()" queries may become increasingly slow, potentially failing builds.

      These types of queries have at least two impacts.

      1. They lease all of the available HTTP outbound connections per route ( proxy repository remote host, currently 20 per route as of 3.13.0), impacting any other inbound requests that will also require an outbound HTTP request for that same proxy repository route in order to complete work.
      2. They potentially steal all of the available component database connections available for the entire repository instance ( currently 25 as of 3.13.0 ), impacting other non-related requests that also need a database connection to perform work.

      The problem is exasperated when

      • the parallel processing feature of a NuGet or MSBuild client is used so that concurrent requests are sent to the same Nexus instance from the same build
      • a Nuget build is configured with more than one source gallery instead of just a single Nexus group repository or single proxy repository
      • the nexus host is a virtual machine or one with limited I/O resources such as one being run in a cloud service - the network I/O capacity could have considerable impact on triggering this issue
      Example Request Format
      "GET /repository/nuget-group/FindPackagesById()?id='Microsoft.AspNetCore.Hosting' HTTP/1.0"
      

      Examining Nexus thread dumps taken at the time of concurrent requests may reveal two types of bottlenecks.

      Bottleneck 1: HTTPClient Connection Pool Leases

      A high number of waiting threads trying to lease an outbound HTTP connection from the Nuget proxy repository outbound HTTP connection pool. The 20 available connections are already leased by running threads.

      Example waiting thread on outbound Nuget HTTP connections
      qtp993807585-20058 id=20058 state=TIMED_WAITING
          - waiting on <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          - locked <0x106a98ec> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.parkUntil(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(Unknown Source)
          at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:378)
          at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
          at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
          at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
          at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
          at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
          at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
          at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
          at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
          at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
          at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
          at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.lambda$0(FilteredHttpClientSupport.java:56)
          at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport$$Lambda$406/1505474857.call(Unknown Source)
          at org.sonatype.nexus.repository.httpclient.internal.BlockingHttpClient.filter(BlockingHttpClient.java:116)
          at org.sonatype.nexus.repository.httpclient.FilteredHttpClientSupport.doExecute(FilteredHttpClientSupport.java:56)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
          at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.getPayload(NugetFeedFetcher.java:160)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:77)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
          at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
      
      

      Bottleneck 2: Database Connection Pool Exhaustion and Internal Record Locking

      Nuget Gallery Database WAITING Thread on finding component
      qtp993807585-20080 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20080 state=WAITING
          - waiting on <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
          - locked <0x6cdcbf25> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)
          at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
          at com.orientechnologies.common.collection.closabledictionary.OClosableEntry.acquireStateLock(OClosableEntry.java:84)
          at com.orientechnologies.common.collection.closabledictionary.OClosableLinkedContainer.acquire(OClosableLinkedContainer.java:292)
          at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.getFilledUpTo(OWOWCache.java:959)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.getFilledUpTo(ODurableComponent.java:132)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:762)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
          at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
          at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
          at com.sun.proxy.$Proxy225.findComponents(Unknown Source)
          at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310)
          at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.CGLIB$putMetadata$25(<generated>)
          at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b$$FastClassByGuice$$c1b28c7a.invoke(<generated>)
          at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
          at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
          at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
          at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
          at com.sonatype.nexus.repository.nuget.internal.proxy.NugetProxyGalleryFacet$$EnhancerByGuice$$d8e9779b.putMetadata(<generated>)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader$1.consume(NugetRemoteGalleryFacetSupport.java:396)
          at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.ended(FeedSplicer.java:109)
          at com.sonatype.nexus.repository.nuget.odata.XmlSplicer.consume(XmlSplicer.java:81)
          at com.sonatype.nexus.repository.nuget.odata.FeedSplicer.consumePage(FeedSplicer.java:58)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.parseFeed(NugetFeedFetcher.java:103)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher.cachePackageFeed(NugetFeedFetcher.java:86)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport$FeedLoader.call(NugetRemoteGalleryFacetSupport.java:392)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.passQueryToRemoteRepos(NugetRemoteGalleryFacetSupport.java:327)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:206)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
          at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
      
      Nuget Gallery Database WAITING Orderby Thread
      qtp993807585-20116 <command>sql.select from asset where (component.ci_name = :p0) and (bucket=#13:0 or bucket=#13:25) ORDER BY attributes.nuget.download_count DESC, component.ci_name ASC, component.version ASC LIMIT 40</command> id=20116 state=RUNNABLE
          at java.util.WeakHashMap.hash(Unknown Source)
          at java.util.WeakHashMap.get(Unknown Source)
          at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44)
          at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034)
          at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:187)
          at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:162)
          at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:291)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1725)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
          at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:329)
          at com.orientechnologies.orient.core.sql.method.misc.OSQLMethodField.execute(OSQLMethodField.java:62)
          at com.orientechnologies.orient.core.sql.method.OSQLMethodRuntime.execute(OSQLMethodRuntime.java:139)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemAbstract.transformValue(OSQLFilterItemAbstract.java:140)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemField.getValue(OSQLFilterItemField.java:138)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:379)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:384)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilterCondition.evaluate(OSQLFilterCondition.java:88)
          at com.orientechnologies.orient.core.sql.filter.OSQLFilter.evaluate(OSQLFilter.java:105)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.evaluateRecord(OCommandExecutorSQLResultsetAbstract.java:414)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.filter(OCommandExecutorSQLResultsetAbstract.java:404)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:609)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:522)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
          at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:371)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssets(StorageTxImpl.java:377)
          at sun.reflect.GeneratedMethodAccessor268.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
          at com.sun.proxy.$Proxy225.findAssets(Unknown Source)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findAssetsForQuery(NugetLocalGalleryFacetSupport.java:244)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.queryFeed(NugetLocalGalleryFacetSupport.java:211)
          at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.CGLIB$queryFeed$27(<generated>)
          at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9$$FastClassByGuice$$8717b62f.invoke(<generated>)
          at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
          at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
          at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
          at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
          at com.sonatype.nexus.repository.nuget.internal.NugetGroupGalleryFacet$$EnhancerByGuice$$e42c13d9.queryFeed(<generated>)
          at com.sonatype.nexus.repository.nuget.internal.NugetRemoteGalleryFacetSupport.feed(NugetRemoteGalleryFacetSupport.java:211)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.feed(NugetFeedHandler.java:69)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.doHandle(NugetFeedHandler.java:49)
          at com.sonatype.nexus.repository.nuget.internal.AbstractNugetHandler.handle(AbstractNugetHandler.java:46)
          at com.sonatype.nexus.repository.nuget.internal.NugetFeedHandler.handle(NugetFeedHandler.java:1)
      
      RUNNABLE db query expunging weak references
      qtp993807585-20176 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20176 state=RUNNABLE
          at java.util.WeakHashMap.expungeStaleEntries(Unknown Source)
          at java.util.WeakHashMap.getTable(Unknown Source)
          at java.util.WeakHashMap.get(Unknown Source)
          at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44)
          at com.orientechnologies.orient.core.cache.OLocalRecordCache.updateRecord(OLocalRecordCache.java:66)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2034)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
          at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
          at sun.reflect.GeneratedMethodAccessor269.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
          at com.sun.proxy.$Proxy225.findComponents(Unknown Source)
          at com.sonatype.nexus.repository.nuget.NugetComponentHelper.findComponent(NugetComponentHelper.java:71)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findComponent(NugetLocalGalleryFacetSupport.java:800)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.findOrCreateComponent(NugetLocalGalleryFacetSupport.java:790)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.createOrUpdateComponent(NugetLocalGalleryFacetSupport.java:725)
          at com.sonatype.nexus.repository.nuget.internal.NugetLocalGalleryFacetSupport.putMetadata(NugetLocalGalleryFacetSupport.java:310)
      
      Acquiring exclusive lock
      qtp993807585-20180 <command>sql.select from component where (ci_name = :p0 AND version = :p1) and (bucket=#13:25)</command> id=20180 state=WAITING
          - waiting on <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
          - locked <0x7aac32e3> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(Unknown Source)
          at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213)
          at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:766)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:742)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:656)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
          at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:202)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:170)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:449)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:455)
      

      Limited Workaround

      One should attempt to limit the concurrent requests to nuget-proxy repositories from the same build command.

      For example if nuget restore or nuget install is being used, one may pass the -DisableParallelProcessing argument:

      nuget restore -DisableParallelProcessing
      

      For dotnet CLI the argument is

      --disable-parallel
      

      Reference: https://github.com/dotnet/docs/blob/master/docs/core/tools/dotnet-restore.md

      For nuget CLI the argument is

      -DisableParallelProcessing
      

      Reference: https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore

      Related: https://github.com/NuGet/Home/issues/1556
      https://docs.microsoft.com/en-us/nuget/tools/cli-ref-restore#options
      https://docs.microsoft.com/en-us/nuget/tools/cli-ref-install#options

      Also only configure your builds to access a single nuget group per Nexus instance used.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jstephens Joseph Stephens
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch
              Team:
              NXRM - Cypher
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title