Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28362

slow request processing due to group trying to locate blobs in the wrong blobstore

    XMLWordPrintable

    Details

      Description

      Customer is noticing NuGet V3 requests to a group repo taking a long time to complete - 4-5 plus minutes. Eventually some of these end with Jetty idle. timeout.

      3.30.1-01 running in Docker image.

      On examining the logs, logging suggests NXRM may be looking in the wrong blobstore for an asset properties file. Example:

      2021-07-14 08:56:14,581+0000 WARN [qtp858140173-81850] *UNKNOWN org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob a603385e-4365-4fa4-b50b-a96e204d5dd1 (/nexus-data/blobs/nuget-group-libs/content/vol-23/chap-19/a603385e-4365-4fa4-b50b-a96e204d5dd1.properties)
      2021-07-14 08:56:14,613+0000 WARN [qtp858140173-81850] *UNKNOWN org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.storage.MissingBlobException: Blob nuget-proxy-libs-nuget.org@693EFE57-0FDC92FE-26D78B5F-126A415A-85B1D3A7:a603385e-4365-4fa4-b50b-a96e204d5dd1 exists in metadata, but is missing from the blobstore)

      /nexus-data/blobs/nuget-group-libs is the blobstore for the group repo.

      nuget-proxy-libs-nuget.org proxy repo has its blobstore configured to be different than that.

      When a script is used to try and find missing blobs in blobstores NO MISSING ASSETS ARE FOUND.

      Also, logging suggests that each time a properties file is looked for, the time it takes to fail finding the file varies to multiple seconds per check. This is where the majority of time is spent per request - this is not an HTTP outbound latency issue.

      The customer tells us that no content files are present at all in the group blobstore. However the same files reported missing by NXRM logging are present in the proxy repo blobstore.

      Thread stacks for the retries look like this:

      at java.lang.Thread.sleep(Native Method)
      at org.sonatype.nexus.transaction.RetryController.backoff(RetryController.java:236)
      at org.sonatype.nexus.transaction.RetryController.allowRetry(RetryController.java:212)
      at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:217)
      at sun.reflect.GeneratedMethodAccessor532.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      at com.sun.proxy.$Proxy240.allowRetry(Unknown Source)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:82)
      at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
      at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
      at com.sonatype.nexus.repository.nuget.orient.internal.v3.OrientNugetV3ProxyFacetImpl$$EnhancerByGuice$$8ec4a4c7.getItem()
      at com.sonatype.nexus.repository.nuget.orient.internal.OrientNugetProxyFacet.getCachedContent(OrientNugetProxyFacet.java:74)
      at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.maybeGetCachedContent(ProxyFacetSupport.java:383)
      at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:235)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationPageFacet.downloadRegistrationPage(NugetV3GroupRegistrationPageFacet.java:107)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationPageFacet.rewritePage(NugetV3GroupRegistrationPageFacet.java:57)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationIndexFacet.rewriteIndexes(NugetV3GroupRegistrationIndexFacet.java:81)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationIndexFacet.lambda$0(NugetV3GroupRegistrationIndexFacet.java:52)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationIndexFacet$$Lambda$1558/1760791316.apply(Unknown Source)
      at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
      at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
      at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationIndexFacet.mergeResponses(NugetV3GroupRegistrationIndexFacet.java:62)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationHandler.mergeIndexes(NugetV3GroupRegistrationHandler.java:93)
      at com.sonatype.nexus.repository.nuget.internal.v3.NugetV3GroupRegistrationHandler.doGet(NugetV3GroupRegistrationHandler.java:68)
      at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:100)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at com.sonatype.nexus.repository.nuget.internal.NugetSimpleHandlers.lambda$0(NugetSimpleHandlers.java:26)
      at com.sonatype.nexus.repository.nuget.internal.NugetSimpleHandlers$$Lambda$705/2073077092.handle(Unknown Source)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at com.sonatype.nexus.repository.nuget.internal.NugetSimpleHandlers.lambda$1(NugetSimpleHandlers.java:33)
      at com.sonatype.nexus.repository.nuget.internal.NugetSimpleHandlers$$Lambda$709/234237477.handle(Unknown Source)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at org.sonatype.nexus.repository.storage.LastDownloadedHandler.handle(LastDownloadedHandler.java:59)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at com.sonatype.nexus.repository.nuget.internal.NugetGroupVersionHandler.handle(NugetGroupVersionHandler.java:71)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at org.sonatype.nexus.repository.routing.internal.RoutingRuleHandler.handle(RoutingRuleHandler.java:52)
      at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
      at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
      
      

      Diagnosis

      The problem of a NuGet Group Repository looking in the blobstore of its member repository for group repository assets instead of its own blobstore only occurs if the group blobstore and the member repository repository blobstores are different.

      Expected

      Make NuGet group repositories only look in their own blobstore for their own blobs.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Mahendra Surani Mahendra Surani
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title