Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-23800

Race condition in lazy maven metadata rebuild causes build failures, slow builds

    XMLWordPrintable

    Details

      Description

      1. Configure an s3 blob store in Nexus Repo 3.22.1
      2. Create a hosted maven snapshot repo that uses the s3 blob store
      3. Deploy two GAV's into two maven snapshot repository (e.g. 1.2.3-SNAPSHOT and 1.2.4-SNAPAHOT
      4. Create a cleanup policy that completely removes one of the GAV's. This can be done via regex in the policy's conditions (".1.2.4-SNAPSHOT.")
      5. Run the cleanup task, and let it finish
      6. Simultaneously request the GA level metadata for the snapshot:
      curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml & curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml & curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml &
      

      The above reproduce steps will periodically cause 500 responses from Nexus Repo. The logs show that the component database is referencing a soft deleted blob:

      2020-05-05 14:04:40,496-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 75ca6995-e8f4-497f-9285-716769e64730 attributes: s3://rich-nxrm3-s3/content/vol-29/chap-37/75ca6995-e8f4-497f-9285-716769e64730.properties {deletedDateTime=1588705458575, deleted=true, @BlobStore.created-by=admin, creationTime=1588705409287, @BlobStore.created-by-ip=127.0.0.1, @BlobStore.content-type=application/xml, sha1=a68071a6a4d033994a32afbf0fdb0812926ecaba, @BlobStore.blob-name=org/foo/bar/project/maven-metadata.xml, deletedReason=Updating asset AttachedEntityId

      Unknown macro: {asset->#59}

      , @Bucket.repo-name=snapshots2, size=278}
      2020-05-05 14:04:40,496-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.storage.MissingBlobException: Blob rich-nxrm3-s3-promoted-promoted@09D8B6F0-AB3A8DF9-E6138B67-E24F7A75-C6857C62:75ca6995-e8f4-497f-9285-716769e64730 exists in metadata, but is missing from the blobstore)
      2020-05-05 14:04:40,497-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/snapshots2/org/foo/bar/project/maven-metadata.xml
      org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
      at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:206)
      at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      at com.sun.proxy.$Proxy226.allowRetry(Unknown Source)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:82)
      at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)

      After the initial 500 response(s) every subsequent request for the GA maven-metadata does succeed, but every single one triggers a rebuild of the maven-metadata.xml:

      2020-05-05 14:06:49,278-0500 INFO [qtp1884423982-53] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

      Unknown macro: {path='org/foo/bar/project/maven-metadata.xml', fileName='maven-metadata.xml', hashType=null}

      hasn't changed, skipping
      2020-05-05 14:07:02,868-0500 INFO [qtp1884423982-183] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

      hasn't changed, skipping
      2020-05-05 14:08:13,914-0500 INFO [qtp1884423982-296] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

      Unknown macro: {path='org/foo/bar/project/maven-metadata.xml', fileName='maven-metadata.xml', hashType=null}

      hasn't changed, skipping

      Rebuilding the GA metadata is a slow operation on s3, and so after the initial failure it causes builds to run very slowly.

        Attachments

          Activity

            People

            Assignee:
            oshiyanbade Olu Shiyanbade
            Reporter:
            rseddon Rich Seddon
            Last Updated By:
            Michael Prescott Michael Prescott
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title