Race condition in lazy maven metadata rebuild causes build failures, slow builds

1. Configure an s3 blob store in Nexus Repo 3.22.1
2. Create a hosted maven snapshot repo that uses the s3 blob store
3. Deploy two GAV's into two maven snapshot repository (e.g. 1.2.3-SNAPSHOT and 1.2.4-SNAPAHOT
4. Create a cleanup policy that completely removes one of the GAV's. This can be done via regex in the policy's conditions (".1.2.4-SNAPSHOT.")
5. Run the cleanup task, and let it finish
6. Simultaneously request the GA level metadata for the snapshot:

curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml & curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml & curl  http://localhost:8081/repository/snapshots2/org/foo/bar/project/maven-metadata.xml &

The above reproduce steps will periodically cause 500 responses from Nexus Repo. The logs show that the component database is referencing a soft deleted blob:

2020-05-05 14:04:40,496-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 75ca6995-e8f4-497f-9285-716769e64730 attributes: s3://rich-nxrm3-s3/content/vol-29/chap-37/75ca6995-e8f4-497f-9285-716769e64730.properties {deletedDateTime=1588705458575, deleted=true, @BlobStore.created-by=admin, creationTime=1588705409287, @BlobStore.created-by-ip=127.0.0.1, @BlobStore.content-type=application/xml, sha1=a68071a6a4d033994a32afbf0fdb0812926ecaba, @BlobStore.blob-name=org/foo/bar/project/maven-metadata.xml, deletedReason=Updating asset AttachedEntityId

Unknown macro: {asset->#59}

, @Bucket.repo-name=snapshots2, size=278}
2020-05-05 14:04:40,496-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.storage.MissingBlobException: Blob rich-nxrm3-s3-promoted-promoted@09D8B6F0-AB3A8DF9-E6138B67-E24F7A75-C6857C62:75ca6995-e8f4-497f-9285-716769e64730 exists in metadata, but is missing from the blobstore)
2020-05-05 14:04:40,497-0500 WARN [qtp1884423982-162] *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/snapshots2/org/foo/bar/project/maven-metadata.xml
org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:206)
at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy226.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:82)
at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)

After the initial 500 response(s) every subsequent request for the GA maven-metadata does succeed, but every single one triggers a rebuild of the maven-metadata.xml:

2020-05-05 14:06:49,278-0500 INFO [qtp1884423982-53] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

Unknown macro: {path='org/foo/bar/project/maven-metadata.xml', fileName='maven-metadata.xml', hashType=null}

hasn't changed, skipping
2020-05-05 14:07:02,868-0500 INFO [qtp1884423982-183] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

hasn't changed, skipping
2020-05-05 14:08:13,914-0500 INFO [qtp1884423982-296] *UNKNOWN org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataUpdater - metadata for MavenPath

Unknown macro: {path='org/foo/bar/project/maven-metadata.xml', fileName='maven-metadata.xml', hashType=null}

hasn't changed, skipping

Rebuilding the GA metadata is a slow operation on s3, and so after the initial failure it causes builds to run very slowly.