Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-26683

"Remove a member from a blob store group" task can't remove a blob store if one bytes file is missing

    XMLWordPrintable

    Details

    • Sprint:
      NXRM Groot Sprint 3
    • Notability:
      3

      Description

      SYMPTOM:

      Running "Remove a member from a blob store group" task failed with below error:

      2021-02-10 02:04:31,459+0000 ERROR [quartz-17-thread-12]  *SYSTEM com.sonatype.nexus.blobstore.group.internal.tasks.BlobStoreGroupMemberRemovalTask - Member removal task did not complete successfully
      java.util.concurrent.ExecutionException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: CD69E0E78387DACD; S3 Extended Request ID: 3JDbsXeD0lemZZq/xUiXJbWXWaeSjjrsSW9DZPintwN/8/DrPpZ4W2YWm1Xu1W5Y9KI2/UFP1YE=; Proxy: null), S3 Extended Request ID: 3JDbsXeD0lemZZq/xUiXJbWXWaeSjjrsSW9DZPintwN/8/DrPpZ4W2YWm1Xu1W5Y9KI2/UFP1YE=
          ... (snip) ...
      Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: CD69E0E78387DACD; S3 Extended Request ID: 3JDbsXeD0lemZZq/xUiXJbWXWaeSjjrsSW9DZPintwN/8/DrPpZ4W2YWm1Xu1W5Y9KI2/UFP1YE=; Proxy: null)
          ... (snip) ...
              at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:647)
              at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
              at java.util.Optional.map(Optional.java:215)
              at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.inputStreamOfBlob(BlobStoreManagerImpl.java:465)
              at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.moveBlob(BlobStoreManagerImpl.java:426)
              at com.sonatype.nexus.blobstore.group.internal.tasks.BlobStoreGroupMemberRemovalTask.lambda$2(BlobStoreGroupMemberRemovalTask.java:130)
              at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
              at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
              at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
              at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
              at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
              at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
              at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
              at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
      

      REPRODUCE STEPS:

      Similar error is reproducible with the File type blobstore, so using File type blobstore in the following steps.

      1. Create a File blobstore "test" and assign to a raw-hosted repository
      2. Upload a few files
      3. Promote this File blobstore to group
      4. Create and add another blobstore into this group blobstore
      5. Login to Nexus and go to the actual blobstore location for "test-promoted", and delete any .bytes file
      6. Run the task to remove the "test-promoted" member blobstore
        2021-02-11 02:01:37,395+0000 ERROR [quartz-14-thread-18] node-nxrm-ha1.standalone.localdomain *SYSTEM com.sonatype.nexus.blobstore.group.internal.tasks.BlobStoreGroupMemberRemovalTask - Member removal task did not complete successfully
        java.util.concurrent.ExecutionException: org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: d7a7de8a-5a9d-499c-8e9a-189430fd336a, Blob has been deleted
            ... (snip) ...
        Caused by: org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: d7a7de8a-5a9d-499c-8e9a-189430fd336a, Blob has been deleted
                at org.sonatype.nexus.blobstore.file.FileBlobStore.checkExists(FileBlobStore.java:630)
                at org.sonatype.nexus.blobstore.file.FileBlobStore.access$1(FileBlobStore.java:625)
                at org.sonatype.nexus.blobstore.file.FileBlobStore$FileBlob.doGetInputStream(FileBlobStore.java:917)
                at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
                at java.util.Optional.map(Optional.java:215)
                at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.inputStreamOfBlob(BlobStoreManagerImpl.java:465)
                at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.moveBlob(BlobStoreManagerImpl.java:426)
        ...
        

      EXPECTED BEHAVIOUR:

      As Nexus already knew the file does not exist, it shouldn't fail with this particular error.
      It would be nice if "org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.moveBlob" has better error handling.

      Or, actually for this customer, the location was <prefix>/content/directpath/nexus-repository-docker/10a93c18-b3a0-4ae2-918f-a290d786a54d/0.bytes, so similar logic to NEXUS-21197 may work too (ignore all files under "nexus-repository-docker").

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              hosako Hajime Osako
              Last Updated By:
              Michael Oliverio Michael Oliverio
              Team:
              NXRM - Groot
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title