Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-34681

Blobstore Group Round Robin should try next blobstore member if one member if full

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.39.0
    • Fix Version/s: 3.43.0
    • Component/s: Blobstore
    • Labels:
    • Story Points:
      3
    • Sprint:
      NXRM Neo Sprint 44
    • Notability:
      3

      Description

      Write to a blobstore which is member of blobstore group with round robin write policy fails with 500 error response when that group member has no free storage space.

      request.log

      10.10.10.10 - UserA [15/Aug/2022:11:03:28 +0100] "PUT /nexus/repository/docker-internal-devn3p/v2/com/hase/gbb/cmb-dbbhase-banking-ea-account-settings/blobs/uploads/745221c6-2764-4dc5-916d-f6dc26c5041c?digest=sha256%3A8be10e0ca1172d3f527edd71c1809b0ddf362d20b59d2053bfd4e0266350ddc2 HTTP/1.1" 500 0 65 2461 "docker/20.10.9 go/go1.16.8 git-commit/79ea9d3 kernel/3.10.0-1160.59.1.el7.x8664 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.9 (linux))" [qtp436308547-2090566]
      

      nexus.log

      2022-08-15 11:03:28,050+0100 WARN [qtp436308547-2090566] gbl19678 UserA org.sonatype.nexus.blobstore.file.internal.SimpleFileOperations - Using copy to move /Nexus/blobs/n3-prd-docker-store/content/tmp/path$nexus-repository-docker-745221c6-2764-4dc5-916d-f6dc26c5041c-c976e9c3-9236-4c50-9b98-f45bdd20955b.1bc5765e-b1ec-4098-8ed9-c01bd9ce431b.bytes to /Nexus/blobs/n3-prd-docker-store/content/directpath/nexus-repository-docker/745221c6-2764-4dc5-916d-f6dc26c5041c/c976e9c3-9236-4c50-9b98-f45bdd20955b.bytes
      
      2022-08-15 11:03:28,059+0100 WARN  [qtp436308547-2090566] gbl19678 UserA org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/com/hase/gbb/cmb-dbbhase-banking-ea-account-settings/blobs/uploads/745221c6-2764-4dc5-916d-f6dc26c5041c
      org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: path$nexus-repository-docker/745221c6-2764-4dc5-916d-f6dc26c5041c/c976e9c3-9236-4c50-9b98-f45bdd20955b, java.nio.file.FileSystemException: /Nexus/blobs/n3-prd-docker-store/content/directpath/nexus-repository-docker/745221c6-2764-4dc5-916d-f6dc26c5041c: No space left on device, Cause: /Nexus/blobs/n3-prd-docker-store/content/directpath/nexus-repository-docker/745221c6-2764-4dc5-916d-f6dc26c5041c: No space left on device
       at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:423)
       at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:357)
       at org.sonatype.nexus.blobstore.file.FileBlobStore.doCreate(FileBlobStore.java:337)
       at org.sonatype.nexus.blobstore.metrics.BlobStoreAnalyticsInterceptor.invoke(BlobStoreAnalyticsInterceptor.java:60)
       at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:125)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
       at org.sonatype.nexus.blobstore.group.BlobStoreGroup.lambda$0(BlobStoreGroup.java:185)
       at org.sonatype.nexus.blobstore.group.BlobStoreGroup.create(BlobStoreGroup.java:200)
       at org.sonatype.nexus.blobstore.group.BlobStoreGroup.create(BlobStoreGroup.java:185)
       at org.sonatype.nexus.blobstore.metrics.BlobStoreAnalyticsInterceptor.invoke(BlobStoreAnalyticsInterceptor.java:60)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
       at org.sonatype.nexus.blobstore.group.BlobStoreGroup.create(BlobStoreGroup.java:178)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
       at org.sonatype.nexus.repository.docker.internal.Upload.addBlob(Upload.java:125)
       at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.complete(UploadManagerImpl.java:113)
       at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerHostedFacetSupport.upload(DockerHostedFacetSupport.java:193)
       at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerHostedFacetSupport.completeBlobUpload(DockerHostedFacetSupport.java:112)
       at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$completeBlobUpload$6.call(Unknown Source)
       at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure5.doCall(V2Handlers.groovy:153)
      

       

      Expected Behaviour

      Blobstore Group Round Robin should try next blobstore member if current member blobstore has fails to write data , as documented in below link.

      https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/configuring-blob-stores#ConfiguringBlobStores-WhatisaFillPolicy?

       

      If writing to one blob store fails, then it attempts again with the next blob store in the group.

      Actual Behaviour

      Upload fails with 500 error, with no attempt to try other blobstore members in that group blobstore.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              tahir.awan Tahir Awan
              Reporter:
              msurani Mahendra Surani
              Last Updated By:
              Michael Prescott Michael Prescott
              Team:
              NXRM - Neo
              Owner:
              Tahir Awan Tahir Awan
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title