Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-23756

concurrent S3 blob writes of the same asset may use the same temporary key and fail

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.22.0
    • Fix Version/s: None
    • Component/s: S3
    • Notability:
      3

      Description

      Concurrent writes into the same S3 blobstore may attempt to store files with the same temporary key, which can lead to 500 response.

      These GET requests both end with 500 response:

      123.123.123.123 - - [23/Apr/2020:13:31:07 +0000] "GET /nexus/repository/public/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar HTTP/1.0" 500 - 2200 137 "Apache-Maven/3.6.0 (Java 1.8.0_212; Linux 4.15.0-54-generic)" [qtp504495395-526]
      123.123.123.123 - - [23/Apr/2020:13:31:07 +0000] "GET /nexus/repository/public/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar HTTP/1.0" 500 - 2200 143 "Apache-Maven/3.6.0 (Java 1.8.0_212; Linux 4.15.0-54-generic)" [qtp504495395-1346]
      

      The cause appears to be Connection reset because both were writing into the same key:

      2020-04-23 13:31:07,874+0000 WARN  [qtp504495395-526]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /nexus/repository/public/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar
      org.sonatype.nexus.blobstore.api.BlobStoreException: Error uploading blob to bucket:sea-nexus-3 key:blob-store/content/tmp/tmp$8ddff7dc-4dfe-4a27-9787-eacd0faeb51a.bytes, Cause: Connection reset
       at org.sonatype.nexus.blobstore.s3.internal.ParallelUploader.upload(ParallelUploader.java:86)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.lambda$1(S3BlobStore.java:233)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.create(S3BlobStore.java:268)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.doCreate(S3BlobStore.java:230)
       at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:127)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:111)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:240)
       at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:256)
       at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:298)
       
      2020-04-23 13:31:07,892+0000 WARN  [qtp504495395-1346]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /nexus/repository/public/com/fasterxml/jackson/core/jackson-databind/2.9.10.1/jackson-databind-2.9.10.1.jar
      org.sonatype.nexus.blobstore.api.BlobStoreException: Error uploading blob to bucket:sea-nexus-3 key:blob-store/content/tmp/tmp$8ddff7dc-4dfe-4a27-9787-eacd0faeb51a.bytes, Cause: Connection reset
       at org.sonatype.nexus.blobstore.s3.internal.ParallelUploader.upload(ParallelUploader.java:86)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.lambda$1(S3BlobStore.java:233)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.create(S3BlobStore.java:268)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.doCreate(S3BlobStore.java:230)
       at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:127)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:111)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:240)
       at org.sonatype.nexus.repository.storage.StorageFacetImpl.createTempBlob(StorageFacetImpl.java:256)
       at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.put(MavenFacetImpl.java:298)
       at org.sonatype.nexus.repository.maven.internal.proxy.MavenProxyFacet.store(MavenProxyFacet.java:89)
       at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:271)
      

      Expected

      S3 writes are expensive. Do not re-use keys.

      However since if both writes actually worked, they may have ultimately failed, or perhaps only one will "win" while the other is soft-deleted.

      Probably best to not write concurrently into the blobstore for two assets that represent the same entity.

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Michael Bucher Michael Bucher
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title