Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-24842

Slow performance pushing large images to S3 based blobstore

    XMLWordPrintable

    Details

      Description

      Customer originally reported this using docker. They were also able to reproduce slow performance using raw repository.

      The logs show examples of failed uploads due to "Early EOF":

      2020-07-29 01:50:09,797+0000 WARN [qtp1982132812-85841] US002SH org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT /repository/PBRawDev/com/pb/miraster/Runtime-MiPro/Trunk\6.0/2020-07-29/Runtime-MiPro.zip

      org.sonatype.nexus.blobstore.api.BlobStoreException: Error executing parallel requests for bucket:nexus-dev-blob key:content/tmp/tmp$e7e066a5-0543-4f1e-b229-9f57fcefbb23.bytes with uploadId:8serHhUMlcrmMGyvhMgkjebIJ2xxW9TVE_xon9VqOmnGjT0_ymtFht9XuJTqFNAzAFI6wp2M.rH3MO3L24_1uVCUbnlVa0BQ0UZAbpR1d4aPjFfvRTmwyny31vGH6TGz, Cause: org.eclipse.jetty.io.EofException: Early EOF​​

      Which means that an upload was abruptly ended midstream/upload and indicates something outside of Nexus i.e. ELB is closing the connection.

      Looking at the thread dump in the support.zip, there are a number of threads waiting on S3 blob uploads e.g.

      "qtp1982132812-87101" id=87101 state=WAITING

      • waiting on <0x2ce7efc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      • locked <0x2ce7efc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

      at sun.misc.Unsafe.park(Native Method)

      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)

      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

      at java.util.concurrent.ExecutorCompletionService.take(ExecutorCompletionService.java:193)

      at org.sonatype.nexus.blobstore.s3.internal.ParallelRequester.parallelRequests(ParallelRequester.java:98)

      at org.sonatype.nexus.blobstore.s3.internal.ParallelUploader.upload(ParallelUploader.java:80)

      at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.lambda$1(S3BlobStore.java:233)

      at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$$Lambda$852/2129009552.ingestTo(Unknown Source)

      at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.create(S3BlobStore.java:268)

      at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.doCreate(S3BlobStore.java:230)

      at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:127)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mbucher Michael Bucher
              Reporter:
              cseney Cassandra Seney
              Last Updated By:
              Peter Lynch
              Team:
              NXRM - Operations/Groot
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title