Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19566

Multithread S3 blob store upload

    Details

      Description

      The current MultipartUploader is single threaded (https://github.com/sonatype/nexus-public/blob/master/plugins/nexus-blobstore-s3/src/main/java/org/sonatype/nexus/blobstore/s3/internal/MultipartUploader.java) and uploads chunks sequentially.

      For files larger than 5mb, this introduces a considerable slowdown in upload times. In a local test with a 1gb file, comparing against the S3 command line: uploading via S3 CLI took 21 seconds, NXRM took 680 seconds. Much of this time difference appears to be due the the sequential 5mb chunks.

      If the uploader utilizes multiple threads, it should be considerably faster.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ataylor Andrew Taylor
              Reporter:
              kendallt Kendall Trego
              Last Updated By:
              Michael Prescott Michael Prescott
              Team:
              NXRM - Morpheus
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title