Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-25941

S3 blobstore bucket object creation algorithm is not optimal because it relies on rename

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.28.1
    • Fix Version/s: None
    • Component/s: S3

      Description

      The current S3 blobstore implemention creates new tmp assets(objects) in an S3 bucket first, then renames these assets(objects) to their final destination names after being successfully uploaded.

      Problem

      S3 operations use HTTP and new uploads to S3 buckets are done using chunks of a fixed size. Depending on the file size being created, many HTTP requests are made to create a single object in the bucket. For example uploading a 127MB file at default chunk size of 5MB, 25 individual 5MB chunks were uploaded using 25 HTTP PUT requests, taking a total of 14 seconds to complete.

      Next NXRM tries to rename that tmp file to its final location in the blobstore - since there is no pure "rename" operation for S3 blobstores, NXRM code is forced to ask S3 SDK to copy the existing S3 object to a new file, then delete the original. This takes an additional 14 seconds.

      Expected

      The total time to upload a new file into an S3 bucket can be reduced roughly by half, if tmp files are not used at all in S3 blobstores. Instead, upload the file to its final location - when it is confirmed the upload is whole and complete ( checksummed using S3 API) , then mark the blob/asset available to be consumed by NXRM for other operations.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch Peter Lynch
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title