Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-26453

saving the s3 blobstore config can break async tmp blob deletes by not restarting the async delete thread pool

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.29.2
    • Fix Version/s: 3.30.0
    • Component/s: S3

      Description

      By default, S3Blobstore uses asynchronous operations to delete temp blobs. There is a thread pool created when a blobstore is first created, in the constructor.

      However, if the blobstore configuration is saved, then S3Blobstore.doStop() method is called, stopping the executor service. Both methods responsible for starting the blobstore again doInit() and doStart(), do not restart the executor thread pool. This means that on any further operations against any repository using this S3 blobstore which delete tmp blobs, will fail with these types of messages:

      2021-01-19 14:32:32,270-0400 ERROR [qtp483989768-236] admin org.sonatype.nexus.coreui.UploadResource - Unable to perform upload to repository test-s3-deletes
      java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@22a59b4d rejected from java.util.concurrent.ThreadPoolExecutor@147abede[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
      

      The end result when this bug happens is any writes to the affected S3 blobstore will fail, since writes use temp blobs that need to be deleted. This usually is first noticed by an end user as a failed upload of a component.

      Workaround

      Restarting the NXRM instance affected will restart the S3 blobstore properly.

      One can also set a property inside of $data-dir/etc/nexus.proeprties to disable asynchronous deletes which rely on this thread pool. Restart of NXRM is required to pick up this property. If using an HA-C cluster, all nodes will need to have the property set and gracefully restarted.

      nexus.s3.preferAsyncCleanup=false
      

      Expected

      Make sure the stopped executor service is gracefully restarted upon saving blobstore config.

        Attachments

          Activity

            People

            Assignee:
            mallen Mick Allen
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch Peter Lynch
            Team:
            NXRM - Groot
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title