Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-14198

outbound proxy repository requests for identical files which take more than 10 minutes duplicate work and disk space

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Sprint:
      Platform Sprint 103, Platform Sprint 104

      Description

      Configure a Proxy repository which points to a remote server hosting a large file that will take over 10 minutes to download.

      Send 2 or more identical requests into the proxy repository for that large file.

      One request will begin downloading the remote file. The other requests will be put into a queue waiting for it to complete.

      The first request starts writing the remote file into the blobstore as a temporary file. It may look like this:

      > ls -la blobs/default/content/tmp
      
      drwxrwxr-x 3 plynch staff 102 Aug 31 17:17 ./
      drwxrwxr-x 3 plynch staff 102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff 43K Aug 31 17:17 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 2.0M
      

      After 10 minutes, the initial request is not done. The second thread that was in the queue will get tired of waiting and begin downloading the same file. Now two threads are actively downloading the same file from the remote and writing the content to the blobstore to a temporary file.

      drwxrwxr-x 3 plynch staff  102 Aug 31 17:17 ./
      drwxrwxr-x 3 plynch staff  102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff 1.3G Aug 31 17:27 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 1.3G
      
      drwxrwxr-x 4 plynch staff  136 Aug 31 17:27 ./
      drwxrwxr-x 3 plynch staff  102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff  68K Aug 31 17:27 'tmp$16237c6b-30e8-49ab-9c1d-9d1c817d98ab.43361be6-cac6-459a-aa6b-231c94c790de.bytes'
      -rw-rw-r-- 1 plynch staff 1.3G Aug 31 17:27 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 1.3G
      

      This pattern continues for as long as there are identical inbound requests for the same file in a queue - every 10 minutes, a new outbound request for the same file begins. The disk holding the repository blobstore location becomes more full.

      In the case of

      • the remote file is large ( less common multi-GB )
      • the network is comparatively slow and/or bandwidth starved ( common )
      • requests for identical files at the same time come into Nexus concurrently ( common ie. chef )
      • the blobstore does not have considerable free space larger than (concurrent requests * file size)

      there becomes a high probability that the blobstore may fill with these temporary files, overflowing the disk. Running out of disk is a critical event.

      Expected

      • the common cases described should be better mitigated against to prevent high disk usage and otherwise useless outbound requests and negative side effects of this behaviour

      Workaround

      A Nexus administrator needs to guestimate the longest time the largest file should take to be downloaded from a proxy repository remote. Knowing this value, they can change the default 10 minute wait period to this new value.

      For example, if the longest remote request can take up to 8 hrs, then:

      Edit sonatype-work/nexus3/etc/nexus.properties

      Add a new line like this to the file:

      nexus.proxy.passiveCooperationTimeout=8h
      

      or

      nexus.proxy.passiveCooperationTimeout=480m
      

      or

      nexus.proxy.passiveCooperationTimeout=28800s
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              nblair Nicholas Blair
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title