Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-14198

outbound proxy repository requests for identical files which take more than 10 minutes duplicate work and disk space


    • 1
    • Platform Sprint 103, Platform Sprint 104


      Configure a Proxy repository which points to a remote server hosting a large file that will take over 10 minutes to download.

      Send 2 or more identical requests into the proxy repository for that large file.

      One request will begin downloading the remote file. The other requests will be put into a queue waiting for it to complete.

      The first request starts writing the remote file into the blobstore as a temporary file. It may look like this:

      > ls -la blobs/default/content/tmp
      drwxrwxr-x 3 plynch staff 102 Aug 31 17:17 ./
      drwxrwxr-x 3 plynch staff 102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff 43K Aug 31 17:17 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 2.0M

      After 10 minutes, the initial request is not done. The second thread that was in the queue will get tired of waiting and begin downloading the same file. Now two threads are actively downloading the same file from the remote and writing the content to the blobstore to a temporary file.

      drwxrwxr-x 3 plynch staff  102 Aug 31 17:17 ./
      drwxrwxr-x 3 plynch staff  102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff 1.3G Aug 31 17:27 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 1.3G
      drwxrwxr-x 4 plynch staff  136 Aug 31 17:27 ./
      drwxrwxr-x 3 plynch staff  102 Aug 31 17:05 ../
      -rw-rw-r-- 1 plynch staff  68K Aug 31 17:27 'tmp$16237c6b-30e8-49ab-9c1d-9d1c817d98ab.43361be6-cac6-459a-aa6b-231c94c790de.bytes'
      -rw-rw-r-- 1 plynch staff 1.3G Aug 31 17:27 'tmp$f1d08191-df11-47c2-8c6e-845f7599915d.94cabaaf-f9d7-4121-a040-11f8a4aaf769.bytes'
      total 1.3G

      This pattern continues for as long as there are identical inbound requests for the same file in a queue - every 10 minutes, a new outbound request for the same file begins. The disk holding the repository blobstore location becomes more full.

      In the case of

      • the remote file is large ( less common multi-GB )
      • the network is comparatively slow and/or bandwidth starved ( common )
      • requests for identical files at the same time come into Nexus concurrently ( common ie. chef )
      • the blobstore does not have considerable free space larger than (concurrent requests * file size)

      there becomes a high probability that the blobstore may fill with these temporary files, overflowing the disk. Running out of disk is a critical event.


      • the common cases described should be better mitigated against to prevent high disk usage and otherwise useless outbound requests and negative side effects of this behaviour


      A Nexus administrator needs to guestimate the longest time the largest file should take to be downloaded from a proxy repository remote. Knowing this value, they can change the default 10 minute wait period to this new value.

      For example, if the longest remote request can take up to 8 hrs, then:

      Edit sonatype-work/nexus3/etc/nexus.properties

      Add a new line like this to the file:







        Issue Links



              nblair Nicholas Blair
              plynch Peter Lynch
              Peter Lynch Peter Lynch
              0 Vote for this issue
              6 Start watching this issue