Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-23977

Large docker image upload does not work well with slow inbound network

    XMLWordPrintable

    Details

    • Notability:
      2

      Description

      SYMPTOM / ISSUE:

      If your Nexus environment satisfies the following conditions, docker push (or probably trying to upload multiple large files concurrently as well) may keep retrying until you cancel.

      • Using S3 for blobstore
      • Network between your docker client and Nexus is relatively slow (eg: 1.5Mbps)
      • Your docker image has many layers and each layer is much larger than 5MB
      • There is a proxy or load balancer between Nexus and clients, and having small idle timeout (eg: 1 min)

      REPRODUCE STEPS

      1) Setup Nexus in slow network. I used VMWare and set like below:

      2) Setup HAProxy with below timeouts, and also reverse proxying to a docker hosted repo

        timeout client 60000ms
        timeout server 60000ms
      

      3) Prepare a test image and push

      _DOCKER_HOSTED=dh1.standalone.localdomain:18082
      _DEFAULT_USER=admin
      _DEFAULT_PWD=admin123
      _tag_name="rancher/coreos-etcd:v3.4.3-rancher1"
      docker pull rancher/coreos-etcd:v3.4.3-rancher1
      docker login ${_DOCKER_HOSTED} --username ${_DEFAULT_USER} --password ${_DEFAULT_PWD}
      docker tag ${_tag_name} ${_DOCKER_HOSTED}/${_tag_name}
      time docker push ${_DOCKER_HOSTED}/${_tag_name}
      

      EXPECTED BEHAVIOUR

      Even slow inbound network, docker push should not fail.

      ACTUAL BEHAVIOUR

      The docker command kept showing "Retrying in X seconds" and never finish.
      Screen Recording 2020-05-20 at 5.27.33 pm.mov
      nexus.log.zip

        Attachments

        1. error15minutes.png
          error15minutes.png
          318 kB
        2. image-2020-05-22-01-47-54-549.png
          image-2020-05-22-01-47-54-549.png
          264 kB
        3. nexus.log.zip
          205 kB
        4. Screen Recording 2020-05-20 at 5.27.33 pm.mov
          8.55 MB
        5. Screen Shot 2020-05-20 at 5.32.30 pm.png
          Screen Shot 2020-05-20 at 5.32.30 pm.png
          33 kB
        6. Screen Shot 2020-05-26 at 11.53.32 AM.png
          Screen Shot 2020-05-26 at 11.53.32 AM.png
          58 kB
        7. success.png
          success.png
          346 kB
        8. threaddump.png
          threaddump.png
          217 kB

          Issue Links

            Activity

              People

              Assignee:
              mbucher Michael Bucher
              Reporter:
              hosako Hajime Osako
              CC:
              Matt Kubilus, Ryan Drogo
              Last Updated By:
              Michael Prescott Michael Prescott
              Team:
              NXRM - Groot
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title