Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-13363

Conditional GET requests for Docker image layers always download the layer when proxying another Nexus

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.1, 3.3.1
    • Fix Version/s: 3.5.0
    • Component/s: Docker
    • Story Points:
      3

      Description

      Tested this with 3.2.1, although the problem has also been reported against 3.3.1 as well.

      1. (Optional) Keep things simple and use --insecure-registry and plain http ports for the docker repos In this example I will use plain http ports.
      2. (Optional) use https ports and have your docker daemon trust the Nexus certificate.
      3. Configure Nexus A with a docker hosted repo. ( 192.168.2.97:14440 )
        {
              "@type": "d",
              "@rid": "#76:1",
              "@version": 2,
              "@class": "repository",
              "recipe_name": "docker-hosted",
              "repository_name": "docker-hosted",
              "online": true,
              "attributes": {
                "docker": {
                  "httpPort": 14440,
                  "v1Enabled": true
                },
                "storage": {
                  "strictContentTypeValidation": true,
                  "writePolicy": "ALLOW",
                  "blobStoreName": "default"
                }
              }
            },
        
      4. Configure Nexus B with a docker proxy repo which proxies Nexus A hosted repo. ( 192.168.2.97:13350 )
        {
              "@type": "d",
              "@rid": "#73:2",
              "@version": 4,
              "@class": "repository",
              "recipe_name": "docker-proxy",
              "repository_name": "docker-proxy",
              "online": true,
              "attributes": {
                "proxy": {
                  "contentMaxAge": 0,
                  "remoteUrl": "http://192.168.2.97:14440/",
                  "metadataMaxAge": 0
                },
                "negativeCache": {
                  "timeToLive": 0,
                  "enabled": false
                },
                "dockerProxy": {
                  "indexType": "REGISTRY"
                },
                "storage": {
                  "strictContentTypeValidation": true,
                  "blobStoreName": "default"
                },
                "httpclient": {
                  "blocked": false,
                  "autoBlock": false
                },
                "docker": {
                  "httpPort": 13350,
                  "v1Enabled": true
                }
              }
            },
        
      5. Configure the following loggers in Nexus B so that you can monitor the outbound HTTP headers and requests:
        1. org.apache.http at DEBUG
      6. Docker Login to both proxy and hosted repos so that docker v2 API works and Nexus does not fall back automatically to using /v1 API ( which would otherwise void this test).
        1. docker login 192.168.2.97:14440
        2. docker login 192.168.2.97:13350
      7. Find and remove all images with the same name to remove all the layers attached to those images - in this example I will re-use the sonatype/nexus 3 image:
        1. docker images ( list ALL images based on sonatype/nexus3 )
        2. docker rmi <fill sonatype/nexus3 image name as reported under REPOSITORY column of previous command> - there may be more than one image, delete them all to make sure subsequent pulls actually download layers not already cached
      8. tag and push an image to Nexus A docker hosted.
        1. docker pull sonatype/nexus3 ( download from official docker registry )
        2. docker tag sonatype/nexus3 192.168.2.97:14440/sonatype/nexus3 ( tag the nexus 3 image against the hosted repo )
        3. docker push 192.168.2.97:14440/sonatype/nexus3 ( push the image and layers to hosted repo )
      9. remove the local image you just tagged from the docker client cache. This should also remove dependent layers forcing them to re-download on the next pull.
        1. docker rmi 192.168.2.97:14440/sonatype/nexus3
      10. request the image through the proxy repository which will in turn download from the hosted repo
        1. docker pull 192.168.2.97:13350/sonatype/nexus3
      11. Examine the nexus.log of Nexus B. There you will find outbound requests to the hosted repo, downloading the layers as expected.
      12. Delete the cached image and layers from the proxy now and then pull again
        1. docker rmi 192.168.2.97:13350/sonatype/nexus3
        2. docker pull 192.168.2.97:13350/sonatype/nexus3
      13. Examine the nexus.log of Nexus B. There you will find identical outbound requests to the hosted repo again. The manifest request will return 304 indicating it is not changed. However all the conditional GET requests for the layers will return 200 and the full content of the image.

      Problem

      The images are always re-downloaded from the remote Nexus, even if the hash/content was unchanged. The conditional GET does not work as expected - 304 status code and no content was expected.

      For comparison, proxying the official docker registry at https://registry-1.docker.io from Nexus 3.2.1 does work correctly in that the official registry returns 304 for unchanged layers.

      Workaround

      Set Component Max Age to -1 as image layers never change hash identity anyways. This prevents already cached image layers from being requested from the remote.

        Attachments

          Activity

            People

            Assignee:
            jstephens Joseph Stephens
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch
            Team:
            Nexus - Formats
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title