Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-25378

docker PUT and PATCH requests may write to the same blob causing docker pushes to fail due to InvalidContentException: Content type could not be determined

    XMLWordPrintable

    Details

      Description

      docker push command uses HTTP PATCH and HTTP PUT requests as per the Docker API.

      When the PATCH and PUT request are processed within the same second, it is possible that NXRM will attempt to write to the same file in the blobstore associated with the docker repository. These two writes to the same file handled by different threads can conflict, and during the content validation phase to verify the content type of the written file, the PUT request can fail.

      The PUT requests returning a 400 response will fail docker pushes. Retrying the docker push may help the push to succeed. When the push fails, the docker client can see a generic message like:

      blob upload invalid: blob upload invalid

      If TRACE log level is enabled for logger org.sonatype.nexus.repository.docker.internal.DockerContentValidator , then messages like this will be present in the nexus.log:

      2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580]  admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Determining content type for: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 with declaredContentType: application/gzip
2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580]  admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Input is not detected as an accepted image layer type (gzip or tar): v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3
org.sonatype.nexus.repository.InvalidContentException: Content type could not be determined: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 
 at org.sonatype.nexus.repository.mime.DefaultContentValidator.determineContentType(DefaultContentValidator.java:78)
2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580]  admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Invalid JSON file: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3. Content will be written to disk for manual inspect at: ..\sonatype-work\nexus3\tmp\docker-content-validation-failures3193571824385530681
com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
 at [Source: (org.sonatype.nexus.blobstore.PerformanceLoggingInputStream); line: 1, column: 0]
2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580]  admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Input is less than 1024 bytes, not an empty diff: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3
java.io.EOFException: reached end of stream after reading 0 bytes; 1024 bytes expected
2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580]  admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Invalid JSON file: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3. Content will be written to disk for manual inspect at: ..\sonatype-work\nexus3\tmp\docker-content-validation-failures678576674342260287
com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input
2020-09-11 11:28:42,149+0300 WARN  [qtp927920754-3580]  admin org.sonatype.nexus.repository.storage.StorageTxImpl - An exception occurred determining the content type of asset v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 in repository docker-private

      The request log will indicate the PUT request failed with a 400 status code during the same second as the related PATCH request succeeds with 202 response:

      127.0.0.1 - admin [11/Eyl/2020:11:28:42 +0300] "PATCH /repository/docker-private/v2/atroubleshoot/nugetexample/blobs/uploads/9a6fe412-43d6-4430-81fc-26206ef1769e HTTP/1.1" 202 - 0 0 "docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-957.21.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.6 (linux))" [qtp927920754-3540] 
127.0.0.1 - admin [11/Eyl/2020:11:28:42 +0300] "PUT /repository/docker-private/v2/atroubleshoot/nugetexample/blobs/uploads/9a6fe412-43d6-4430-81fc-26206ef1769e?digest=sha256%3A18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 HTTP/1.1" 400 0 192 31 "docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-957.21.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.6 (linux))" [qtp927920754-3580]

      Expected

      Prevent writing to the same blob file by two different threads so that docker push commands don't randomly fail.

        Attachments

          Issue Links

          fixes

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-20871 zero-byte layers uploaded using docker push fail strict content type validation

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Joseph Stephens
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  tigCommentSecurity.panel-title

                    PagerDuty