docker push command uses HTTP PATCH and HTTP PUT requests as per the Docker API.
When the PATCH and PUT request are processed within the same second, it is possible that NXRM will attempt to write to the same file in the blobstore associated with the docker repository. These two writes to the same file handled by different threads can conflict, and during the content validation phase to verify the content type of the written file, the PUT request can fail.
The PUT requests returning a 400 response will fail docker pushes. Retrying the docker push may help the push to succeed. When the push fails, the docker client can see a generic message like:
blob upload invalid: blob upload invalid
If TRACE log level is enabled for logger org.sonatype.nexus.repository.docker.internal.DockerContentValidator , then messages like this will be present in the nexus.log:
2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580] admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Determining content type for: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 with declaredContentType: application/gzip 2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580] admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Input is not detected as an accepted image layer type (gzip or tar): v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 org.sonatype.nexus.repository.InvalidContentException: Content type could not be determined: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 at org.sonatype.nexus.repository.mime.DefaultContentValidator.determineContentType(DefaultContentValidator.java:78) 2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580] admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Invalid JSON file: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3. Content will be written to disk for manual inspect at: ..\sonatype-work\nexus3\tmp\docker-content-validation-failures3193571824385530681 com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input at [Source: (org.sonatype.nexus.blobstore.PerformanceLoggingInputStream); line: 1, column: 0] 2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580] admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Input is less than 1024 bytes, not an empty diff: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 java.io.EOFException: reached end of stream after reading 0 bytes; 1024 bytes expected 2020-09-11 11:28:42,149+0300 TRACE [qtp927920754-3580] admin org.sonatype.nexus.repository.docker.internal.DockerContentValidator - Invalid JSON file: v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3. Content will be written to disk for manual inspect at: ..\sonatype-work\nexus3\tmp\docker-content-validation-failures678576674342260287 com.fasterxml.jackson.databind.exc.MismatchedInputException: No content to map due to end-of-input 2020-09-11 11:28:42,149+0300 WARN [qtp927920754-3580] admin org.sonatype.nexus.repository.storage.StorageTxImpl - An exception occurred determining the content type of asset v2/-/blobs/sha256:18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 in repository docker-private
The request log will indicate the PUT request failed with a 400 status code during the same second as the related PATCH request succeeds with 202 response:
127.0.0.1 - admin [11/Eyl/2020:11:28:42 +0300] "PATCH /repository/docker-private/v2/atroubleshoot/nugetexample/blobs/uploads/9a6fe412-43d6-4430-81fc-26206ef1769e HTTP/1.1" 202 - 0 0 "docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-957.21.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.6 (linux))" [qtp927920754-3540] 127.0.0.1 - admin [11/Eyl/2020:11:28:42 +0300] "PUT /repository/docker-private/v2/atroubleshoot/nugetexample/blobs/uploads/9a6fe412-43d6-4430-81fc-26206ef1769e?digest=sha256%3A18830b41d748d94369a10d934f92cb2b496fab31bc24ed89a8adda52c3b6c9a3 HTTP/1.1" 400 0 192 31 "docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-957.21.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.6 (linux))" [qtp927920754-3580]
Expected
Prevent writing to the same blob file by two different threads so that docker push commands don't randomly fail.
- fixes
-
NEXUS-20871 zero-byte layers uploaded using docker push fail strict content type validation
-
- Closed
-