Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-22627

Disk space never freed after deleting docker manifests

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.20.1, 3.28.0
    • Fix Version/s: None
    • Component/s: Docker
    • Labels:
    • Environment:
      Docker container from image sonatype/nexus3:3.20.1
    • Notability:
      3

      Description

      We're working on cleaning up docker images from CI that are no longer needed but running into an issue.  We use the docker REST api to query tags and manifests, then delete manifests.  This appears to work as expected, and can be seen with tags and manifests going away from the web UI too.

      We initially missed needing to run the Docker GC task and blob store compaction, but after setting up and running those,  we see no change in free space.  Meanwhile we've deleted somewhere around 50+GB worth of images.  Each of which contained most of this space in a single unique layer that is not linked by any other manifests.

      I've checked logs and no warnings or errors are thrown during both the Docker GC, or the blob store compaction.

      2020-01-27 01:00:00,011+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Docker - Delete Unused - accusoft internal' [repository.docker.gc] state change WAITING -> RUNNING
      2020-01-27 01:00:00,052+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask - Task log: /nexus-data/log/tasks/repository.docker.gc-20200127010000018.log
      2020-01-27 01:00:00,058+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=hosted, format=docker, name='docker-accusoft'}
      2020-01-27 01:00:05,756+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection completed on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=hosted, format=docker, name='docker-accusoft'}
      2020-01-27 01:00:05,758+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Docker - Delete Unused - accusoft internal' [repository.docker.gc] state change RUNNING -> WAITING (OK)
      

      The other log file mentioned in there is pretty sparse, but also shows no warnings/errors.

      2020-01-27 01:00:00,039+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask - Task information:
      2020-01-27 01:00:00,051+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask -  ID: d794b03a-4d7c-45c0-86d3-10a119217575
      2020-01-27 01:00:00,051+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask -  Type: repository.docker.gc
      2020-01-27 01:00:00,051+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask -  Name: Docker - Delete Unused - accusoft internal
      2020-01-27 01:00:00,051+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask -  Description: Docker - Delete unused manifests and images
      2020-01-27 01:00:00,058+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=hosted, format=docker, name='docker-accusoft'}
      2020-01-27 01:00:05,756+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection completed on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=hosted, format=docker, name='docker-accusoft'}
      2020-01-27 01:00:05,757+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask - Task complete
      

      and later blob compaction

      2020-01-27 01:30:00,034+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Compact Blob Store - default' [blobstore.compact] state change WAITING -> RUNNING
      2020-01-27 01:30:00,050+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task log: /nexus-data/log/tasks/blobstore.compact-20200127013000039.log
      2020-01-27 01:30:00,060+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Deletions index file rebuild not required
      2020-01-27 01:30:00,061+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Begin deleted blobs processing
      2020-01-27 01:30:00,063+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Compact Blob Store - default' [blobstore.compact] state change RUNNING -> WAITING (OK)
      

      With a similar clean seperate log file:

      2020-01-27 01:30:00,040+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task information:
      2020-01-27 01:30:00,047+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  ID: a1c24cf8-02fa-4616-80c3-b7a4f2d4d45d
      2020-01-27 01:30:00,047+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Type: blobstore.compact
      2020-01-27 01:30:00,049+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Name: Compact Blob Store - default
      2020-01-27 01:30:00,049+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Description: Compacting default blob store
      2020-01-27 01:30:00,060+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Deletions index file rebuild not required
      2020-01-27 01:30:00,061+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Begin deleted blobs processing
      2020-01-27 01:30:00,062+0000 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task complete
      

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jshumaker Jody Shumaker
              Last Updated By:
              Rich Seddon Rich Seddon
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title