Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28717

"Docker - Delete unused manifests and images" task attempts to read files from the wrong Blob Store.

    XMLWordPrintable

    Details

      Description

      Running the "Docker - Delete unused manifests and images" results in multiple WARN level messages similar to this sample:

      2021-08-17 11:00:58,104+0200 WARN  [quartz-12-thread-2]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 7d57c0aa-5e0e-4216-98a0-1f1649742a9d (/nexusblob/default/content/vol-08/chap-10/7d57c0aa-5e0e-4216-98a0-1f1649742a9d.properties)
      2021-08-17 11:00:58,105+0200 WARN  [quartz-12-thread-2]  *SYSTEM org.sonatype.nexus.repository.docker.internal.orient.DockerGCFacetImpl - Unable to read V2 Manifest for asset Asset{metadata=AttachedEntityMetadata{schema=asset, document=asset#20:24471{bucket:#16:14,attributes:[5],name:v2/library/debian/manifests/8,format:docker,last_updated:Wed May 26 15:35:16 CEST 2021,component:#17:5210,size:982,content_type:application/vnd.docker.distribution.manifest.list.v2+json,created_by:xxx,created_by_ip:xxx,blob_ref:NO_BACKUP@05FFD4F6-04D9EB88-3DE8462B-C1F3D4F3-B769A027:7d57c0aa-5e0e-4216-98a0-1f1649742a9d,last_downloaded:Wed May 26 15:35:16 CEST 2021,blob_created:Tue Nov 24 12:05:55 CET 2020,blob_updated:Wed May 26 15:35:16 CEST 2021} v7}, name=v2/library/debian/manifests/8}, Blob NO_BACKUP@05FFD4F6-04D9EB88-3DE8462B-C1F3D4F3-B769A027:7d57c0aa-5e0e-4216-98a0-1f1649742a9d exists in metadata, but is missing from the blobstore
      
      2021-08-17 11:00:58,116+0200 WARN  [quartz-12-thread-2]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 07558376-9c67-4732-9b04-8f51767aff6b (/nexusblob/default/content/vol-35/chap-44/07558376-9c67-4732-9b04-8f51767aff6b.properties)
      2021-08-17 11:00:58,116+0200 WARN  [quartz-12-thread-2]  *SYSTEM org.sonatype.nexus.repository.docker.internal.orient.DockerGCFacetImpl - Unable to read V2 Manifest for asset Asset{metadata=AttachedEntityMetadata{schema=asset, document=asset#20:24495{bucket:#15:42,attributes:[5],name:v2/repository/ttt/asd/manifests/ttt,format:docker,last_updated:Fri Jun 04 13:10:07 CEST 2021,component:#17:5212,size:527,content_type:application/vnd.docker.distribution.manifest.v2+json,created_by:xxx,created_by_ip:xxx,blob_ref:docker@05FFD4F6-04D9EB88-3DE8462B-C1F3D4F3-B769A027:07558376-9c67-4732-9b04-8f51767aff6b,last_downloaded:null,blob_created:Fri Jun 04 13:07:45 CEST 2021,blob_updated:Fri Jun 04 13:07:45 CEST 2021} v2}, name=v2/repository/ttt/asd/manifests/ttt}, Blob docker@05FFD4F6-04D9EB88-3DE8462B-C1F3D4F3-B769A027:07558376-9c67-4732-9b04-8f51767aff6b exists in metadata, but is missing from the blobstore
      

      We use multiple "File" blob stores, and it looks like jobs are trying to read some files from different blob stores than the files actually exists in.
      In my sample, the are attempts to read files with "blob_ref:NO_BACKUP" and "blob_ref:docker" from "default" blob store. Both of those files physically exists on disk, but in different locations:

      # ls -l /nexusblob/default/content/vol-08/chap-10/7d57c0aa-5e0e-4216-98a0-1f1649742a9d.properties
      ls: cannot access /nexusblob/default/content/vol-08/chap-10/7d57c0aa-5e0e-4216-98a0-1f1649742a9d.properties: No such file or directory
      # ls -l /nexusblob/blob_no_backup/content/vol-08/chap-10/7d57c0aa-5e0e-4216-98a0-1f1649742a9d.properties
      -rw-r--r--. 1 nexus nexus 378 May 26 15:35 /nexusblob/blob_no_backup/content/vol-08/chap-10/7d57c0aa-5e0e-4216-98a0-1f1649742a9d.properties
      
      # ls -l /nexusblob/default/content/vol-35/chap-44/07558376-9c67-4732-9b04-8f51767aff6b.properties
      ls: cannot access /nexusblob/default/content/vol-35/chap-44/07558376-9c67-4732-9b04-8f51767aff6b.properties: No such file or directory
      # ls -l /nexusblob/blob_docker/content/vol-35/chap-44/07558376-9c67-4732-9b04-8f51767aff6b.properties
      -rw-r--r--. 1 nexus nexus 367 Jun  4 13:07 /nexusblob/blob_docker/content/vol-35/chap-44/07558376-9c67-4732-9b04-8f51767aff6b.properties

      I do not know if this have any effect besides those WARN messages, but given the recent problems with the ""Docker - Delete unused manifests and images"" task, it looks worrying.

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mbucher Michael Bucher
              Reporter:
              piotr.filip@comarch.com Piotr Filip
              Last Updated By:
              Hajime Osako Hajime Osako
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title