Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-21011

Compact blob store task cannot hard delete blob assets when the blobstore is out of space

    XMLWordPrintable

    Details

    • Notability:
      3

      Description

      The Compact Blob Store task has a goal to hard delete blobs that are marked soft-deleted. Especially important when you want to free up a full disk. However when a blob store disk is completely full, the task tries to write a queue file into the blobstore, which could prevent the task from running.

       

      2019-08-29 14:54:48,409-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task information:
      2019-08-29 14:54:48,411-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  ID: bf434673-65b2-48d5-969b-f67a28b7cf12
      2019-08-29 14:54:48,412-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Type: blobstore.compact
      2019-08-29 14:54:48,412-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Name: Compact Blob Store
      2019-08-29 14:54:48,412-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Description: Compacting docker blob store
      2019-08-29 14:54:48,413-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Deletions index file rebuild not required
      2019-08-29 14:54:48,414-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Begin deleted blobs processing
      2019-08-29 14:54:48,416-0700 ERROR [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Failed to run task 'Compacting docker blob store'
      org.sonatype.nexus.blobstore.api.BlobStoreException: java.io.IOException: No space left on device, Cause: No space left on device
      	at org.sonatype.nexus.blobstore.file.FileBlobStore.doCompact(FileBlobStore.java:562)
      	at org.sonatype.nexus.blobstore.BlobStoreSupport.compact(BlobStoreSupport.java:169)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      	at org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask.execute(CompactBlobStoreTask.java:53)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: No space left on device
      	at java.io.RandomAccessFile.writeBytes(Native Method)
      	at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
      	at com.squareup.tape.QueueFile.ringWrite(QueueFile.java:237)
      	at com.squareup.tape.QueueFile.ringErase(QueueFile.java:252)
      	at com.squareup.tape.QueueFile.remove(QueueFile.java:503)
      	at org.sonatype.nexus.blobstore.file.FileBlobStore.doCompact(FileBlobStore.java:543)
      	... 18 common frames omitted
      2019-08-29 14:54:48,416-0700 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task complete
       

      Expected

      Provide a supported way to free up disk on a full blobstore, without need to write more bytes to the same full blob store.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dbradicich Damian Bradicich
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Michael Prescott Michael Prescott
              Team:
              NXRM - Neo
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title