Uploaded image for project: 'Dev - Nexus'
  1. Dev - Nexus
  2. NEXUS-12083

Exception while executing "Purge unused docker manifests and images" task

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.1
    • Component/s: Docker, Scheduled Tasks
    • Labels:
      None

      Description

      I have created a task of the type "Purge unused docker manifests and images" and when I execute it I always get this exception:

      2017-01-15 09:33:35,269+0000 INFO  [qtp1912801850-168] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Purge unused docker images and manifests ' [repository.docker.gc] state change WAITING -> RUNNING
      2017-01-15 09:33:35,362+0000 INFO  [quartz-2-thread-4] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=hosted, format=docker, name='docker-REPO'}
      2017-01-15 09:33:36,111+0000 ERROR [quartz-2-thread-4] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCTask - Failed to run task 'Purge unused docker manifests and images' on repository 'docker-REPO'
      org.sonatype.nexus.repository.docker.internal.V2Exception$ManifestInvalid: manifest invalid
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_112]
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_112]
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_112]
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_112]
      	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) [na:na]
      	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77) [na:na]
      	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84) [na:na]
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60) [na:na]
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235) [na:na]
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247) [na:na]
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.readManifest(V2ManifestUtilImpl.groovy:106) [na:na]
      	at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:155) [na:na]
      	at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:90) [na:na]
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) [org.sonatype.nexus.transaction:3.2.0.01]
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) [org.sonatype.nexus.transaction:3.2.0.01]
      	at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:74) [na:na]
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
      	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
      	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
      	at org.sonatype.nexus.repository.docker.internal.DockerGCTask.execute(DockerGCTask.java:42) [na:na]
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:56) [na:na]
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:89) [org.sonatype.nexus.scheduling:3.2.0.01]
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:144) [org.sonatype.nexus.quartz:3.2.0.01]
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:105) [org.sonatype.nexus.quartz:3.2.0.01]
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.quartz-scheduler.quartz:2.2.2]
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) [org.sonatype.nexus.thread:3.2.0.01]
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [org.apache.shiro.core:1.3.2]
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [org.apache.shiro.core:1.3.2]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
      2017-01-15 09:33:36,121+0000 INFO  [quartz-2-thread-4] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection starting on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=proxy, format=docker, name='docker-hub'}
      2017-01-15 09:33:36,137+0000 INFO  [quartz-2-thread-4] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl - Garbage collection completed on repository: RepositoryImpl$$EnhancerByGuice$$afd74251{type=proxy, format=docker, name='docker-hub'}
      2017-01-15 09:33:36,143+0000 WARN  [quartz-2-thread-4] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 3cad6e8d-18a2-437b-9898-8914dc8443b1 : 'Purge unused docker images and manifests ' [repository.docker.gc] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Purge unused docker manifests and images'; 1 failure
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95) [org.sonatype.goodies.common:2.2.3]
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:64) [na:na]
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:89) [org.sonatype.nexus.scheduling:3.2.0.01]
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:144) [org.sonatype.nexus.quartz:3.2.0.01]
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:105) [org.sonatype.nexus.quartz:3.2.0.01]
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.quartz-scheduler.quartz:2.2.2]
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) [org.sonatype.nexus.thread:3.2.0.01]
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [org.apache.shiro.core:1.3.2]
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [org.apache.shiro.core:1.3.2]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_112]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
      	Suppressed: org.sonatype.nexus.repository.docker.internal.V2Exception$ManifestInvalid: manifest invalid
      		at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_112]
      		at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_112]
      		at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_112]
      		at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_112]
      		at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83) [na:na]
      		at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77) [na:na]
      		at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84) [na:na]
      		at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60) [na:na]
      		at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235) [na:na]
      		at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247) [na:na]
      		at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.readManifest(V2ManifestUtilImpl.groovy:106) [na:na]
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:155) [na:na]
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:90) [na:na]
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) [org.sonatype.nexus.transaction:3.2.0.01]
      		at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) [org.sonatype.nexus.transaction:3.2.0.01]
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:74) [na:na]
      		at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
      		at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
      		at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
      		at org.sonatype.nexus.repository.docker.internal.DockerGCTask.execute(DockerGCTask.java:42) [na:na]
      		at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:56) [na:na]
      		... 12 common frames omitted
      
      

      I am trying to get the disk space utilization of nexus under control. I was hoping that it is better at doing this than the docker-registry provided by the docker guys, because this is also clogging the HD and is not releasing unused disk space.
      The Blobstore is at 30Gig right now and no compress etc. let's me get it to sane levels...

        Activity

        Hide
        muove Nikita Rogatov added a comment - - edited
        Show
        muove Nikita Rogatov added a comment - - edited The same story with me read here https://groups.google.com/a/glists.sonatype.com/forum/?utm_medium=email&utm_source=footer#!msg/nexus-users/xmZux266yuA/Pi3Yb57tEQAJ found manifest (just sha256) which create me problem
        Hide
        muove Nikita Rogatov added a comment -

        find the wrong manifest and delete it, if you can

        Show
        muove Nikita Rogatov added a comment - find the wrong manifest and delete it, if you can
        Hide
        pcornelissen@orchit.de Patrick Cornelissen added a comment -

        Well, the harddisk was full on saturday so I have removed the whole installation and started with a dedicated share for the data. I'll check if the error still occurs later when I have some more images in the blobstore again.

        Show
        pcornelissen@orchit.de Patrick Cornelissen added a comment - Well, the harddisk was full on saturday so I have removed the whole installation and started with a dedicated share for the data. I'll check if the error still occurs later when I have some more images in the blobstore again.
        Hide
        muove Nikita Rogatov added a comment -

        try to find which image is wrong.
        for it enable org.sonatype.nexus.repository.docker ot DEBUG
        find wrong images
        and delete it

        Show
        muove Nikita Rogatov added a comment - try to find which image is wrong. for it enable org.sonatype.nexus.repository.docker ot DEBUG find wrong images and delete it
        Hide
        Nikita.Loginov Nikita Loginov added a comment - - edited

        It is fixed for me. I've found and deleted manifest. -

        2017-01-24 12:46:12,206+0300 DEBUG [quartz-2-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl - Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 of type unspecified
        2017-01-24 12:46:12,300+0300 ERROR [quartz-2-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCTask - Failed to run task 'Purge unused docker manifests and images' on repository 'ADP'

        Thanks a lot, Nikita!

        Show
        Nikita.Loginov Nikita Loginov added a comment - - edited It is fixed for me. I've found and deleted manifest. - 2017-01-24 12:46:12,206+0300 DEBUG [quartz-2-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl - Parsing manifest v2/-/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 of type unspecified 2017-01-24 12:46:12,300+0300 ERROR [quartz-2-thread-20] *TASK org.sonatype.nexus.repository.docker.internal.DockerGCTask - Failed to run task 'Purge unused docker manifests and images' on repository 'ADP' Thanks a lot, Nikita!
        Hide
        jtom Joe Tom added a comment - - edited

        The manifests causing the failures will be corrected with an upgrade step in 3.2.1. After that the task should run as normal. We are still evaluting what caused the manifests to come into conflict.

        Show
        jtom Joe Tom added a comment - - edited The manifests causing the failures will be corrected with an upgrade step in 3.2.1. After that the task should run as normal. We are still evaluting what caused the manifests to come into conflict.
        Hide
        ddarby Dave Darby added a comment -

        FYI - I was on the phone today with TAPAD (customer) that is evaluating 3.2.0 PRO. They said they get new "corrupted" manifests on a daily basis. Including a manifest that got corrupted this morning at 8:00am, and none of the 3 users that evaluating the repo were working then. Is the fix in 3.2.1 just something that scans and fixes the "broken" manifests, or does the fix also "prevent" the broken manifests from getting created?

        Show
        ddarby Dave Darby added a comment - FYI - I was on the phone today with TAPAD (customer) that is evaluating 3.2.0 PRO. They said they get new "corrupted" manifests on a daily basis. Including a manifest that got corrupted this morning at 8:00am, and none of the 3 users that evaluating the repo were working then. Is the fix in 3.2.1 just something that scans and fixes the "broken" manifests, or does the fix also "prevent" the broken manifests from getting created?
        Hide
        jtom Joe Tom added a comment -

        Dave Darby This ticket fix is just for the Upgrade step. We believe we have already implemented the fix for the manifests elsewhere however are not 100% sure because there could be multiple reasons for the occurance. Anyone who has upgraded to 3.2.1 and continues to see this issue, please file a distinct ticket as we will likely need more information about your system.

        Show
        jtom Joe Tom added a comment - Dave Darby This ticket fix is just for the Upgrade step. We believe we have already implemented the fix for the manifests elsewhere however are not 100% sure because there could be multiple reasons for the occurance. Anyone who has upgraded to 3.2.1 and continues to see this issue, please file a distinct ticket as we will likely need more information about your system.

          People

          • Assignee:
            jtom Joe Tom
            Reporter:
            pcornelissen@orchit.de Patrick Cornelissen
            Last Updated By:
            Krzysztof DziÄ…dziak
          • Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Date of First Response: