Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-21835

Unable to run docker cleanup task 'delete unused'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.18.1, 3.19.1
    • Fix Version/s: None
    • Component/s: Docker, Scheduled Tasks
    • Labels:

      Description

      We have a somewhat large/busy nexus3 server running but lately some cleanup tasks have been failing due to heap memory issues on orientdb (afaik).

      Some details:

      nexus-status

      version 3.18.1-01
      edition OSS
      buildRevision 72e5414c4f3fb232e32cbfc9fca740111c94d07f
      buildTimestamp 2019-08-06-1749-11540
      java.runtime.version 1.8.0_202-b08

      Our jvm memory settings are in line with examples here: System Requirements

      so there is only a 4g heap with plenty of direct access to memory (39g)

      The blobstore in question holds ~ 420000 objects adding up to ~6tb. I ran the 'component count' script i found on another ticket here:

       

      "docker-dev":{"components":81047,"assets":417870} 

       

      Error:

      2019-11-12 14:55:20,517-0800 WARN  [quartz-3-thread-19]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task f9000db4-0354-4d54-98bd-7af383bf94b3 : 'delete-unused-docker' [repository.docker.gc] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Docker - Delete unused manifests and images'; 1 failure
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
      	at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: com.orientechnologies.orient.core.exception.OCommandExecutionException: Cannot execute query "SELECT FROM asset WHERE (attributes.docker.asset_kind = "BLOB" ) AND (bucket = #57:10 )": low heap memory
      	DB name="component"
      		at com.orientechnologies.orient.core.sql.OSoftQueryResultList.throwCanExecuteException(OSoftQueryResultList.java:688)
      		at com.orientechnologies.orient.core.sql.OSoftQueryResultList.checkQueue(OSoftQueryResultList.java:78)
      		at com.orientechnologies.orient.core.sql.OSoftQueryResultList.size(OSoftQueryResultList.java:88)
      		at java.util.Collections$SynchronizedCollection.size(Collections.java:2015)
      		at com.orientechnologies.orient.core.sql.query.OBasicResultSet.size(OBasicResultSet.java:66)
      		at com.orientechnologies.orient.core.sql.query.OBasicResultSet$1.hasNext(OBasicResultSet.java:86)
      		at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
      		at com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
      		at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
      		at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:200)
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:92)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
      		at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:75)
      		at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      		at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      		at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      		at org.sonatype.nexus.repository.docker.tasks.DockerGCTask.execute(DockerGCTask.java:44)
      		at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:75)
      		... 13 common frames omitted
      2019-11-12 14:55:20,518-0800 INFO  [quartz-3-thread-19]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'delete-unused-docker' [repository.docker.gc] state change RUNNING -> WAITING (FAILED) 

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              andrewkro Andrew Kroenert
              Last Updated By:
              Joe Tom Joe Tom
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title