Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-20709

Recovery of large docker layer runs out of heap with recommended heap defined

Details

    • 5
    • NXRM MadMax Sprint 21
    • 3
    • 3

    Description

      I noticed that if I had a large layer deleted from my database to simulate a condition where I would need to run the "Repair - Reconcile component database from blob store" task, that it would error with an out of heap error (below) despite me having the recommended heap for my machine/hardware.

      2019-08-01 18:30:43,816-0400 INFO  [qtp341555250-206] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'dockermac' [blobstore.rebuildComponentDB] state change WAITING -> RUNNING
      2019-08-01 18:30:43,840-0400 INFO  [quartz-3-thread-3] *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - Task log: /Users/jtom/Documents/Work/nexus/NX3/nexus-internal/target/sonatype-work/nexus3/log/tasks/blobstore.rebuildComponentDB-20190801183043832.log
      2019-08-01 18:30:48,304-0400 INFO  [quartz-3-thread-3] *SYSTEM com.sonatype.nexus.blobstore.restore.internal.DockerRestoreBlobStrategy - Restoring a v2 generated asset/component: v2/-/blobs/sha256:7fa8ea8ddbeedf89f7e47e9f8f0d7fdc7e238d1faf3d2b33b785a801bbcd5f23
      2019-08-01 18:31:03,433-0400 ERROR [quartz-3-thread-3] *SYSTEM org.quartz.core.JobRunShell - Job nexus.70f68763-e7bc-4315-9313-cdb2502d9da7 threw an unhandled Exception: 
      java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOfRange(Arrays.java:3664)
      	at java.lang.String.<init>(String.java:207)
      	at java.lang.StringBuilder.toString(StringBuilder.java:407)
      	at com.google.common.io.CharStreams.toString(CharStreams.java:164)
      	at com.google.common.io.CharStreams$toString.call(Unknown Source)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl$_manifestDigest_closure1.doCall(V2ManifestUtilImpl.groovy:155)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1630)
      	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
      	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.manifestDigest(V2ManifestUtilImpl.groovy:154)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getV2Digest(DockerRestoreFacetImpl.java:281)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getDigest(DockerRestoreFacetImpl.java:271)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.doSaveAsset(DockerRestoreFacetImpl.java:235)
      2019-08-01 18:31:03,435-0400 ERROR [quartz-3-thread-3] *SYSTEM org.quartz.core.ErrorLogger - Job (nexus.70f68763-e7bc-4315-9313-cdb2502d9da7 threw an exception.
      org.quartz.SchedulerException: Job threw an unhandled exception.
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
      	at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOfRange(Arrays.java:3664)
      	at java.lang.String.<init>(String.java:207)
      	at java.lang.StringBuilder.toString(StringBuilder.java:407)
      	at com.google.common.io.CharStreams.toString(CharStreams.java:164)
      	at com.google.common.io.CharStreams$toString.call(Unknown Source)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl$_manifestDigest_closure1.doCall(V2ManifestUtilImpl.groovy:155)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1630)
      	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
      	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.manifestDigest(V2ManifestUtilImpl.groovy:154)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getV2Digest(DockerRestoreFacetImpl.java:281)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getDigest(DockerRestoreFacetImpl.java:271)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.doSaveAsset(DockerRestoreFacetImpl.java:235)
      2019-08-01 18:31:03,439-0400 INFO  [quartz-3-thread-3] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'dockermac' [blobstore.rebuildComponentDB] state change RUNNING -> WAITING (FAILED)
      

      The test image I used was couchbase which currently has a ~400MB layer. I originally noticed testing a 1.6GB foreign layer but it is reproducable without that.

      This may be an improvement (optimization) but I'm starting with bug.

      Attachments

        Activity

          People

            ybulatnikov Eugene Bulatnikov
            jtom Joe Tom
            Michael Oliverio Michael Oliverio
            NXRM - Mad Max
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              tigCommentSecurity.panel-title