Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-20709

Recovery of large docker layer runs out of heap with recommended heap defined

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Sprint:
      NXRM MadMax Sprint 21
    • Notability:
      3

      Description

      I noticed that if I had a large layer deleted from my database to simulate a condition where I would need to run the "Repair - Reconcile component database from blob store" task, that it would error with an out of heap error (below) despite me having the recommended heap for my machine/hardware.

      2019-08-01 18:30:43,816-0400 INFO  [qtp341555250-206] admin org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'dockermac' [blobstore.rebuildComponentDB] state change WAITING -> RUNNING
      2019-08-01 18:30:43,840-0400 INFO  [quartz-3-thread-3] *SYSTEM org.sonatype.nexus.blobstore.restore.RestoreMetadataTask - Task log: /Users/jtom/Documents/Work/nexus/NX3/nexus-internal/target/sonatype-work/nexus3/log/tasks/blobstore.rebuildComponentDB-20190801183043832.log
      2019-08-01 18:30:48,304-0400 INFO  [quartz-3-thread-3] *SYSTEM com.sonatype.nexus.blobstore.restore.internal.DockerRestoreBlobStrategy - Restoring a v2 generated asset/component: v2/-/blobs/sha256:7fa8ea8ddbeedf89f7e47e9f8f0d7fdc7e238d1faf3d2b33b785a801bbcd5f23
      2019-08-01 18:31:03,433-0400 ERROR [quartz-3-thread-3] *SYSTEM org.quartz.core.JobRunShell - Job nexus.70f68763-e7bc-4315-9313-cdb2502d9da7 threw an unhandled Exception: 
      java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOfRange(Arrays.java:3664)
      	at java.lang.String.<init>(String.java:207)
      	at java.lang.StringBuilder.toString(StringBuilder.java:407)
      	at com.google.common.io.CharStreams.toString(CharStreams.java:164)
      	at com.google.common.io.CharStreams$toString.call(Unknown Source)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl$_manifestDigest_closure1.doCall(V2ManifestUtilImpl.groovy:155)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1630)
      	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
      	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.manifestDigest(V2ManifestUtilImpl.groovy:154)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getV2Digest(DockerRestoreFacetImpl.java:281)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getDigest(DockerRestoreFacetImpl.java:271)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.doSaveAsset(DockerRestoreFacetImpl.java:235)
      2019-08-01 18:31:03,435-0400 ERROR [quartz-3-thread-3] *SYSTEM org.quartz.core.ErrorLogger - Job (nexus.70f68763-e7bc-4315-9313-cdb2502d9da7 threw an exception.
      org.quartz.SchedulerException: Job threw an unhandled exception.
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
      	at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOfRange(Arrays.java:3664)
      	at java.lang.String.<init>(String.java:207)
      	at java.lang.StringBuilder.toString(StringBuilder.java:407)
      	at com.google.common.io.CharStreams.toString(CharStreams.java:164)
      	at com.google.common.io.CharStreams$toString.call(Unknown Source)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl$_manifestDigest_closure1.doCall(V2ManifestUtilImpl.groovy:155)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.IOGroovyMethods.withCloseable(IOGroovyMethods.java:1630)
      	at org.codehaus.groovy.runtime.NioGroovyMethods.withCloseable(NioGroovyMethods.java:1759)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
      	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:56)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.repository.docker.internal.V2ManifestUtilImpl.manifestDigest(V2ManifestUtilImpl.groovy:154)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getV2Digest(DockerRestoreFacetImpl.java:281)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.getDigest(DockerRestoreFacetImpl.java:271)
      	at org.sonatype.nexus.repository.docker.internal.DockerRestoreFacetImpl.doSaveAsset(DockerRestoreFacetImpl.java:235)
      2019-08-01 18:31:03,439-0400 INFO  [quartz-3-thread-3] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'dockermac' [blobstore.rebuildComponentDB] state change RUNNING -> WAITING (FAILED)
      

      The test image I used was couchbase which currently has a ~400MB layer. I originally noticed testing a 1.6GB foreign layer but it is reproducable without that.

      This may be an improvement (optimization) but I'm starting with bug.

        Attachments

          Activity

            People

            Assignee:
            ybulatnikov Eugene Bulatnikov
            Reporter:
            jtom Joe Tom
            Last Updated By:
            Eugene Bulatnikov Eugene Bulatnikov
            Team:
            NXRM - Mad Max
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title