Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-17272

Docker uploads threads blocked

    Details

      Description

      A user was complaining of slow uploads for docker in 3.12.0. The thread dump shows over a hundred threads blocked:

      qtp167805473-277 id=277 state=BLOCKED
          - waiting to lock <0x0637e780> (a org.sonatype.nexus.repository.docker.internal.UploadManagerImpl)
           owned by qtp167805473-1130 id=1130
          at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.startUpload(UploadManagerImpl.java:47)
          at org.sonatype.nexus.repository.docker.internal.DockerHostedFacetImpl.beginBlobUpload(DockerHostedFacetImpl.java:553)
          at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$beginBlobUpload$4.call(Unknown Source)
          at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3.doCall(V2Handlers.groovy:111)
          at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
          at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
          at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
          at groovy.lang.Closure.call(Closure.java:414)
          at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
          at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
          at com.sun.proxy.$Proxy188.handle(Unknown Source)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
          at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
          at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
          at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:294)
          at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
      

      These are all waiting for this RUNNABLE thread:

      qtp167805473-1130 id=1130 state=RUNNABLE
          at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
          at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
          at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:417)
          at java.nio.file.Files.newDirectoryStream(Files.java:457)
          at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
          at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
          at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:95)
          at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
          at java.util.Iterator.forEachRemaining(Iterator.java:115)
          at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
          at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
          at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
          at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
          at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
          at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
          at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.blobsByName(UploadManagerImpl.java:146)
          at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.startUpload(UploadManagerImpl.java:48)
            - locked org.sonatype.nexus.repository.docker.internal.UploadManagerImpl@637e780
          at org.sonatype.nexus.repository.docker.internal.DockerHostedFacetImpl.beginBlobUpload(DockerHostedFacetImpl.java:553)
          at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$beginBlobUpload$4.call(Unknown Source)
          at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3.doCall(V2Handlers.groovy:111)
          at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
          at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
          at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
          at groovy.lang.Closure.call(Closure.java:414)
      

      Some of this problem is clearly due to the file system being unresponsive, but this has revealed a bottleneck in our code, uploads should not be getting serialized while waiting for this map to be built, as it seems like building it can be expensive:

      https://github.com/sonatype/nexus-internal/blob/release-3.12.0-01/private/plugins/nexus-repository-docker/src/main/java/org/sonatype/nexus/repository/docker/internal/UploadManagerImpl.java#L48

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch Peter Lynch
              Team:
              Nexus - Platform
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title