Details
Description
A user was complaining of slow uploads for docker in 3.12.0. The thread dump shows over a hundred threads blocked:
qtp167805473-277 id=277 state=BLOCKED
- waiting to lock <0x0637e780> (a org.sonatype.nexus.repository.docker.internal.UploadManagerImpl)
owned by qtp167805473-1130 id=1130
at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.startUpload(UploadManagerImpl.java:47)
at org.sonatype.nexus.repository.docker.internal.DockerHostedFacetImpl.beginBlobUpload(DockerHostedFacetImpl.java:553)
at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$beginBlobUpload$4.call(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3.doCall(V2Handlers.groovy:111)
at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
at com.sun.proxy.$Proxy188.handle(Unknown Source)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:294)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
These are all waiting for this RUNNABLE thread:
qtp167805473-1130 id=1130 state=RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:417)
at java.nio.file.Files.newDirectoryStream(Files.java:457)
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
at java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:95)
at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
at java.util.Iterator.forEachRemaining(Iterator.java:115)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.blobsByName(UploadManagerImpl.java:146)
at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.startUpload(UploadManagerImpl.java:48)
- locked org.sonatype.nexus.repository.docker.internal.UploadManagerImpl@637e780
at org.sonatype.nexus.repository.docker.internal.DockerHostedFacetImpl.beginBlobUpload(DockerHostedFacetImpl.java:553)
at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$beginBlobUpload$4.call(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure3.doCall(V2Handlers.groovy:111)
at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at groovy.lang.Closure.call(Closure.java:414)
Some of this problem is clearly due to the file system being unresponsive, but this has revealed a bottleneck in our code, uploads should not be getting serialized while waiting for this map to be built, as it seems like building it can be expensive:
Attachments
Issue Links
- is duplicated by
-
NEXUS-17466 S3 Blob Store really slow to upload with docker
-
- Closed
-
