Slow delete performance when using REST API

Deletions of yum components are taking 60 seconds and sometimes failing with 500.

We can see database query reaching 60 second timeout.

2019-01-10 04:51:29,684+0000 WARN  [qtp1990821672-1160244]  xxxxx org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID f3abb01b-7dda-4c74-8f03-4eeb03d78911) Unexpected exception: java.lang.IllegalStateException: Timed out reading query result from queue 11e742fa after 60 seconds

java.lang.IllegalStateException: Timed out reading query result from queue 11e742fa after 60 seconds

 at org.sonatype.nexus.repository.storage.OrientAsyncHelper$QueueConsumingIterable.hasNext(OrientAsyncHelper.java:201)

 at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1811)

 at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)

 at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)

 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)

 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)

 at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)

 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

 at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)

 at org.sonatype.nexus.repository.browse.internal.BrowseServiceImpl.getById(BrowseServiceImpl.java:309)

 at org.sonatype.nexus.repository.browse.internal.BrowseServiceImpl.getComponentById(BrowseServiceImpl.java:282)

 at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.getComponent(ComponentsResource.java:210)

 at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.deleteComponent(ComponentsResource.java:228)

We can also see the the following thread is taking time:

event-6-thread-2281 - priority:5 - threadId:0x00007fe298055800 - nativeId:0x7e23 - nativeId (decimal):32291 - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)

• locked <0x0000000707ae4788> (a java.util.zip.ZStreamRef)
  at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:311)
  at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
• locked <0x0000000707aec6a8> (a java.io.BufferedInputStream)
  at com.google.common.io.CountingInputStream.read(CountingInputStream.java:63)
  at org.sonatype.nexus.common.hash.MultiHashingInputStream.read(MultiHashingInputStream.java:66)
  at java.io.FilterInputStream.read(FilterInputStream.java:107)
  at com.google.common.io.ByteStreams.copy(ByteStreams.java:109)
  at org.sonatype.nexus.repository.yum.internal.utils.YumMetadataUtils.readCompressedMetadata(YumMetadataUtils.java:79)
  at org.sonatype.nexus.repository.yum.internal.utils.YumMetadataUtils.readCompressedMetadata(YumMetadataUtils.java:59)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.readMetadataAndAppendToRepomd(CreateRepoServiceImpl.java:371)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.writeRepomd(CreateRepoServiceImpl.java:355)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.convertDirectoriesToMetadata(CreateRepoServiceImpl.java:184)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.buildMetadata(CreateRepoServiceImpl.java:150)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.buildMetadata(CreateRepoServiceImpl.java:134)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl.buildMetadataUsingCaching(CreateRepoServiceImpl.java:120)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl$$EnhancerByGuice$$7fce96dd.CGLIB$buildMetadataUsingCaching$0(<generated>)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl$$EnhancerByGuice$$7fce96dd$$FastClassByGuice$$6d83d52f.invoke(<generated>)
  at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
  at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
  at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
  at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
  at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
  at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoServiceImpl$$EnhancerByGuice$$7fce96dd.buildMetadataUsingCaching(<generated>)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoFacetImpl.buildMetadata(CreateRepoFacetImpl.java:190)
  at org.sonatype.nexus.repository.yum.internal.createrepo.CreateRepoFacetImpl.on(CreateRepoFacetImpl.java:175)
  at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
  at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144)
• locked <0x00000006cec7b2a8> (a com.google.common.eventbus.Subscriber$SynchronizedSubscriber)
  at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
  at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
  at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
  at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  Locked ownable synchronizers:
  - <0x00000006ec0c4ee8> (a java.util.concurrent.ThreadPoolExecutor$Worker)