Many threads blocked in Elasticsearch while updating LastDownloaded attribute

The attached thread dump shows 291 threads blocked with the stack trace below.

The method "org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded" sometimes initiates a database transaction. When this transaction completes, the post-commit hook triggers an event that normally triggers an asynchronous elasticsearch update. However if the event thread pool is totally depleted then it will fall back to dispatching the update on the calling thread, in other words the update will become synchronous (due to a lack of threads).

The database connection will only be released back to the pool once the post-commit hook is complete, so if elasticsearch is also running slow then this can result in the database connection being held open for a long time.

In the case of the user this thread dump came from there is evidence that the disk elasticsearch is as running on is slow, and so this caused a massive backup of database connections.

"qtp540936684-234" #234 prio=5 os_prio=0 tid=0x00007f3480013000 nid=0x170 waiting for monitor entry [0x00007f31f8f49000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:283)

• waiting to lock <0x00000006cbbeb6c0> (a org.elasticsearch.action.bulk.BulkProcessor)
  at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
  at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
  at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:250)
  at org.sonatype.nexus.repository.search.SearchServiceImpl.lambda$0(SearchServiceImpl.java:321)
  at org.sonatype.nexus.repository.search.SearchServiceImpl$$Lambda$289/275393005.accept(Unknown Source)
  at java.lang.Iterable.forEach(Iterable.java:75)
  at org.sonatype.nexus.repository.search.SearchServiceImpl.bulkPut(SearchServiceImpl.java:315)
  at org.sonatype.nexus.repository.search.SearchFacetImpl.bulkPut(SearchFacetImpl.java:133)
  at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2.CGLIB$bulkPut$3(<generated>)
  at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2$$FastClassByGuice$$753cd40b.invoke(<generated>)
  at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
  at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
  at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
  at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
  at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
  at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
  at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
  at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2.bulkPut(<generated>)
  at org.sonatype.nexus.repository.search.IndexRequest.bulkApply(IndexRequest.java:92)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor.lambda$0(IndexRequestProcessor.java:139)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor$$Lambda$406/297358118.accept(Unknown Source)
  at java.util.Optional.ifPresent(Optional.java:159)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor.doUpdateSearchIndex(IndexRequestProcessor.java:135)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor.maybeUpdateSearchIndex(IndexRequestProcessor.java:115)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor$$Lambda$385/1669743823.accept(Unknown Source)
  at java.util.HashMap.forEach(HashMap.java:1289)
  at org.sonatype.nexus.repository.search.IndexBatchRequest.apply(IndexBatchRequest.java:93)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor.process(IndexRequestProcessor.java:99)
  at org.sonatype.nexus.repository.search.IndexRequestProcessor.on(IndexRequestProcessor.java:88)
  at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
  at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
  at org.sonatype.nexus.internal.event.AffinityBarrier.lambda$1(AffinityBarrier.java:91)
  at org.sonatype.nexus.internal.event.AffinityBarrier$$Lambda$384/1417277946.run(Unknown Source)
  at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
  at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
  at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
  at org.sonatype.nexus.internal.event.EventExecutor.lambda$0(EventExecutor.java:72)
  at org.sonatype.nexus.internal.event.EventExecutor$$Lambda$24/155969703.rejectedExecution(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
  at org.apache.shiro.concurrent.SubjectAwareExecutor.execute(SubjectAwareExecutor.java:129)
  at org.sonatype.nexus.internal.event.AffinityBarrier.execute(AffinityBarrier.java:89)
  at org.sonatype.nexus.internal.event.EventExecutor.execute(EventExecutor.java:215)
  at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)
  at com.google.common.eventbus.Dispatcher$ImmediateDispatcher.dispatch(Dispatcher.java:186)
  at com.google.common.eventbus.EventBus.post(EventBus.java:212)
  at org.sonatype.nexus.internal.event.EventManagerImpl.lambda$0(EventManagerImpl.java:132)
  at org.sonatype.nexus.internal.event.EventManagerImpl$$Lambda$213/1347710971.run(Unknown Source)
  at org.sonatype.nexus.internal.event.AffinityBarrier.lambda$0(AffinityBarrier.java:75)
  at org.sonatype.nexus.internal.event.AffinityBarrier$$Lambda$252/1880833496.run(Unknown Source)
  at com.google.common.util.concurrent.SequentialExecutor$1.run(SequentialExecutor.java:120)
  at com.google.common.util.concurrent.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:227)
  at com.google.common.util.concurrent.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:171)
  at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
  at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
  at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
  at org.sonatype.nexus.internal.event.EventExecutor.lambda$0(EventExecutor.java:72)
  at org.sonatype.nexus.internal.event.EventExecutor$$Lambda$24/155969703.rejectedExecution(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
  at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
  at org.apache.shiro.concurrent.SubjectAwareExecutor.execute(SubjectAwareExecutor.java:129)
  at com.google.common.util.concurrent.SequentialExecutor.execute(SequentialExecutor.java:128)
  at org.sonatype.nexus.internal.event.AffinityBarrier.coordinate(AffinityBarrier.java:71)
  at org.sonatype.nexus.internal.event.EventExecutor.executeWithAffinity(EventExecutor.java:199)
  at org.sonatype.nexus.internal.event.EventManagerImpl.post(EventManagerImpl.java:132)
  at org.sonatype.nexus.orient.entity.EntityHook.postEvents(EntityHook.java:325)
  at org.sonatype.nexus.orient.entity.EntityHook.flushEvents(EntityHook.java:289)
  at org.sonatype.nexus.orient.entity.EntityHook.onAfterTxCommit(EntityHook.java:174)
  at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2949)
  at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
  at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:183)
  at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
  at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
  at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
  at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
  at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
  at com.sun.proxy.$Proxy218.commit(Unknown Source)
  at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
  at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
  at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
  at org.sonatype.nexus.repository.view.handlers.LastDownloadedHand.maybeUpdateLastDownloaded(LastDownloadedHandler.java:107)