Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19084

Many threads blocked in Elasticsearch while updating LastDownloaded attribute

    XMLWordPrintable

    Details

    • Story Points:
      3

      Description

      The attached thread dump shows 291 threads blocked with the stack trace below.

      The method "org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded" sometimes initiates a database transaction. When this transaction completes, the post-commit hook triggers an event that normally triggers an asynchronous elasticsearch update. However if the event thread pool is totally depleted then it will fall back to dispatching the update on the calling thread, in other words the update will become synchronous (due to a lack of threads).

      The database connection will only be released back to the pool once the post-commit hook is complete, so if elasticsearch is also running slow then this can result in the database connection being held open for a long time.

      In the case of the user this thread dump came from there is evidence that the disk elasticsearch is as running on is slow, and so this caused a massive backup of database connections.

       

      "qtp540936684-234" #234 prio=5 os_prio=0 tid=0x00007f3480013000 nid=0x170 waiting for monitor entry [0x00007f31f8f49000]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:283)

      • waiting to lock <0x00000006cbbeb6c0> (a org.elasticsearch.action.bulk.BulkProcessor)
        at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:268)
        at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:264)
        at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:250)
        at org.sonatype.nexus.repository.search.SearchServiceImpl.lambda$0(SearchServiceImpl.java:321)
        at org.sonatype.nexus.repository.search.SearchServiceImpl$$Lambda$289/275393005.accept(Unknown Source)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.sonatype.nexus.repository.search.SearchServiceImpl.bulkPut(SearchServiceImpl.java:315)
        at org.sonatype.nexus.repository.search.SearchFacetImpl.bulkPut(SearchFacetImpl.java:133)
        at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2.CGLIB$bulkPut$3(<generated>)
        at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2$$FastClassByGuice$$753cd40b.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
        at org.sonatype.nexus.repository.search.SearchFacetImpl$$EnhancerByGuice$$7cd025a2.bulkPut(<generated>)
        at org.sonatype.nexus.repository.search.IndexRequest.bulkApply(IndexRequest.java:92)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor.lambda$0(IndexRequestProcessor.java:139)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor$$Lambda$406/297358118.accept(Unknown Source)
        at java.util.Optional.ifPresent(Optional.java:159)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor.doUpdateSearchIndex(IndexRequestProcessor.java:135)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor.maybeUpdateSearchIndex(IndexRequestProcessor.java:115)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor$$Lambda$385/1669743823.accept(Unknown Source)
        at java.util.HashMap.forEach(HashMap.java:1289)
        at org.sonatype.nexus.repository.search.IndexBatchRequest.apply(IndexBatchRequest.java:93)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor.process(IndexRequestProcessor.java:99)
        at org.sonatype.nexus.repository.search.IndexRequestProcessor.on(IndexRequestProcessor.java:88)
        at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
        at org.sonatype.nexus.internal.event.AffinityBarrier.lambda$1(AffinityBarrier.java:91)
        at org.sonatype.nexus.internal.event.AffinityBarrier$$Lambda$384/1417277946.run(Unknown Source)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at org.sonatype.nexus.internal.event.EventExecutor.lambda$0(EventExecutor.java:72)
        at org.sonatype.nexus.internal.event.EventExecutor$$Lambda$24/155969703.rejectedExecution(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at org.apache.shiro.concurrent.SubjectAwareExecutor.execute(SubjectAwareExecutor.java:129)
        at org.sonatype.nexus.internal.event.AffinityBarrier.execute(AffinityBarrier.java:89)
        at org.sonatype.nexus.internal.event.EventExecutor.execute(EventExecutor.java:215)
        at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)
        at com.google.common.eventbus.Dispatcher$ImmediateDispatcher.dispatch(Dispatcher.java:186)
        at com.google.common.eventbus.EventBus.post(EventBus.java:212)
        at org.sonatype.nexus.internal.event.EventManagerImpl.lambda$0(EventManagerImpl.java:132)
        at org.sonatype.nexus.internal.event.EventManagerImpl$$Lambda$213/1347710971.run(Unknown Source)
        at org.sonatype.nexus.internal.event.AffinityBarrier.lambda$0(AffinityBarrier.java:75)
        at org.sonatype.nexus.internal.event.AffinityBarrier$$Lambda$252/1880833496.run(Unknown Source)
        at com.google.common.util.concurrent.SequentialExecutor$1.run(SequentialExecutor.java:120)
        at com.google.common.util.concurrent.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:227)
        at com.google.common.util.concurrent.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:171)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at org.sonatype.nexus.internal.event.EventExecutor.lambda$0(EventExecutor.java:72)
        at org.sonatype.nexus.internal.event.EventExecutor$$Lambda$24/155969703.rejectedExecution(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
        at org.apache.shiro.concurrent.SubjectAwareExecutor.execute(SubjectAwareExecutor.java:129)
        at com.google.common.util.concurrent.SequentialExecutor.execute(SequentialExecutor.java:128)
        at org.sonatype.nexus.internal.event.AffinityBarrier.coordinate(AffinityBarrier.java:71)
        at org.sonatype.nexus.internal.event.EventExecutor.executeWithAffinity(EventExecutor.java:199)
        at org.sonatype.nexus.internal.event.EventManagerImpl.post(EventManagerImpl.java:132)
        at org.sonatype.nexus.orient.entity.EntityHook.postEvents(EntityHook.java:325)
        at org.sonatype.nexus.orient.entity.EntityHook.flushEvents(EntityHook.java:289)
        at org.sonatype.nexus.orient.entity.EntityHook.onAfterTxCommit(EntityHook.java:174)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2949)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
        at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:183)
        at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
        at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
        at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
        at com.sun.proxy.$Proxy218.commit(Unknown Source)
        at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
        at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
        at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
        at org.sonatype.nexus.repository.view.handlers.LastDownloadedHand.maybeUpdateLastDownloaded(LastDownloadedHandler.java:107)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mjohnson Matt Johnson
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Peter Lynch
              Team:
              NXRM - Morpheus
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title