Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-14520

Nexus unresponsive due to healthcheck asset download count

    XMLWordPrintable

    Details

    • Story Points:
      1

      Description

      Nexus becomes unresponsive and CPU is high.

      Following threads are seen in the thread dump.

      pool-21-thread-3 id=412 state=WAITING
      - waiting on <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      - locked <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      owned by pool-20-thread-1 id=216
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
      at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
      at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:169)
      at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:101)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:519)
      at com.orientechnologies.orient.core.index.sbtree.local.OSBTree$OSBTreeCursorForward.next(OSBTree.java:2066)
      at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine$OSBTreeIndexCursor.nextEntry(OSBTreeIndexEngine.java:271)
      at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:97)
      at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:34)
      at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:154)
      at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:18)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1636)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1584)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2464)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2278)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1019)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:209)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:530)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:512)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:488)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
      at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
      at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
      at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291)
      at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
      at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
      at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392)
      at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282)
      at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55)
      at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener$$Lambda$222/1068609197.accept(Unknown Source)
      at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$105/715556946.run(Unknown Source)
      at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      at org.sonatype.nexus.transaction.OperationPoint$$Lambda$106/1996542719.call(Unknown Source)
      at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52)
      at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51)
      at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      

      See the following errors in the log:

      2017-10-09 14:16:05,826+0200 ERROR [pool-21-thread-2] anonymous com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=component}} Exception `7D7A47DF` in storage `component`
      com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select from assetdownloadcount WHERE node_id = "41C4E904-CCB7359A-3056945C-A04D87F5-EA169947" AND repository_name = "public" AND asset_name = "org/aspectj/aspectjrt/1.8.6/aspectjrt-1.8.6.jar.sha1" AND date_type = :dateType AND date = date('2017-10-01 00:00:00.000', 'yyyy-MM-dd HH:mm:ss.SSS')
      	DB name="component"
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3229)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
      	at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
      	at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
      	at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291)
      	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
      	at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
      	at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392)
      	at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282)
      	at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55)
      	at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      	at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      	at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      	at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      	at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52)
      	at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.NullPointerException: null
      

      Workaround is to disable asset download count.

      How do I disable recording download asset counts?

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dbradicich Damian Bradicich
              Reporter:
              christopher.dobbs@evry.com Christopher Richard Dobbs
              Last Updated By:
              Peter Lynch
              Team:
              Nexus - UX
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title