Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-14520

Nexus unresponsive due to healthcheck asset download count

    XMLWordPrintable

    Details

    • Story Points:
      1

      Description

      Nexus becomes unresponsive and CPU is high.

      Following threads are seen in the thread dump.

      pool-21-thread-3 id=412 state=WAITING
- waiting on <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- locked <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
owned by pool-20-thread-1 id=216
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:169)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:101)
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:519)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree$OSBTreeCursorForward.next(OSBTree.java:2066)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine$OSBTreeIndexCursor.nextEntry(OSBTreeIndexEngine.java:271)
at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:97)
at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:34)
at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:154)
at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:18)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1636)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1584)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2464)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2278)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1019)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:209)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:530)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:512)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:488)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392)
at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282)
at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55)
at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener$$Lambda$222/1068609197.accept(Unknown Source)
at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$105/715556946.run(Unknown Source)
at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
at org.sonatype.nexus.transaction.OperationPoint$$Lambda$106/1996542719.call(Unknown Source)
at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52)
at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

      See the following errors in the log:

      2017-10-09 14:16:05,826+0200 ERROR [pool-21-thread-2] anonymous com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=component}} Exception `7D7A47DF` in storage `component`
com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select from assetdownloadcount WHERE node_id = "41C4E904-CCB7359A-3056945C-A04D87F5-EA169947" AND repository_name = "public" AND asset_name = "org/aspectj/aspectjrt/1.8.6/aspectjrt-1.8.6.jar.sha1" AND date_type = :dateType AND date = date('2017-10-01 00:00:00.000', 'yyyy-MM-dd HH:mm:ss.SSS')
	DB name="component"
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3229)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
	at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
	at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
	at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
	at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
	at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392)
	at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282)
	at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55)
	at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
	at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
	at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
	at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
	at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
	at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52)
	at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51)
	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null

      Workaround is to disable asset download count.

      How do I disable recording download asset counts?

        Attachments

          Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-14612 unclosed transactions in groovy scripts can lead to instability

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-13087 caching asset download counts under high unique request volume can lead to OutOfMemoryError and instability

          • Major - Major loss of function.
          • Closed

            Activity

              People

              Assignee:
              dbradicich Damian Bradicich
              Reporter:
              christopher.dobbs@evry.com Christopher Richard Dobbs
              Last Updated By:
              Peter Lynch Peter Lynch
              Team:
              Nexus - UX
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title