Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: 3.5.0
-
Fix Version/s: None
-
Component/s: Repository Health Check
-
Labels:
-
Story Points:1
Description
Nexus becomes unresponsive and CPU is high.
Following threads are seen in the thread dump.
pool-21-thread-3 id=412 state=WAITING - waiting on <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) - locked <0x3976f8a9> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) owned by pool-20-thread-1 id=216 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:169) at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.acquireLock(OOneEntryPerKeyLockManager.java:101) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.acquireReadLock(OAtomicOperationsManager.java:519) at com.orientechnologies.orient.core.index.sbtree.local.OSBTree$OSBTreeCursorForward.next(OSBTree.java:2066) at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine$OSBTreeIndexCursor.nextEntry(OSBTreeIndexEngine.java:271) at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:97) at com.orientechnologies.orient.core.index.OIndexAbstractCursor.next(OIndexAbstractCursor.java:34) at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:154) at com.orientechnologies.orient.core.index.OIndexChangesWrapper.next(OIndexChangesWrapper.java:18) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1636) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1584) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2464) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2278) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1019) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:209) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:530) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:512) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:488) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146) at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78) at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74) at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282) at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55) at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener$$Lambda$222/1068609197.accept(Unknown Source) at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63) at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$105/715556946.run(Unknown Source) at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53) at org.sonatype.nexus.transaction.OperationPoint$$Lambda$106/1996542719.call(Unknown Source) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.run(Operations.java:155) at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63) at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52) at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
See the following errors in the log:
2017-10-09 14:16:05,826+0200 ERROR [pool-21-thread-2] anonymous com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - $ANSI{green {db=component}} Exception `7D7A47DF` in storage `component` com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select from assetdownloadcount WHERE node_id = "41C4E904-CCB7359A-3056945C-A04D87F5-EA169947" AND repository_name = "public" AND asset_name = "org/aspectj/aspectjrt/1.8.6/aspectjrt-1.8.6.jar.sha1" AND date_type = :dateType AND date = date('2017-10-01 00:00:00.000', 'yyyy-MM-dd HH:mm:ss.SSS') DB name="component" at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3229) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146) at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78) at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74) at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:755) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLUpdate.execute(OCommandExecutorSQLUpdate.java:291) at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146) at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:392) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountEntityAdapter.incrementCount(AssetDownloadCountEntityAdapter.java:282) at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.lambda$0(CacheRemovalListener.java:55) at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63) at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.run(Operations.java:155) at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63) at com.sonatype.nexus.plugins.healthcheck.downloads.CacheRemovalListener.onRemoval(CacheRemovalListener.java:52) at com.google.common.cache.RemovalListeners$1$1.run(RemovalListeners.java:51) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null
Workaround is to disable asset download count.
Attachments
Issue Links
- is related to
-
NEXUS-14612 unclosed transactions in groovy scripts can lead to instability
-
- Closed
-
-
NEXUS-13087 caching asset download counts under high unique request volume can lead to OutOfMemoryError and instability
-
- Closed
-