Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.3.0, 3.3.1, 3.3.2
-
Fix Version/s: 3.4.0
-
Component/s: Repository Health Check
-
Labels:
Description
Nexus 3.3.0 introduced a feature that started to record download counts for repository assets. The new feature stores counts per unique asset requested during 1hr periods, before writing the counts to a database.
Given enough unique asset requests in a short period of time, a large amount of memory can be consumed tracking these counts. The memory consumption may in some cases lead to OutOfMemoryError of the Nexus JVM and contribute to server instability. Request processing and server throughput may appear to slow under high memory use and more frequent garbage collection.
Testing has revealed for every 30 unique asset GET requests/second (sustained over the course of hour) the cache would max out at ~100mb before being written to disk.
Analysis of heap dumps where OutOfMemoryError was triggered has shown a thread such as this retaining the vast majority of heap space:
pool-20-thread-1 at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48) at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALRecordsFactory.toStream(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWALRecord;)[B (OWALRecordsFactory.java:38) at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.log(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWALRecord;)Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OLogSequenceNumber; (ODiskWriteAheadLog.java:464) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.commitChanges(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWriteAheadLog;)V (OAtomicOperation.java:399) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(ZLjava/lang/Exception;Ljava/lang/String;)Lcom/orientechnologies/orient/core/storage/impl/local/paginated/atomicoperations/OAtomicOperation; (OAtomicOperationsManager.java:468) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.endStorageTx()V (OAbstractPaginatedStorage.java:2968) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(Lcom/orientechnologies/orient/core/tx/OTransaction;Ljava/lang/Runnable;)Ljava/util/List; (OAbstractPaginatedStorage.java:1445) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit()V (OTransactionOptimistic.java:539) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(Z)V (OTransactionOptimistic.java:104) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(Z)Lcom/orientechnologies/orient/core/db/document/ODatabaseDocument; (ODatabaseDocumentTx.java:2840) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit()Lcom/orientechnologies/orient/core/db/ODatabase; (ODatabaseDocumentTx.java:2809) at org.sonatype.nexus.orient.transaction.OrientTransaction.commit()V (OrientTransaction.java:83) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(Lorg/sonatype/nexus/transaction/Transaction;)Ljava/lang/Object; (TransactionalWrapper.java:67) at org.sonatype.nexus.transaction.Operations.transactional(Lorg/sonatype/nexus/transaction/OperationPoint;)Ljava/lang/Object; (Operations.java:200) at org.sonatype.nexus.transaction.Operations.run(Lorg/sonatype/nexus/transaction/VoidOperation;)V (Operations.java:155) at org.sonatype.nexus.orient.transaction.OrientOperations.run(Lorg/sonatype/nexus/orient/transaction/OrientConsumer;)V (OrientOperations.java:63) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountStoreImpl.lambda$8()V (AssetDownloadCountStoreImpl.java:205) at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountStoreImpl$$Lambda$158.run()V (Unknown Source) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run()V (MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(Ljava/lang/Runnable;)V (SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run()V (SubjectRunnable.java:108) at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:511) at java.util.concurrent.FutureTask.run()V (FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:617) at java.lang.Thread.run()V (Thread.java:748)
Performance may be improved by a setting a system property to disable recording download asset counts.
Workaround: Disabling Asset Download Counts
- Edit <data-dir>/etc/nexus.properties
- Add a new line at the end of that file that looks like this: nexus.assetdownloads.enabled=false
- Restart Nexus for the property change to have effect.
Expected
- recording asset download count performance should result in negligible performance regression
Reference
Attachments
Issue Links
- relates
-
-
- Closed
-
-
-
- Closed
-