Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-13087

caching asset download counts under high unique request volume can lead to OutOfMemoryError and instability

    XMLWordPrintable

    Details

      Description

      Nexus 3.3.0 introduced a feature that started to record download counts for repository assets. The new feature stores counts per unique asset requested during 1hr periods, before writing the counts to a database.

      Given enough unique asset requests in a short period of time, a large amount of memory can be consumed tracking these counts. The memory consumption may in some cases lead to OutOfMemoryError of the Nexus JVM and contribute to server instability. Request processing and server throughput may appear to slow under high memory use and more frequent garbage collection.

      Testing has revealed for every 30 unique asset GET requests/second (sustained over the course of hour) the cache would max out at ~100mb before being written to disk.

      Analysis of heap dumps where OutOfMemoryError was triggered has shown a thread such as this retaining the vast majority of heap space:

      pool-20-thread-1
      at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OWALRecordsFactory.toStream(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWALRecord;)[B (OWALRecordsFactory.java:38)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.wal.ODiskWriteAheadLog.log(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWALRecord;)Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OLogSequenceNumber; (ODiskWriteAheadLog.java:464)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.commitChanges(Lcom/orientechnologies/orient/core/storage/impl/local/paginated/wal/OWriteAheadLog;)V (OAtomicOperation.java:399)
      at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(ZLjava/lang/Exception;Ljava/lang/String;)Lcom/orientechnologies/orient/core/storage/impl/local/paginated/atomicoperations/OAtomicOperation; (OAtomicOperationsManager.java:468)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.endStorageTx()V (OAbstractPaginatedStorage.java:2968)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(Lcom/orientechnologies/orient/core/tx/OTransaction;Ljava/lang/Runnable;)Ljava/util/List; (OAbstractPaginatedStorage.java:1445)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit()V (OTransactionOptimistic.java:539)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(Z)V (OTransactionOptimistic.java:104)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(Z)Lcom/orientechnologies/orient/core/db/document/ODatabaseDocument; (ODatabaseDocumentTx.java:2840)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit()Lcom/orientechnologies/orient/core/db/ODatabase; (ODatabaseDocumentTx.java:2809)
      at org.sonatype.nexus.orient.transaction.OrientTransaction.commit()V (OrientTransaction.java:83)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(Lorg/sonatype/nexus/transaction/Transaction;)Ljava/lang/Object; (TransactionalWrapper.java:67)
      at org.sonatype.nexus.transaction.Operations.transactional(Lorg/sonatype/nexus/transaction/OperationPoint;)Ljava/lang/Object; (Operations.java:200)
      at org.sonatype.nexus.transaction.Operations.run(Lorg/sonatype/nexus/transaction/VoidOperation;)V (Operations.java:155)
      at org.sonatype.nexus.orient.transaction.OrientOperations.run(Lorg/sonatype/nexus/orient/transaction/OrientConsumer;)V (OrientOperations.java:63)
      at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountStoreImpl.lambda$8()V (AssetDownloadCountStoreImpl.java:205)
      at com.sonatype.nexus.plugins.healthcheck.downloads.AssetDownloadCountStoreImpl$$Lambda$158.run()V (Unknown Source)
      at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run()V (MDCAwareRunnable.java:40)
      at org.apache.shiro.subject.support.SubjectRunnable.doRun(Ljava/lang/Runnable;)V (SubjectRunnable.java:120)
      at org.apache.shiro.subject.support.SubjectRunnable.run()V (SubjectRunnable.java:108)
      at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:511)
      at java.util.concurrent.FutureTask.run()V (FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:617)
      at java.lang.Thread.run()V (Thread.java:748)
      
      

      Performance may be improved by a setting a system property to disable recording download asset counts.

      Workaround: Disabling Asset Download Counts

      1. Edit <data-dir>/etc/nexus.properties
      2. Add a new line at the end of that file that looks like this: nexus.assetdownloads.enabled=false
      3. Restart Nexus for the property change to have effect.

      Expected

      • recording asset download count performance should result in negligible performance regression

      Reference

      An article has been added discussing this.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dbradicich Damian Bradicich
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title