Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16852

HA-C with Nexus Firewall evaluations may fail due to OConcurrentModificationException Cannot UPDATE the record because the version is not the latest

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.10.0, 3.12.1, 3.13.0
    • Fix Version/s: 3.16.0
    • Component/s: HA

      Description

      OConcurrentModificationException ERROR message in Nexus application log (and Firewall.Audit task log) in an HA-C cluster.

      When Nexus Firewall is enabled in an HA cluster, bulk component checks to IQ server attempt to  update all the given assets in the same transaction, which could be a large number.

      If just one of those assets has been concurrently modified by another thread since the transaction started then it will start over from scratch.

      When failure happens, the quarantine status for the affected components may be prevented from being served through repository manager as their quarantine status will change the PENDING. The amount of time it may take for the affected components to leave PENDING status is indeterminate.

      Example log messages that may be seen:

      Example 1
      2018-04-12 15:29:17,315-0400 ERROR [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Failed to run task 'Auditing components in repository npmjs'
      org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
      at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224)
      at sun.reflect.GeneratedMethodAccessor946.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      at com.sun.proxy.$Proxy197.allowRetry(Unknown Source)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
      at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      at com.sonatype.nexus.clm.internal.EvaluationSender.sendAuditRequest(EvaluationSender.java:102)
      at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:88)
      at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:130)
      at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:1)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2002 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v3 your=v2)
      DB name="component"
      at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
      at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884)
      at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607)
      at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603)
      at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98)
      at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88)
      at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127)
      2018-04-12 15:29:17,315-0400 INFO [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Task complete
      
      Example 2
      2018-04-17 16:23:45,165-0400 ERROR [fte-4-thread-13] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.FirewallTransactionQueue - Could not send update to IQ Server: EVALUATE_COMPONENT npmjs:sinon-chai/-/sinon-chai-2.14.0.tgz. Error was: Reached max retries: 8/8
      org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
      at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224)
      at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
      at com.sun.proxy.$Proxy196.allowRetry(Unknown Source)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
      at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      at com.sonatype.nexus.clm.internal.EvaluationSender.sendQuarantineRequest(EvaluationSender.java:129)
      at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:85)
      at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:64)
      at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.evaluateComponent(FirewallTransactionProcessor.java:163)
      at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransactionWithoutErrorHandler(FirewallTransactionProcessor.java:110)
      at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransaction(FirewallTransactionProcessor.java:83)
      at com.sonatype.nexus.clm.internal.FirewallTransactionQueue.processTransaction(FirewallTransactionQueue.java:155)
      at com.sonatype.nexus.clm.internal.FirewallTransactionQueueTransactionTask.run(FirewallTransactionQueueTransactionTask.java:51)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2282 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
      DB name="component"
      at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
      at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591)
      at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884)
      at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607)
      at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603)
      at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98)
      at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88)
      at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359)
      at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bradbeck Brad Beck
              Reporter:
              drollo Dan Rollo
              Last Updated By:
              Peter Lynch
              Team:
              NXRM - Morpheus
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title