Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16852

HA-C with Nexus Firewall evaluations may fail due to OConcurrentModificationException Cannot UPDATE the record because the version is not the latest

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.10.0, 3.12.1, 3.13.0
    • Fix Version/s: 3.16.0
    • Component/s: HA

      Description

      OConcurrentModificationException ERROR message in Nexus application log (and Firewall.Audit task log) in an HA-C cluster.

      When Nexus Firewall is enabled in an HA cluster, bulk component checks to IQ server attempt to  update all the given assets in the same transaction, which could be a large number.

      If just one of those assets has been concurrently modified by another thread since the transaction started then it will start over from scratch.

      When failure happens, the quarantine status for the affected components may be prevented from being served through repository manager as their quarantine status will change the PENDING. The amount of time it may take for the affected components to leave PENDING status is indeterminate.

      Example log messages that may be seen:

      Example 1
      2018-04-12 15:29:17,315-0400 ERROR [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Failed to run task 'Auditing components in repository npmjs'
org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224)
at sun.reflect.GeneratedMethodAccessor946.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy197.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at com.sonatype.nexus.clm.internal.EvaluationSender.sendAuditRequest(EvaluationSender.java:102)
at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:88)
at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:130)
at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2002 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v3 your=v2)
DB name="component"
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884)
at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603)
at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98)
at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88)
at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98)
at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359)
at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127)
2018-04-12 15:29:17,315-0400 INFO [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Task complete
      Example 2
      2018-04-17 16:23:45,165-0400 ERROR [fte-4-thread-13] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.FirewallTransactionQueue - Could not send update to IQ Server: EVALUATE_COMPONENT npmjs:sinon-chai/-/sinon-chai-2.14.0.tgz. Error was: Reached max retries: 8/8
org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224)
at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy196.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at com.sonatype.nexus.clm.internal.EvaluationSender.sendQuarantineRequest(EvaluationSender.java:129)
at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:85)
at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:64)
at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.evaluateComponent(FirewallTransactionProcessor.java:163)
at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransactionWithoutErrorHandler(FirewallTransactionProcessor.java:110)
at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransaction(FirewallTransactionProcessor.java:83)
at com.sonatype.nexus.clm.internal.FirewallTransactionQueue.processTransaction(FirewallTransactionQueue.java:155)
at com.sonatype.nexus.clm.internal.FirewallTransactionQueueTransactionTask.run(FirewallTransactionQueueTransactionTask.java:51)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2282 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
DB name="component"
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884)
at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603)
at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98)
at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88)
at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98)
at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359)
at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127)

        Attachments

          Issue Links

          depends on

          Improvement - An improvement or enhancement to an existing feature or task. NEXUS-13462 Extend proxy cooperation across HA nodes

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          discovered while testing

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-17628 Asset stuck in quarantined state if HA exception occurs during download when firewall enabled

          • Major - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-16310 HA - errors when pushing nuget packages

          • Medium - Medium
          • Closed
          mentioned in

          Page Loading...

            Activity

              People

              • Assignee:
                bradbeck Brad Beck
                Reporter:
                drollo Dan Rollo
                Last Updated By:
                Michael Bucher
                Team:
                NXRM - Morpheus
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Date of First Response: