Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 3.10.0, 3.12.1, 3.13.0
-
Fix Version/s: 3.16.0
-
Labels:
-
Release Note:Yes
-
Epic Link:
Description
OConcurrentModificationException ERROR message in Nexus application log (and Firewall.Audit task log) in an HA-C cluster.
When Nexus Firewall is enabled in an HA cluster, bulk component checks to IQ server attempt to update all the given assets in the same transaction, which could be a large number.
If just one of those assets has been concurrently modified by another thread since the transaction started then it will start over from scratch.
When failure happens, the quarantine status for the affected components may be prevented from being served through repository manager as their quarantine status will change the PENDING. The amount of time it may take for the affected components to leave PENDING status is indeterminate.
Example log messages that may be seen:
2018-04-12 15:29:17,315-0400 ERROR [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Failed to run task 'Auditing components in repository npmjs' org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8 at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224) at sun.reflect.GeneratedMethodAccessor946.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69) at com.sun.proxy.$Proxy197.allowRetry(Unknown Source) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.call(Operations.java:146) at com.sonatype.nexus.clm.internal.EvaluationSender.sendAuditRequest(EvaluationSender.java:102) at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:88) at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:130) at com.sonatype.nexus.clm.internal.RepositoryAuditExecutor$EvaluationTask.call(RepositoryAuditExecutor.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2002 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v3 your=v2) DB name="component" at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56) at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856) at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884) at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132) at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607) at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70) at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603) at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98) at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88) at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98) at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359) at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127) 2018-04-12 15:29:17,315-0400 INFO [quartz-6-thread-20] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.RepositoryAuditTask - Task complete
2018-04-17 16:23:45,165-0400 ERROR [fte-4-thread-13] NODE1 *SYSTEM com.sonatype.nexus.clm.internal.FirewallTransactionQueue - Could not send update to IQ Server: EVALUATE_COMPONENT npmjs:sinon-chai/-/sinon-chai-2.14.0.tgz. Error was: Reached max retries: 8/8 org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8 at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:224) at sun.reflect.GeneratedMethodAccessor179.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69) at com.sun.proxy.$Proxy196.allowRetry(Unknown Source) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.call(Operations.java:146) at com.sonatype.nexus.clm.internal.EvaluationSender.sendQuarantineRequest(EvaluationSender.java:129) at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:85) at com.sonatype.nexus.clm.internal.EvaluationSender.sendEvaluationRequest(EvaluationSender.java:64) at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.evaluateComponent(FirewallTransactionProcessor.java:163) at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransactionWithoutErrorHandler(FirewallTransactionProcessor.java:110) at com.sonatype.nexus.clm.internal.FirewallTransactionProcessor.processTransaction(FirewallTransactionProcessor.java:83) at com.sonatype.nexus.clm.internal.FirewallTransactionQueue.processTransaction(FirewallTransactionQueue.java:155) at com.sonatype.nexus.clm.internal.FirewallTransactionQueueTransactionTask.run(FirewallTransactionQueueTransactionTask.java:51) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #19:2282 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1) DB name="component" at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56) at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4529) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4005) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4591) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1856) at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1379) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2922) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884) at com.orientechnologies.orient.server.distributed.impl.task.OTxTask.execute(OTxTask.java:132) at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin$1.call(ODistributedAbstractPlugin.java:607) at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70) at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.executeOnLocalNode(ODistributedAbstractPlugin.java:603) at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.lambda$0(SharedHazelcastPlugin.java:98) at org.sonatype.nexus.orient.entity.EntityHook.asRemote(EntityHook.java:88) at com.sonatype.nexus.hazelcast.internal.orient.SharedHazelcastPlugin.executeOnLocalNode(SharedHazelcastPlugin.java:98) at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.onMessage(ODistributedWorker.java:359) at com.orientechnologies.orient.server.distributed.impl.ODistributedWorker.run(ODistributedWorker.java:127)
Attachments
Issue Links
- depends on
-
NEXUS-13462 Extend proxy cooperation across HA nodes
-
- Closed
-
- discovered while testing
-
NEXUS-17628 Asset stuck in quarantined state if HA exception occurs during download when firewall enabled
-
- Closed
-
- is related to
-
NEXUS-16310 HA - errors when pushing nuget packages
-
- Closed
-
- mentioned in
-
Page Loading...