Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-22666

concurrent uploads to the same maven GA may result in 500 response due to OConcurrentModificationException

Details

    • 2

    Description

      The jmeter-reproduce-case.zip makes concurrent uploads to a Maven 2 hosted releases repo with Deployment policy Allow redeploy at the same GA at multiple versions and same GAV with multiple classified artifacts. This results in 500 status responses for some of the uploads due to concurrency issues rebuilding metadata.

      Exception #1
      2020-02-03 15:38:40,272-0400 WARN  [qtp751364107-247] admin org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID e363d1db-7718-4f26-b582-7932cc986f31) Unexpected exception: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #71:904: found duplicated key 'OCompositeKey{keys=[#52:0, null, com/example/artifact/maven-metadata.xml]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #69:906
      	DB name="component" INDEX=asset_bucket_component_name_idx RID=#69:906
      com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #71:904: found duplicated key 'OCompositeKey{keys=[#52:0, null, com/example/artifact/maven-metadata.xml]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #69:906
      	DB name="component"
      	at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
      	at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
      	at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
      	at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
      	at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2488)
      	at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
      	at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.putInSnapshot(OIndexAbstract.java:956)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.applyIndexTxEntry(OIndexAbstract.java:790)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.addTxOperation(OIndexAbstract.java:762)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitIndexes(OAbstractPaginatedStorage.java:1904)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1861)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:179)
      	at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      	at com.sun.proxy.$Proxy220.commit(Unknown Source)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:68)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:125)
      	at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.updateMetadata(MavenUploadHandler.java:297)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.doUpload(MavenUploadHandler.java:181)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.handle(MavenUploadHandler.java:139)
      	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:98)
      	at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.uploadComponent(ComponentsResource.java:247)
      	at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source)
      
      
      Exception #2
      2020-02-03 15:38:40,315-0400 WARN  [qtp751364107-199] admin org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID c4e85003-0ae1-47e5-b400-eecfeb9e9ec9) Unexpected exception: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #69:906 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
      	DB name="component"
      com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #69:906 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
      	DB name="component"
      	at org.sonatype.nexus.orient.entity.ConflictHook.onUpdate(ConflictHook.java:154)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4545)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4022)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4607)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1857)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:179)
      	at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      	at com.sun.proxy.$Proxy220.commit(Unknown Source)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:68)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:125)
      	at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.updateMetadata(MavenUploadHandler.java:297)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.doUpload(MavenUploadHandler.java:181)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.handle(MavenUploadHandler.java:139)
      	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:98)
      	at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.uploadComponent(ComponentsResource.java:247)
      	at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498) 

      Regression

      In 3.19.1, the uploads implemented an internal retry mechanism on an asynchronous thread, which helped both recover gracefully from the db record conflicts and hide these issues from the end user client doing the uploads. See [^sonatype-work-3.19.1.zip] logs where there were no failures, but retries.

      In 3.20.1, now when there is a DB conflict, the upload immediately fails with a 500 response - which in turn causes builds to fail. See [^sonatype-work-3.20.1.zip]

      Expected

      Concurrent uploads to the same GA or GAV, which require NXRM to eventually rebuild or make available for a future request, the associated GA level maven-metadata.xml, should not cause a non-maven-metadata upload to fail.

      Attachments

        Issue Links

          Activity

            People

              jhill Joshua Hill
              plynch Peter Lynch
              Michael Prescott Michael Prescott
              NXRM - Groot
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                tigCommentSecurity.panel-title