Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-22666

concurrent uploads to the same maven GA may result in 500 response due to OConcurrentModificationException

    XMLWordPrintable

    Details

    • Notability:
      2

      Description

      The jmeter-reproduce-case.zip makes concurrent uploads to a Maven 2 hosted releases repo with Deployment policy Allow redeploy at the same GA at multiple versions and same GAV with multiple classified artifacts. This results in 500 status responses for some of the uploads due to concurrency issues rebuilding metadata.

      Exception #1
      2020-02-03 15:38:40,272-0400 WARN  [qtp751364107-247] admin org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID e363d1db-7718-4f26-b582-7932cc986f31) Unexpected exception: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #71:904: found duplicated key 'OCompositeKey{keys=[#52:0, null, com/example/artifact/maven-metadata.xml]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #69:906
      	DB name="component" INDEX=asset_bucket_component_name_idx RID=#69:906
      com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #71:904: found duplicated key 'OCompositeKey{keys=[#52:0, null, com/example/artifact/maven-metadata.xml]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #69:906
      	DB name="component"
      	at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
      	at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
      	at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
      	at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
      	at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2488)
      	at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
      	at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.putInSnapshot(OIndexAbstract.java:956)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.applyIndexTxEntry(OIndexAbstract.java:790)
      	at com.orientechnologies.orient.core.index.OIndexAbstract.addTxOperation(OIndexAbstract.java:762)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitIndexes(OAbstractPaginatedStorage.java:1904)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1861)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:179)
      	at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      	at com.sun.proxy.$Proxy220.commit(Unknown Source)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:68)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:125)
      	at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.updateMetadata(MavenUploadHandler.java:297)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.doUpload(MavenUploadHandler.java:181)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.handle(MavenUploadHandler.java:139)
      	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:98)
      	at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.uploadComponent(ComponentsResource.java:247)
      	at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source)
      
      
      Exception #2
      2020-02-03 15:38:40,315-0400 WARN  [qtp751364107-199] admin org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID c4e85003-0ae1-47e5-b400-eecfeb9e9ec9) Unexpected exception: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #69:906 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
      	DB name="component"
      com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #69:906 because the version is not the latest. Probably you are updating an old record or it has been modified by another user (db=v2 your=v1)
      	DB name="component"
      	at org.sonatype.nexus.orient.entity.ConflictHook.onUpdate(ConflictHook.java:154)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4545)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:4022)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4607)
      	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1857)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:179)
      	at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      	at com.sun.proxy.$Proxy220.commit(Unknown Source)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:68)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:51)
      	at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:125)
      	at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.updateMetadata(MavenUploadHandler.java:297)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.doUpload(MavenUploadHandler.java:181)
      	at org.sonatype.nexus.repository.maven.MavenUploadHandler.handle(MavenUploadHandler.java:139)
      	at org.sonatype.nexus.repository.upload.internal.UploadManagerImpl.handle(UploadManagerImpl.java:98)
      	at org.sonatype.nexus.repository.rest.internal.resources.ComponentsResource.uploadComponent(ComponentsResource.java:247)
      	at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498) 

      Regression

      In 3.19.1, the uploads implemented an internal retry mechanism on an asynchronous thread, which helped both recover gracefully from the db record conflicts and hide these issues from the end user client doing the uploads. See [^sonatype-work-3.19.1.zip] logs where there were no failures, but retries.

      In 3.20.1, now when there is a DB conflict, the upload immediately fails with a 500 response - which in turn causes builds to fail. See [^sonatype-work-3.20.1.zip]

      Expected

      Concurrent uploads to the same GA or GAV, which require NXRM to eventually rebuild or make available for a future request, the associated GA level maven-metadata.xml, should not cause a non-maven-metadata upload to fail.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jhill Joshua Hill
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Wes Wannemacher
              Team:
              NXRM - Operations/Groot
              Votes:
              2 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title