Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19229

LastDownloadedHandler conflicts in HA can cause long-running retries/rollbacks

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.15.2
    • Fix Version/s: 3.16.0
    • Component/s: HA, Repository
    • Labels:
      None

      Description

      To recreate:

      • Setup a 3-node NXRM cluster with NPM proxy repository
      • Optional: add a separate NXRM instance between the cluster and https://registry.npmjs.org/ to reduce network effects (and avoid extra downloads when repeating the test)
      • Either setup a load-balancer and configure the NPM clients to use that, or configure each NPM client to use a different node
      • Send parallel requests to install the npm-bomb package from each NPM client - make sure they each have a different working directory

      You should see the NPM clients stall and the nexus logs fill with fix messages (from repairing/rolling back distributed transactions) as well as other exceptions and messages indicating ongoing retries and conflicts.

      If you take a thread dump you should see at least one thread is repeatedly waiting for the distributed results of updating the last downloaded time via LastDownloadedHandler

      "qtp1481853177-291" #291 prio=5 os_prio=31 tid=0x00007f9337170800 nid=0x19d03 waiting on condition [0x000070000bc68000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000007b23ccab0> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
      	at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.waitForSynchronousResponses(ODistributedResponseManager.java:282)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.waitForResponse(ODistributedDatabaseImpl.java:1067)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:517)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:589)
      	at com.orientechnologies.orient.server.distributed.impl.OConflictResolverDatabaseRepairer.repairRecords(OConflictResolverDatabaseRepairer.java:662)
      	at com.orientechnologies.orient.server.distributed.impl.OConflictResolverDatabaseRepairer.repairRecord(OConflictResolverDatabaseRepairer.java:409)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.processCommitResult(ODistributedTransactionManager.java:774)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.commit(ODistributedTransactionManager.java:191)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1426)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
      	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870)
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:184)
      	at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
      	at com.sun.proxy.$Proxy199.commit(Unknown Source)
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
      	at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      	at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
      	at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded(LastDownloadedHandler.java:116)
      	at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded(LastDownloadedHandler.java:89)
      	at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.handle(LastDownloadedHandler.java:74)
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
      	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
      	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
      	at org.sonatype.nexus.repository.npm.internal.NpmProxyRecipe$_closure1.doCall(NpmProxyRecipe.groovy:189)
      

        Attachments

          Activity

            People

            Assignee:
            mbucher Michael Bucher
            Reporter:
            mcculls Stuart McCulloch
            Last Updated By:
            Peter Lynch
            Team:
            NXRM - Morpheus
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title