Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 3.15.2
-
Fix Version/s: 3.16.0
-
Component/s: HA, Repository
-
Labels:None
Description
To recreate:
- Setup a 3-node NXRM cluster with NPM proxy repository
- Optional: add a separate NXRM instance between the cluster and https://registry.npmjs.org/ to reduce network effects (and avoid extra downloads when repeating the test)
- Either setup a load-balancer and configure the NPM clients to use that, or configure each NPM client to use a different node
- Send parallel requests to install the npm-bomb package from each NPM client - make sure they each have a different working directory
You should see the NPM clients stall and the nexus logs fill with fix messages (from repairing/rolling back distributed transactions) as well as other exceptions and messages indicating ongoing retries and conflicts.
If you take a thread dump you should see at least one thread is repeatedly waiting for the distributed results of updating the last downloaded time via LastDownloadedHandler
"qtp1481853177-291" #291 prio=5 os_prio=31 tid=0x00007f9337170800 nid=0x19d03 waiting on condition [0x000070000bc68000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007b23ccab0> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at com.orientechnologies.orient.server.distributed.ODistributedResponseManager.waitForSynchronousResponses(ODistributedResponseManager.java:282) at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.waitForResponse(ODistributedDatabaseImpl.java:1067) at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:517) at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:589) at com.orientechnologies.orient.server.distributed.impl.OConflictResolverDatabaseRepairer.repairRecords(OConflictResolverDatabaseRepairer.java:662) at com.orientechnologies.orient.server.distributed.impl.OConflictResolverDatabaseRepairer.repairRecord(OConflictResolverDatabaseRepairer.java:409) at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.processCommitResult(ODistributedTransactionManager.java:774) at com.orientechnologies.orient.server.distributed.impl.ODistributedTransactionManager.commit(ODistributedTransactionManager.java:191) at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.commit(ODistributedStorage.java:1426) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2908) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2870) at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:184) at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193) at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66) at com.sun.proxy.$Proxy199.commit(Unknown Source) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.call(Operations.java:146) at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded(LastDownloadedHandler.java:116) at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.maybeUpdateLastDownloaded(LastDownloadedHandler.java:89) at org.sonatype.nexus.repository.view.handlers.LastDownloadedHandler.handle(LastDownloadedHandler.java:74) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source) at org.sonatype.nexus.repository.npm.internal.NpmProxyRecipe$_closure1.doCall(NpmProxyRecipe.groovy:189)
