Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-35509

Improve search performance with large Roles and Privileges - OrientDB

    Details

    • Notability:
      3
    • InvestmentLayer:
      support-escalated
    • Aha Concept:
      non-concept

      Description

      Since upgrading to Nexus 3.39 running OrientDB, an increase in search times has been observed due to permission/role checks.

      For instance with over 8k roles, a UI keyword search on 3.34 completed in ~5 seconds:

      [22/Sep/2022:17:25:28 +0000] "POST /service/extdirect HTTP/1.1" 200 154 49910 4900 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" [qtp71312966-1598]
      
      

      The same search with the same user on 3.39 exceeded 60 seconds:

      [22/Sep/2022:14:57:15 +0000] "POST /service/extdirect HTTP/1.1" 200 154 163 62268 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36" [qtp1268820127-1177]

      And most of the time was spent on a request to get all roles from the Security DB e.g.

      "elasticsearch[F245477B-8DE91CB5-855A498B-819478E7-FF215E09][search][T#45]" #1169 daemon prio=5 os_prio=0 tid=0x00007f20100a6000 nid=0x3d64 runnable [0x00007f1b22c9a000]
      java.lang.Thread.State: RUNNABLE
      at java.util.WeakHashMap.get(WeakHashMap.java:403)
      at com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs.get(ORecordCacheWeakRefs.java:44)
      at com.orientechnologies.orient.core.cache.OLocalRecordCache.findRecord(OLocalRecordCache.java:80)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:1963)
      at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:187)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1757)
      at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
      at com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:300)
      at com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:167)
      at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
      at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635)
      at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
      at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
      at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:466)
      at com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:854)
      at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:276)
      at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:233)
      at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration.lambda$13(OrientSecurityConfigurationSource.java:388)
      at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration$$Lambda$2044/1448960950.apply(Unknown Source)
      at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$1(OrientOperations.java:55)
      at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$1141/1726670245.call(Unknown Source)
      at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
      at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
      at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
      at org.sonatype.nexus.transaction.Operations.call(Operations.java:166)
      at org.sonatype.nexus.orient.transaction.OrientOperations.call(OrientOperations.java:55)
      at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration.getRoles(OrientSecurityConfigurationSource.java:388)
      at org.sonatype.nexus.security.internal.SecurityConfigurationManagerImpl.listRoles(SecurityConfigurationManagerImpl.java:113)
      at org.sonatype.nexus.security.internal.AuthorizationManagerImpl.listRoles(AuthorizationManagerImpl.java:188)
      at org.sonatype.nexus.security.internal.DefaultSecuritySystem.listRoles(DefaultSecuritySystem.java:174)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl.getRoles(SelectorManagerImpl.java:334)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl.browseActive(SelectorManagerImpl.java:270)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl$$EnhancerByGuice$$1122148912.GUICE$TRAMPOLINE(<generated>)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl$$EnhancerByGuice$$1122148912$$Lambda$716/1866695187.apply(Unknown Source)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
      at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl$$EnhancerByGuice$$1122148912.browseActive(<generated>)
      at org.sonatype.nexus.repository.security.internal.ContentPermissionCheckerImpl.isPermitted(ContentPermissionCheckerImpl.java:260)
      at org.sonatype.nexus.repository.search.selector.ContentAuthPluginScript.run(ContentAuthPluginScript.java:95)
      at org.elasticsearch.index.query.ScriptQueryParser$ScriptQuery$1$1.get(ScriptQueryParser.java:171)
      at org.apache.lucene.search.RandomAccessWeight$1.matches(RandomAccessWeight.java:63)
      at org.elasticsearch.common.lucene.Lucene$3.get(Lucene.java:765)
      at org.elasticsearch.common.lucene.search.FilteredCollector$1.collect(FilteredCollector.java:54)
      at org.apache.lucene.search.TimeLimitingCollector$1.collect(TimeLimitingCollector.java:158)
      at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:221)
      at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:172)
      at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:821)
      at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:535)
      at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:384)
      at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
      at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:372)
      at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:385)
      at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
      at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)

       The following calls in the above stacktrace:

      at org.sonatype.nexus.internal.selector.SelectorManagerImpl.getRoles(SelectorManagerImpl.java:334)
      at org.sonatype.nexus.internal.selector.SelectorManagerImpl.browseActive(SelectorManagerImpl.java:270)

      Were both added in 3.39 -  the first was added via https://issues.sonatype.org/browse/NEXUS-31501 and the second was added via https://issues.sonatype.org/browse/NEXUS-30997

      This request to get all roles on a busy instance will also cause other threads to wait their turn to getRoles e.g. 

      qtp288042502-3398&quot; #3398 prio=5 os_prio=0 tid=0x00007fde88058800 nid=0x77ea waiting on condition [0x00007fcf284d5000]
      java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method) - parking to wait for  &lt;0x00007fdf8fdf3878&gt; (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLock(OPartitionedLockManager.java:213) at com.orientechnologies.common.concur.lock.OPartitionedLockManager.acquireExclusiveLocksInBatch(OPartitionedLockManager.java:266) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:355) at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:145) at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPositionMap.get(OClusterPositionMap.java:435) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:734) at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:721) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4220) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3807) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1410) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3395) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2008) at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:187) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1757) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103) at com.orientechnologies.orient.core.iterator.OIdentifiableIterator.readCurrentRecord(OIdentifiableIterator.java:300) at com.orientechnologies.orient.core.iterator.ORecordIteratorClusters.hasNext(ORecordIteratorClusters.java:167) at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42) at com.google.common.collect.Iterators$5.computeNext(Iterators.java:635) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:466) at com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:854) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:276) at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:233) at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration.lambda$13(OrientSecurityConfigurationSource.java:388) at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration$$Lambda$2181/769546930.apply(Unknown Source) at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$1(OrientOperations.java:55) at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$1104/71289133.call(Unknown Source) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58) at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223) at org.sonatype.nexus.transaction.Operations.call(Operations.java:166) at org.sonatype.nexus.orient.transaction.OrientOperations.call(OrientOperations.java:55) at org.sonatype.nexus.internal.security.model.orient.OrientSecurityConfigurationSource$OrientSecurityConfiguration.getRoles(OrientSecurityConfigurationSource.java:388)
      

       

      EXPECTED:

      Improve role querying to handle large roles efficiently and improve nexus response time.

       

       

       

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              hardeepn Hardeep Nagra
              Last Updated By:
              Peter Lynch Peter Lynch
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title