Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-17759

Maven metadata rebuild task runs extremely slow for large datasets

    XMLWordPrintable

    Details

    • Release Note:
      Yes

      Description

      Running the Maven metadata rebuild task on a large dataset (1.2m+ components) will take over 60 days to complete.

      • The task slows down exponentially as more components are added.
      • Adding additional repositories containing maven components with the groups and artificers that existing in the repository being re-generated also slows down the generation.

      Example thread stack from 3.10.0 related to this problem:

      quartz-5-thread-17 <command>sql.select from component where (group = :groupId and name = :artifactId and attributes.maven2.baseVersion = :baseVersion) and (bucket=#17:44)</command> id=162 state=RUNNABLE
          at java.util.concurrent.ConcurrentHashMap$TreeNode.findTreeNode(ConcurrentHashMap.java:2703)
          at java.util.concurrent.ConcurrentHashMap$TreeNode.findTreeNode(ConcurrentHashMap.java:2699)
          at java.util.concurrent.ConcurrentHashMap$TreeNode.findTreeNode(ConcurrentHashMap.java:2699)
          at java.util.concurrent.ConcurrentHashMap$TreeBin.putTreeVal(ConcurrentHashMap.java:2893)
          at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1052)
          at java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.ConcurrentLRUList.putToMRU(ConcurrentLRUList.java:101)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.entryIsInAmQueue(O2QCache.java:1091)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.updateCache(O2QCache.java:1147)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:349)
          at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.load(O2QCache.java:294)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.loadPage(ODurableComponent.java:148)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:713)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:689)
          at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:668)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:4203)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3819)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1409)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3411)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2022)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:660)
          at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:103)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1638)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:211)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
          at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3417)
          at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3335)
          at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
          at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.browseByQuery(MetadataNodeEntityAdapter.java:178)
          at org.sonatype.nexus.repository.storage.StorageTxImpl.findComponents(StorageTxImpl.java:431)
          at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
          at com.sun.proxy.$Proxy213.findComponents(Unknown Source)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$2(MetadataRebuilder.java:341)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker$$Lambda$358/867278903.call(Unknown Source)
          at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
          at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
          at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
          at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:340)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:307)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:115)
          at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:168)
          at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:122)
          at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl.removeSnapshots(RemoveSnapshotsFacetImpl.java:141)
          at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl$$EnhancerByGuice$$1eec7e88.CGLIB$removeSnapshots$1(<generated>)
          at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl$$EnhancerByGuice$$1eec7e88$$FastClassByGuice$$30d590a2.invoke(<generated>)
          at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
          at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
          at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl$$EnhancerByGuice$$1eec7e88.removeSnapshots(<generated>)
          at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:72)
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            jstephens Joseph Stephens
            Last Updated By:
            Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title