Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19642

Snapshot removal task runs very slowly when there are large numbers of group repos

    XMLWordPrintable

    Details

      Description

      A user was observed who had 357 maven group repositories.  This is causing the snapshot remover to run very slowly for them when it is rebuilding maven-metadata.xml files.

       

      2019-04-10 10:06:14,769-0400 INFO [quartz-2-thread-15] *SYSTEM org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Elapsed time: 17.00 h, updated metadata for 1724 GAVs
      2019-04-10 10:07:17,054-0400 INFO [quartz-2-thread-15] *SYSTEM org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Elapsed time: 17.02 h, updated metadata for 1726 GAVs
      2019-04-10 10:08:35,058-0400 INFO [quartz-2-thread-15] *SYSTEM org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Elapsed time: 17.04 h, updated metadata for 1729 GAVs

      As can be seen above, it's only processing about 100 GAV's an hour.

      The cause of this is excessive calls to query to find the containing group repositories.

      In the thread dump, there were 21 threads blocked trying to get thread pool threads to execute elasticsearch queries:

      elasticsearch[0ACADDEF-558108CA-57530157-84BF626E-9654F0BF][search]T#2 id=4454 state=BLOCKED

      • waiting to lock <0x322485c4> (a java.lang.reflect.Method)
        owned by elasticsearch[0ACADDEF-558108CA-57530157-84BF626E-9654F0BF][search]T#3 id=4455
        at java.lang.reflect.Executable.declaredAnnotations(Executable.java:594)
        at java.lang.reflect.Executable.getAnnotation(Executable.java:570)
        at java.lang.reflect.Method.getAnnotation(Method.java:622)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:46)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
        at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$549439df.member(<generated>)
        at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.lambda$2(RepositoryManagerImpl.java:410)
        at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl$$Lambda$445/1808499580.test(Unknown Source)
        at java.util.Optional.filter(Optional.java:178)
        at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.findContainingGroups(RepositoryManagerImpl.java:410)

      And 5 RUNNABLE doing the same query:

      elasticsearch[0ACADDEF-558108CA-57530157-84BF626E-9654F0BF][search]T#3 id=4455 state=RUNNABLE
      at java.lang.reflect.Executable.declaredAnnotations(Executable.java:594)
      at java.lang.reflect.Executable.getAnnotation(Executable.java:570)
      at java.lang.reflect.Method.getAnnotation(Method.java:622)
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:46)
      at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet$$EnhancerByGuice$$549439df.member(<generated>)
      at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.lambda$2(RepositoryManagerImpl.java:410)
      at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl$$Lambda$445/1808499580.test(Unknown Source)
      at java.util.Optional.filter(Optional.java:178)
      at org.sonatype.nexus.repository.manager.internal.RepositoryManagerImpl.findContainingGroups(RepositoryManagerImpl.java:410)

      It seems to me that we don't need to keep making this query over and over again, we should cache it. Bonus points for clearing the cache if there are changes to the repository structure while the task is running.

       

       

       

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            rseddon Rich Seddon
            Last Updated By:
            Peter Lynch Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title