Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-13092

slow performance and metadata rebuild failures when running "Remove snapshots from Maven repository" against large datasets

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Release Note:
      Yes
    • Sprint:
      Platform Team - Sprint 96, Platform Team - Sprint 97, Platform - Sprint 98

      Description

      Looking at the logs we can see that the task "Deleted 576 components from 522 distinct GAVs", 13 minutes after task 'Remove2 old snapshot from Amundi Snapshots' has started.

      2017-05-03 01:45:00,013+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Remove2 old snapshot from Amundi Snapshots' [repository.maven.remove-snapshots] state change WAITING -> RUNNING
      2017-05-03 01:45:00,013+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask - Executing removal of snapshots on repository: RepositoryImpl$$EnhancerByGuice$$25533b5f{type=hosted, format=maven2, name='amundi-snapshots'}
      2017-05-03 01:45:00,013+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Removing snapshots on repository amundi-snapshots with configuration: org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsConfig(3, 15, true, 7)
      2017-05-03 01:45:02,104+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Processing 1856 GAVs found with more than minimum 3 snapshot versions
      2017-05-03 01:58:01,994+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Finished processing snapshots with more than 3 versions created before 2017-04-18T01:45:00.013+02:00
      2017-05-03 01:58:01,994+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Deleted 576 components from 522 distinct GAVs
      

      Then there is a 4+ hours gap before a next log entry on 42 GAVs found that have already been released.

      2017-05-03 06:28:29,238+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Processing 42 GAVs found that have already been released and for which the grace period has elapsed
      2017-05-03 06:39:39,086+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Finished processing snapshots with associated releases
      

      Rebuild of metadata has now started.

      2017-05-03 06:39:39,086+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl - Updating metadata on repository: amundi-snapshots
      

      After 9 hours of rebuilding metadata you get the following failure.

      2017-05-03 15:40:43,938+0200 ERROR [quartz-6-thread-18] *TASK org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask - Failed to run task 'Remove Maven snapshots from amundi-snapshots' on repository 'amundi-snapshots'
      java.lang.NullPointerException: null
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:271) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:102) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:152) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:122) [na:na]
      at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl.removeSnapshots(RemoveSnapshotsFacetImpl.java:149) [na:na]
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
      at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
      at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:58) [na:na]
      at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:56) [na:na]
      at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:89) [org.sonatype.nexus.scheduling:3.2.0.01]
      at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:144) [org.sonatype.nexus.quartz:3.2.0.01]
      at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:105) [org.sonatype.nexus.quartz:3.2.0.01]
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.quartz-scheduler.quartz:2.2.2]
      at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) [org.sonatype.nexus.thread:3.2.0.01]
      at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [org.apache.shiro.core:1.3.2]
      at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [org.apache.shiro.core:1.3.2]
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_102]
      at java.lang.Thread.run(Unknown Source) [na:1.8.0_102]
      2017-05-03 15:40:43,938+0200 WARN [quartz-6-thread-18] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 62767367-9306-4056-a82f-e831ddd5a261 : 'Remove2 old snapshot from Amundi Snapshots' [repository.maven.remove-snapshots] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Remove Maven snapshots from amundi-snapshots'; 1 failure
      at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95) [org.sonatype.goodies.common:2.2.3]
      at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:64) [na:na]
      at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:89) [org.sonatype.nexus.scheduling:3.2.0.01]
      at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:144) [org.sonatype.nexus.quartz:3.2.0.01]
      at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:105) [org.sonatype.nexus.quartz:3.2.0.01]
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.quartz-scheduler.quartz:2.2.2]
      at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) [org.sonatype.nexus.thread:3.2.0.01]
      at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) [org.apache.shiro.core:1.3.2]
      at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) [org.apache.shiro.core:1.3.2]
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_102]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_102]
      at java.lang.Thread.run(Unknown Source) [na:1.8.0_102]
      Suppressed: java.lang.NullPointerException: null
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:271) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:102) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.deleteAndRebuild(MetadataRebuilder.java:152) [na:na]
      at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.deleteMetadata(MavenHostedFacetImpl.java:122) [na:na]
      at org.sonatype.nexus.repository.maven.internal.RemoveSnapshotsFacetImpl.removeSnapshots(RemoveSnapshotsFacetImpl.java:149) [na:na]
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
      at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
      at org.sonatype.nexus.repository.maven.tasks.RemoveSnapshotsTask.execute(RemoveSnapshotsTask.java:58) [na:na]
      at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:56) [na:na]
      ... 12 common frames omitted
      2017-05-03 15:40:43,938+0200 INFO [quartz-6-thread-18] *TASK org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Remove2 old snapshot from Amundi Snapshots' [repository.maven.remove-snapshots] state change RUNNING -> WAITING (FAILED)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              nblair Nicholas Blair
              Reporter:
              msurani Mahendra Surani
              CC:
              Antoine Body
              Last Updated By:
              Peter Lynch
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title