[NEXUS-4640] Very slow performance of snapshot remover Created: 11/17/11  Updated: 12/08/11  Resolved: 12/08/11

Status: Closed
Project: Dev - Nexus
Component/s: Scheduled Tasks
Affects Version/s: 1.9.2.3
Fix Version/s: 2.0

Type: Bug Priority: Major
Reporter: Rich Seddon Assignee: Peter Lynch
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File Launcher-snapshots-2011-12-07-14-48.snapshot     Zip Archive snapshot-repo.zip    
Issue Links:
Duplicate
Relates
relates NEXUS-4682 Decouple shadow repository actions fr... Closed
Last Updated By: Rich Seddon
Date of First Response:
Story Points: 15

 Description   

I've been getting quite a few reports of snapshot removal tasks taking a very long time to run (over 24 hours in some cases). On RSO the weekly snapshot removal job takes around 4 hours.

This is far too slow. We need to do some profiling, figure out why this takes so long, and see if we can implement some improvements.



 Comments   
Comment by Rich Seddon [ 11/21/11 ]

There may be a memory issue in play here, see linked issues.

Comment by Brian Demers [ 11/28/11 ]

Do some profiling to figure out what is slow

Comment by Tamás Cservenák [ 12/05/11 ]

Original fixes merged in as:
https://github.com/sonatype/nexus/pull/115

But it should improve more (with default config), or a Lot (in case of custom config with multiple shadows) after this is merged in too:
https://github.com/sonatype/nexus/pull/119

Comment by Peter Lynch [ 12/06/11 ]

At Tamas's request re-opening and put back to in his hands.

I created a hosted repo for SNAPSHOTS, 100 groups x 100 attribute ids * randomly dispersed versions for a total of approx. 13000 artifacts. maven metadata was not created initially before task ran.

I ran snapshot remover task against this. These are the logs:

jvm 1    | 2011-12-06 10:52:35 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, not yet finished.
jvm 1    | 2011-12-06 10:52:35 INFO  [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository.
jvm 1    | 2011-12-06 10:54:29 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Expiring local cache in repository ID='repo1' from path='/'
jvm 1    | 2011-12-06 10:55:25 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/'
jvm 1    | 2011-12-06 11:15:24 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, finished successfully on Tue Dec 06 11:15:23 NST 2011

As can be seen creating the metadata looks like the bulk of the overhead.

Comment by Peter Lynch [ 12/06/11 ]

attaching raw snapshot repo I used ( manually put in storage dir and configured nexus.xml with

<repository>
      <id>repo1</id>
      <name>repo1</name>
      <providerRole>org.sonatype.nexus.proxy.repository.Repository</providerRole>
      <providerHint>maven2</providerHint>
      <localStatus>IN_SERVICE</localStatus>
      <notFoundCacheActive>true</notFoundCacheActive>
      <notFoundCacheTTL>1440</notFoundCacheTTL>
      <userManaged>true</userManaged>
      <exposed>true</exposed>
      <browseable>true</browseable>
      <writePolicy>ALLOW_WRITE</writePolicy>
      <localStorage>
        <provider>file</provider>
      </localStorage>
      <externalConfiguration>
        <repositoryPolicy>SNAPSHOT</repositoryPolicy>
      </externalConfiguration>
    </repository>
<task>
      <id>1</id>
      <name>repo1-test</name>
      <enabled>true</enabled>
      <type>SnapshotRemoverTask</type>
      <status>SUBMITTED</status>
      <schedule>
        <type>manual</type>
      </schedule>
      <properties>
        <property>
          <key>removeOlderThanDays</key>
          <value></value>
        </property>
        <property>
          <key>minSnapshotsToKeep</key>
          <value>0</value>
        </property>
        <property>
          <key>removeIfReleaseExists</key>
          <value>false</value>
        </property>
        <property>
          <key>.name</key>
          <value>repo1-test</value>
        </property>
        <property>
          <key>repositoryId</key>
          <value>repo1</value>
        </property>
      </properties>
    </task>

Comment by Peter Lynch [ 12/07/11 ]

After Tamas's alternate implementation, we have a different result that is much improved using latest master branch.

jvm 1    | 2011-12-07 14:39:04 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, not yet finished.
jvm 1    | 2011-12-07 14:39:04 INFO  [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository.
jvm 1    | 2011-12-07 14:41:22 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/'
jvm 1    | 2011-12-07 14:48:41 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, finished successfully on Wed Dec 07 14:48:41 NST 2011
Comment by Tamás Cservenák [ 12/08/11 ]

Second pull request, wrt "anti-optimization" Peter found out:
https://github.com/sonatype/nexus/pull/131

Comment by Peter Lynch [ 12/08/11 ]

Adding performance snapshot of 1.10.0 nexus-oss-webapp after Tamas's Parent-o-matic implementation was added

Comment by Peter Lynch [ 12/08/11 ]

Notable differences between 1.9.2.3 and latest master branch:

  • latest master now generates GA level metadata files, 1.9.2.3 did not
  • latest master now stores under storage/<repoid>/.nexus proxy attributes and these get generated at end of snapshot removal, 1.9.2.3 generates these as well, but they get stored outside storage in nexus/proxy/attributes folder

The above is intended behavior.

Comment by Peter Lynch [ 12/08/11 ]

prformance improvement and no noticeable regressions

Generated at Fri Sep 22 12:12:39 UTC 2017 using JIRA 7.3.3#73014-sha1:d5be8da522213be2ca9ad7b043c51da6e4cc9754.