Dev - Nexus
  1. Dev - Nexus
  2. NEXUS-4640

Very slow performance of snapshot remover

    Details

    • Global Rank:
      18113
    • Story Points:
      15

      Description

      I've been getting quite a few reports of snapshot removal tasks taking a very long time to run (over 24 hours in some cases). On RSO the weekly snapshot removal job takes around 4 hours.

      This is far too slow. We need to do some profiling, figure out why this takes so long, and see if we can implement some improvements.

      1. Launcher-snapshots-2011-12-07-14-48.snapshot
        3.99 MB
        Peter Lynch

        Issue Links

          Activity

          Hide
          Rich Seddon added a comment -

          There may be a memory issue in play here, see linked issues.

          Show
          Rich Seddon added a comment - There may be a memory issue in play here, see linked issues.
          Hide
          Brian Demers added a comment -

          Do some profiling to figure out what is slow

          Show
          Brian Demers added a comment - Do some profiling to figure out what is slow
          Hide
          Tamás Cservenák added a comment -

          Original fixes merged in as:
          https://github.com/sonatype/nexus/pull/115

          But it should improve more (with default config), or a Lot (in case of custom config with multiple shadows) after this is merged in too:
          https://github.com/sonatype/nexus/pull/119

          Show
          Tamás Cservenák added a comment - Original fixes merged in as: https://github.com/sonatype/nexus/pull/115 But it should improve more (with default config), or a Lot (in case of custom config with multiple shadows) after this is merged in too: https://github.com/sonatype/nexus/pull/119
          Hide
          Peter Lynch added a comment -

          At Tamas's request re-opening and put back to in his hands.

          I created a hosted repo for SNAPSHOTS, 100 groups x 100 attribute ids * randomly dispersed versions for a total of approx. 13000 artifacts. maven metadata was not created initially before task ran.

          I ran snapshot remover task against this. These are the logs:

          jvm 1    | 2011-12-06 10:52:35 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, not yet finished.
          jvm 1    | 2011-12-06 10:52:35 INFO  [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository.
          jvm 1    | 2011-12-06 10:54:29 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Expiring local cache in repository ID='repo1' from path='/'
          jvm 1    | 2011-12-06 10:55:25 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/'
          jvm 1    | 2011-12-06 11:15:24 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, finished successfully on Tue Dec 06 11:15:23 NST 2011
          

          As can be seen creating the metadata looks like the bulk of the overhead.

          Show
          Peter Lynch added a comment - At Tamas's request re-opening and put back to in his hands. I created a hosted repo for SNAPSHOTS, 100 groups x 100 attribute ids * randomly dispersed versions for a total of approx. 13000 artifacts. maven metadata was not created initially before task ran. I ran snapshot remover task against this. These are the logs: jvm 1 | 2011-12-06 10:52:35 INFO [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, not yet finished. jvm 1 | 2011-12-06 10:52:35 INFO [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository. jvm 1 | 2011-12-06 10:54:29 INFO [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Expiring local cache in repository ID='repo1' from path='/' jvm 1 | 2011-12-06 10:55:25 INFO [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/' jvm 1 | 2011-12-06 11:15:24 INFO [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Tue Dec 06 10:52:35 NST 2011, finished successfully on Tue Dec 06 11:15:23 NST 2011 As can be seen creating the metadata looks like the bulk of the overhead.
          Hide
          Peter Lynch added a comment -

          attaching raw snapshot repo I used ( manually put in storage dir and configured nexus.xml with

          <repository>
                <id>repo1</id>
                <name>repo1</name>
                <providerRole>org.sonatype.nexus.proxy.repository.Repository</providerRole>
                <providerHint>maven2</providerHint>
                <localStatus>IN_SERVICE</localStatus>
                <notFoundCacheActive>true</notFoundCacheActive>
                <notFoundCacheTTL>1440</notFoundCacheTTL>
                <userManaged>true</userManaged>
                <exposed>true</exposed>
                <browseable>true</browseable>
                <writePolicy>ALLOW_WRITE</writePolicy>
                <localStorage>
                  <provider>file</provider>
                </localStorage>
                <externalConfiguration>
                  <repositoryPolicy>SNAPSHOT</repositoryPolicy>
                </externalConfiguration>
              </repository>
          
          <task>
                <id>1</id>
                <name>repo1-test</name>
                <enabled>true</enabled>
                <type>SnapshotRemoverTask</type>
                <status>SUBMITTED</status>
                <schedule>
                  <type>manual</type>
                </schedule>
                <properties>
                  <property>
                    <key>removeOlderThanDays</key>
                    <value></value>
                  </property>
                  <property>
                    <key>minSnapshotsToKeep</key>
                    <value>0</value>
                  </property>
                  <property>
                    <key>removeIfReleaseExists</key>
                    <value>false</value>
                  </property>
                  <property>
                    <key>.name</key>
                    <value>repo1-test</value>
                  </property>
                  <property>
                    <key>repositoryId</key>
                    <value>repo1</value>
                  </property>
                </properties>
              </task>
          
          
          Show
          Peter Lynch added a comment - attaching raw snapshot repo I used ( manually put in storage dir and configured nexus.xml with <repository> <id> repo1 </id> <name> repo1 </name> <providerRole> org.sonatype.nexus.proxy.repository.Repository </providerRole> <providerHint> maven2 </providerHint> <localStatus> IN_SERVICE </localStatus> <notFoundCacheActive> true </notFoundCacheActive> <notFoundCacheTTL> 1440 </notFoundCacheTTL> <userManaged> true </userManaged> <exposed> true </exposed> <browseable> true </browseable> <writePolicy> ALLOW_WRITE </writePolicy> <localStorage> <provider> file </provider> </localStorage> <externalConfiguration> <repositoryPolicy> SNAPSHOT </repositoryPolicy> </externalConfiguration> </repository> <task> <id> 1 </id> <name> repo1-test </name> <enabled> true </enabled> <type> SnapshotRemoverTask </type> <status> SUBMITTED </status> <schedule> <type> manual </type> </schedule> <properties> <property> <key> removeOlderThanDays </key> <value> </value> </property> <property> <key> minSnapshotsToKeep </key> <value> 0 </value> </property> <property> <key> removeIfReleaseExists </key> <value> false </value> </property> <property> <key> .name </key> <value> repo1-test </value> </property> <property> <key> repositoryId </key> <value> repo1 </value> </property> </properties> </task>
          Hide
          Peter Lynch added a comment -

          After Tamas's alternate implementation, we have a different result that is much improved using latest master branch.

          jvm 1    | 2011-12-07 14:39:04 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, not yet finished.
          jvm 1    | 2011-12-07 14:39:04 INFO  [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository.
          jvm 1    | 2011-12-07 14:41:22 INFO  [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/'
          jvm 1    | 2011-12-07 14:48:41 INFO  [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, finished successfully on Wed Dec 07 14:48:41 NST 2011
          
          Show
          Peter Lynch added a comment - After Tamas's alternate implementation, we have a different result that is much improved using latest master branch. jvm 1 | 2011-12-07 14:39:04 INFO [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, not yet finished. jvm 1 | 2011-12-07 14:39:04 INFO [pool-1-thread-1] - org.sonatype.nexus.maven.tasks.DefaultSnapshotRemover - Removing old SNAPSHOT deployments from repo1 repository. jvm 1 | 2011-12-07 14:41:22 INFO [pool-1-thread-1] - org.sonatype.nexus.proxy.maven.maven2.M2Repository - Clearing NFC cache in repository ID='repo1' from path='/' jvm 1 | 2011-12-07 14:48:41 INFO [pool-1-thread-1] - org.sonatype.nexus.feeds.DefaultFeedRecorder - Removing snapshots from repository repo1 : Process started on Wed Dec 07 14:39:04 NST 2011, finished successfully on Wed Dec 07 14:48:41 NST 2011
          Hide
          Tamás Cservenák added a comment -

          Second pull request, wrt "anti-optimization" Peter found out:
          https://github.com/sonatype/nexus/pull/131

          Show
          Tamás Cservenák added a comment - Second pull request, wrt "anti-optimization" Peter found out: https://github.com/sonatype/nexus/pull/131
          Hide
          Peter Lynch added a comment -

          Adding performance snapshot of 1.10.0 nexus-oss-webapp after Tamas's Parent-o-matic implementation was added

          Show
          Peter Lynch added a comment - Adding performance snapshot of 1.10.0 nexus-oss-webapp after Tamas's Parent-o-matic implementation was added
          Hide
          Peter Lynch added a comment -

          Notable differences between 1.9.2.3 and latest master branch:

          • latest master now generates GA level metadata files, 1.9.2.3 did not
          • latest master now stores under storage/<repoid>/.nexus proxy attributes and these get generated at end of snapshot removal, 1.9.2.3 generates these as well, but they get stored outside storage in nexus/proxy/attributes folder

          The above is intended behavior.

          Show
          Peter Lynch added a comment - Notable differences between 1.9.2.3 and latest master branch: latest master now generates GA level metadata files, 1.9.2.3 did not latest master now stores under storage/<repoid>/.nexus proxy attributes and these get generated at end of snapshot removal, 1.9.2.3 generates these as well, but they get stored outside storage in nexus/proxy/attributes folder The above is intended behavior.
          Hide
          Peter Lynch added a comment -

          prformance improvement and no noticeable regressions

          Show
          Peter Lynch added a comment - prformance improvement and no noticeable regressions

            People

            • Assignee:
              Peter Lynch
              Reporter:
              Rich Seddon
              Last Updated By:
              Rich Seddon
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Date of First Response: