Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28522

change the default location for storing import task metadata

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Complex Fast-Track
    • Resolution: Done
    • Affects Version/s: 3.32.0
    • Fix Version/s: 3.34.1
    • Component/s: import-export
    • Labels:

      Description

      When the repository import task runs, it stores metadata ( delta files ) about import progress in a hidden sub-directory of the source directory at these paths

      SOURCE_DIR/.nexus/**/imported-files*.dat

      .nexus was chosen as parent because any content in this directory is not accessible by HTTP access if the parent storage directory is for a NXRM 2 repository ( a common use case )

      Sometimes the import task may need to be restarted, and the admin may wish to restart the import from scratch, without retaining any progress thus far. To do this, all progress and files generated by a previous run of the import task need to be manually deleted.

      The problem the file layout creates is if the source directory is a NXRM 2 repository storage location, a .nexus directory already exists which contains well-known directories NXRM 2 uses such as

      .nexus/attributes
      .nexus/trash
      .nexus/tmp
      .nexus/meta
      archetype-catalog.xml

      Combined with the fact that the import task stores dat files are in a nest structure under .nexus, selecting the files to delete without harming files needed by NXRM 2 for normal operation can be overly complicated and dangerous.

      Example NX2 repo storage dir source before import task
      ./.meta/prefixes.txt
      ./.meta/discovery-status.txt
      ./.meta/repository-metadata.xml
      ./.meta/nexus-smartproxy-plugin/handshake
      ./.nexus/attributes/.meta/prefixes.txt
      ./.nexus/attributes/.meta/discovery-status.txt
      ./.nexus/attributes/.meta/repository-metadata.xml
      ./.nexus/attributes/.meta/nexus-smartproxy-plugin/handshake
      ./.nexus/attributes/archetype-catalog.xml
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.pom
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2-sources.jar
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.jar
      ./archetype-catalog.xml
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.pom
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2-sources.jar
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.jar
      
      Example NX2 storage dir source after import task showing mix of dat files
      ./.meta/prefixes.txt
      ./.meta/discovery-status.txt
      ./.meta/repository-metadata.xml
      ./.meta/nexus-smartproxy-plugin/handshake
      ./.nexus/imported-files-7BABA42A-2E0A3591-A6125A30-AFE2018B-CB9D0A6E.dat
      ./.nexus/attributes/.meta/prefixes.txt
      ./.nexus/attributes/.meta/discovery-status.txt
      ./.nexus/attributes/.meta/repository-metadata.xml
      ./.nexus/attributes/.meta/nexus-smartproxy-plugin/handshake
      ./.nexus/attributes/archetype-catalog.xml
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.pom
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2-sources.jar
      ./.nexus/attributes/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.jar
      ./.nexus/com/fasterxml/jackson/core/jackson-databind/2.9.9.2/imported-files-7BABA42A-2E0A3591-A6125A30-AFE2018B-CB9D0A6E.dat
      ./archetype-catalog.xml
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.pom
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2-sources.jar
      ./com/fasterxml/jackson/core/jackson-databind/2.9.9.2/jackson-databind-2.9.9.2.jar
      

      Expected

      Change the default source location to store import task dat files from

      .nexus/

      to

      .nexus/nxrm-import-task

      By doing this, the only command that an admin needs to run to reset the import data files is

      rm .nexus/nxrm-import-task

      Instead of a complicated command excluding important files and finding relevant files in a deep nested structure.

      However to handle the scenario where import task is run before and then after upgrading to an NXRM version that has this change, add a simple check for a file at .nexus/imported-files*.dat at start of task. If that file is present, then act like the root delta directory is still .nexus/

      Always log the location of the delta file root directory at INFO level regardless.

      Workaround to deleting import task source cache files inside Repository 2 repo storage dir

      Locate the source .nexus directory of the import.

      In repository 2 repo storage, the default locations would be like sonatype-work/nexus/storage/<repoid>/.nexus

      For example if the source repo id was 'releases', the directory relative to install dir would be sonatype-work/nexus/storage/releases/.nexus

      Use a command like this to find and delete all the import cache files:

      find sonatype-work/nexus/storage/releases/.nexus -type f -name 'imported-files*.dat' -delete
      

        Attachments

          Activity

            People

            Assignee:
            sdelvalle Santiago Del Valle
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Damian Bradicich Damian Bradicich
            Team:
            NXRM - Neo
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title