Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-34351

import task on flat format directory structure with large number of files may not be performant

    Details

    • Notability:
      3

      Description

      Create a docker repository with 1 million layers(assets).
      Export this to a directory - now the directory contains one million files.
      Use the Import task to import this directory.
      Since the algorithm to import files keeps a special *dat file to track paths imported and this dat file will also contain eventually the same number of paths as the files in the flat dir structure, the import can take a LONG time to complete in some scenarios, like in the scenario of restarting the import with the objective to only import the NEW not yet imported files. In one customer scenario the import was reported to take over 2 months.

      Expected

      Adjust the import algorithm to work more efficiently for flat directory structures.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Michael Oliverio Michael Oliverio
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:

                tigCommentSecurity.panel-title