Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-31215

After migration to PostgreSQL, some assets become invisible

    Details

    • Notability:
      3

      Description

      SYMPTOM:

      After migrating from OrientDB to PostgreSQL, noticed some PyPi assets are not shown in the Browse page.
      Also, the HTML browse page (/service/rest/repository/browse/<repo-name>/<component-name>/) does not show any items but generates many ERRORs like below:

      2022-02-14 19:25:16,972+1000 ERROR [qtp14309831-575] admin org.sonatype.nexus.repository.content.browse.store.BrowseNodeQueryServiceImpl - Could not find expected asset (id): /unit/0.2.2/ (null) in repository: pypi-zzzz-hosted
      

      REPRODUCE STEPS:

      1. Install 3.73.3 with OrientDB
      2. Create pypi-hosted, pypi-group, pypi-zzzz-hosted repositories
      3. Upload some package into pypi-hosted and pypi-zzzz-hosted
        curl -O -L "https://files.pythonhosted.org/packages/24/44/38f25717a71df9992d5bd065fa3e7f85a2673af2ccee56caedf60386de5e/Unit-0.2.2.tar.gz"
        _NXRM_URL="http://localhost:8081/"
        curl -D- -u admin:admin123 "${_NXRM_URL%/}/service/rest/v1/components?repository=pypi-hosted" -H 'Content-Type: multipart/form-data' -F 'pypi.asset=@./Unit-0.2.2.tar.gz'
        curl -u admin:admin123 "${_NXRM_URL%/}/service/rest/v1/components?repository=pypi-zzzz-hosted" -H 'Content-Type: multipart/form-data' -F 'pypi.asset=@./Unit-0.2.2.tar.gz'
        
      4. Create some assets under pypi-group by installing some package
        pip3 install -i http://localhost:8081/repository/pypi-group/simple unit==0.2.2
        
      5. Migrate to NewDB

      EXPECTED:

      Before and after migration should provide same items in Browse page.

      ACTUAL RESULT:


      Also, noticed RebuildBrowseNodesManager#doStart() included group repositories, which might be causing this issue.

      2022-02-14 19:04:42,825+1000 INFO  [quartz-11-thread-1]  *SYSTEM org.sonatype.nexus.repository.browse.node.RebuildBrowseNodesTask -  Name: Rebuild repository browse tree - (pypi-hosted,pypi-group,pypi-proxy,pypi-zzzz-hosted)
      2022-02-14 19:04:42,825+1000 INFO  [quartz-11-thread-1]  *SYSTEM org.sonatype.nexus.repository.browse.node.RebuildBrowseNodesTask -  Description: Rebuilding browse tree for pypi-hosted,pypi-group,pypi-proxy,pypi-zzzz-hosted
      

      When a group repo was rebuilt before hosted repos, it inserts the records with the "asset_id", so that when a hosted repo was rebuilt, the hasAssetNode() method returned true, so that assets are skipped with below log entry:

      2022-02-14 11:18:34,971+1000 DEBUG [quartz-13-thread-18]  *SYSTEM org.sonatype.nexus.repository.content.browse.BrowseFacetImpl - Skipping asset /packages/<name>/<version>/<file-name>
      

      Please also check if other formats have same issue.

      Also, while testing, I always saw the Rebuild repository browse task was automatically scheduled twice.
      Is this expected?

      $ ls -l create.browse.nodes-202202141*
      -rw-r--r--  1 hosako  staff  2935 14 Feb 18:42 create.browse.nodes-20220214184217722.log
      -rw-r--r--  1 hosako  staff  5008 14 Feb 18:43 create.browse.nodes-20220214184247993.log
      

      POTENTIAL WORKAROUND:

      If PostgreSQL, try "TRUNCATE TABLE <format>_browse_node;", then create a new Rebuild repository browse task against the repositories for the format which was just truncated, but should not use the "All repositories" option.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            hosako Hajime Osako
            Last Updated By:
            Rich Seddon Rich Seddon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:

                tigCommentSecurity.panel-title