Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28166

Yum metadata not refreshed because of relying on "metadata_generation_time"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.28.0, 3.30.1
    • Fix Version/s: 3.35.0
    • Component/s: Yum
    • Labels:
    • Environment:
      Pro
    • Notability:
      2

      Description

      ISSUE:

      Occasionally, newly promoted (stage-moved) an rpm asset can't be installed, because filelists.xml.gz, primary.xml.gz and other.xml.gz are not updated.

      REPRODUCE STEPS:

      1. Prepare two yum hosted repos (eg: yum-releases and yum-released-prd) in your Nexus 3.30.1
      2. Prepare assets and metadata:
        _REPO_NAME_FROM="yum-releases"
        _REPO_NAME_TO="yum-releases-prd"
        _RPM_FILE_1="highcontrast-qt-0.1-2.el7.x86_64.rpm"
        _RPM_FILE_2="adwaita-qt-common-1.0-1.el7.x86_64.rpm"
        
        # Downloading files from internet
        curl -f -O "http://mirror.centos.org/centos/7/os/x86_64/Packages/${_RPM_FILE_1}"
        curl -f -O "http://mirror.centos.org/centos/7/os/x86_64/Packages/${_RPM_FILE_2}"
        
        # Upload _RPM_FILE_1 into both repos to create YUM metadata
        curl -u admin:admin123 -X PUT "http://localhost:8081/repository/${_REPO_NAME_FROM}/Packages/${_RPM_FILE_1}" --upload-file ${_RPM_FILE_1}
        curl -u admin:admin123 -X PUT "http://localhost:8081/repository/${_REPO_NAME_TO}/Packages/${_RPM_FILE_1}" --upload-file ${_RPM_FILE_1}
        sleep 60; date;
        
        # Upload _RPM_FILE_2 into the _REPO_NAME_FROM repo
        curl -v -u admin:admin123 -X PUT "http://localhost:8081/repository/${_REPO_NAME_FROM}/Packages/${_RPM_FILE_2}" --upload-file ${_RPM_FILE_2}
        sleep 60; date;
        
      1. Run "Repair - Rebuild Yum repository metadata (repodata)" task against _REPO_NAME_TO repository, so that the metadata_generation_time is updated.
      2. Enable org.sonatype.nexus.repository.yum = DEBUG logging
      3. Promote _RPM_FILE_2 from _REPO_NAME_FROM to _REPO_NAME_TO:
        curl -u admin:admin123 -X POST --header 'Content-Type: application/json' http://localhost:8081/service/rest/v1/tags -d '{ "name": "moveTag" }
        curl -u admin:admin123 -X POST 'http://localhost:8081/service/rest/v1/tags/associate/moveTag?repository=yum-releases&name=adwaita-qt-common&version=1.0-1.el7'
        
        curl -u admin:admin123 -X POST 'http://localhost:8081/service/rest/v1/staging/move/yum-releases-prd?tag=moveTag'
        

      EXPECTED BEHAVIOUR:

      The debug level log in nexus.log should say "Found 1 new RPMs to be added".

      ACTUAL BEHAVIOUR:

      2021-06-28 13:04:25,411+1000 DEBUG [event-19-thread-719]  admin org.sonatype.nexus.repository.yum.orient.internal.database.OrientYumQueryComponent - Finding repomd in yum-releases-prd
      ... (snip) ...
      2021-06-28 13:04:25,414+1000 DEBUG [event-19-thread-719]  admin org.sonatype.nexus.repository.yum.orient.internal.createrepo.OrientCreateRepoServiceImpl - Writing non-repomd metadata for repository yum-releases-prd and directory
      2021-06-28 13:04:25,414+1000 DEBUG [event-19-thread-719]  admin org.sonatype.nexus.repository.yum.orient.internal.createrepo.OrientCreateRepoServiceImpl - Directory  is marked as use cache, appending metadata from existing cached files
      2021-06-28 13:04:25,415+1000 DEBUG [event-19-thread-719]  admin org.sonatype.nexus.repository.yum.orient.internal.database.OrientYumQueryComponent - Finding RPMs last updated with query: SELECT FROM asset WHERE bucket=:bucket AND blob_updated > :blobupdated AND component IS NOT NULL params: {bucket=#111:18, blobupdated=1624849285980, path=%}
      2021-06-28 13:04:25,415+1000 DEBUG [event-19-thread-719]  admin org.sonatype.nexus.repository.yum.orient.internal.createrepo.OrientCreateRepoServiceImpl - Found 0 new RPMs to be added
      

      Also, the generated metadata files do not contain _RPM_FILE_2 name:

      $ zgrep -c 'adwaita' *.xml.gz
      4fed63512c09781edc495e7c2bd60fc112616e8813b255fbe8ab1b27aa732ac7-filelists.xml.gz:0
      7af5aa792efbd8939db5ba8452037511e6ebf9a9110cdfa00a420c7e9546b55f-primary.xml.gz:0
      ee6fe8180a647b1dab5bb5c6a20f81203f359ea16ec6cf5c5419721fb54e3a5c-other.xml.gz:0
      

      NOTE: if I search "highcontrast", i get 1 from all three files.

      POTENTIAL CAUSE:

      This might be intentional but, when we move the asset, we do not update blob_updated.
      Then, OrientCreateRepoServiceImpl#browseNewRpms() generates a SQL statement to find newly uploaded assets, which uses "hosted.getMetadataGenerationStartedTime()", which is basically from the bucket table:

      orientdb {db=component}> select attributes.metadata_generation_time from bucket where repository_name like 'yum-releases-prd';
      
      +----+-------------+
      |#   |attributes   |
      +----+-------------+
      |0   |1624849465408|
      +----+-------------+
      

      This attribute looks like updated only by the Rebuild Yum repository metadata task, hence when a before-moving asset was created before this task run, the SELECT statement does not find this newly moved asset.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mbucher Michael Bucher
              Reporter:
              hosako Hajime Osako
              CC:
              David Milet
              Last Updated By:
              Joe Tom Joe Tom
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title