Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-25502

yum metadata is rebuilt after downloading it which can cause build failures

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.27.0
    • Fix Version/s: 3.29.0
    • Component/s: Yum
    • Notability:
      3

      Description

      Yum repo metadata can get rebuilt by the automatic rebuild task even though the only access to repository content were downloads(reads) of that content. The problem is this can increase the chances of yum build failures that have already fetched the old metadata (repodata/repomd.xml) when those builds make a request for the files referenced by it ( ie. primary, filelists, etc ) and due to the rebuild are now deleted.

      Expected

      Rebuilding metadata just because yum assets have been accessed/downloaded should not happen.

      Reproduce

      1. Create a hosted yum repo with level 1
      2. Deploy two rpms to 123 dir - wait for metadata to be built
      3. Download 123/repodata/repomd.xml , and then the primary.xml referenced by it. Then download one of the two rpms from dir 123.
      4. Upload an rpm to directory 456 - this will trigger automatic metadata rebuild task for entire repo as expected
      5. Notice when the task runs to rebuild metadata for dir 456( due to deploy), the audit log reports that the metadata for directory 123 gets deleted and rebuilt. This is not expected as the only thing that happened in that directory were downloads.

      Potential Client Workarounds

      Yum Client Metadata Expiry

      Since a reported build failure occurred due to this bug, because there was a gap of about 3 minutes between downloading the repomd.xml and the filelists.xml.gz file, the client may be able to adjust some yum specific cache settings to force the client to download all metadata at once instead of waiting until it is actually needed by the build.

      From https://linux.die.net/man/5/yum.conf

      metadata_expire Time (in seconds) after which the metadata will expire. So that if the current metadata downloaded is less than this many seconds old then yum will not update the metadata against the repository. If you find that yum is not downloading information on updates as often as you would like lower the value of this option. You can also change from the default of using seconds to using days, hours or minutes by appending a d, h or m respectively. The default is 6 hours, to compliment yum-updatesd running once an hour. It's also possible to use the word "never", meaning that the metadata will never expire. Note that when using a metalink file the metalink must always be newer than the metadata for the repository, due to the validation, so this timeout also applies to the metalink file.

      Yum Client Metadata Policy Adjustments

      Possibly adjusting the mdpolicy to group:all or group:main ( From https://linux.die.net/man/5/yum.conf )

      mdpolicy You can select from different metadata download policies depending on how much data you want to download with the main repository metadata index. The advantages of downloading more metadata with the index is that you can't get into situations where you need to use that metadata later and the versions available aren't compatible (or the user lacks privileges) and that if the metadata is corrupt in any way yum will revert to the previous metadata.

      'instant' - Just download the new metadata index, this is roughly what yum always did, however it now does some checking on the index and reverts if it classifies it as bad.

      'group:primary' - Download the primary metadata with the index. This contains most of the package information and so is almost always required anyway. This is the default.

      'group:small' - With the primary also download the updateinfo metadata, this is required for yum-security operations and it also used in the graphical clients. This file also tends to be significantly smaller than most others.

      'group:main' - With the primary and updateinfo download the filelists metadata and the group metadata. The filelists data is required for operations like "yum install /bin/bash", and also some dependency resolutions require it. The group data is used in some graphical clients and for group operations like "yum grouplist Base".

      'group:all' - Download all metadata listed in the index, currently the only one not listed above is the other metadata, which contains the changelog information which is used by yum-changelog. This is what "yum makecache" uses.

      Yum Client Command makecache

      Ensure that all yum metadata files for the Nexus yum repo is downloaded (together) by executing yum makecache. Caveat: Using makecache for all yum repos can download way too much data and fills up /var. Increasing /var is not an option.

      yum --enablerepo=NEXUS_YUM_REPO_ID clean all
      yum --disablerepo=* --enablerepo=NEXUS_YUM_REPO_ID makecache

      By pre-caching metadata, it avoids the potentially large delay in the yum client making requests for different metadata files during the same build, thus potentially avoiding what would otherwise be a metadata retrieval 404 not found problem.

        Attachments

          Activity

            People

            Assignee:
            mallen Mick Allen
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Peter Lynch Peter Lynch
            Team:
            NXRM - Groot
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title