Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-24332

optimize how yum repodata metadata is replaced in real time when rebuilt

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.24.0
    • Fix Version/s: 3.36.0
    • Component/s: Yum
    • Story Points:
      5
    • Sprint:
      NXRM Trinitys Sprint 16, NXRM Sentinels Sprint 17, NXRM Sentinels Sprint 18
    • Notability:
      2

      Description

      1. Rebuild metadata of a large yum repo where the rebuild takes many seconds or minutes.
      2. A yum build may request the repomd.xml file and then immediately attempt to download the referenced metadata files within it. In that window, the referenced metadata files may be removed by the rebuild, before the client has finished requesting all the files. This causes a 404 and build failure, even though the now deleted metadata files could have referenced rpms which still exist and would have allowed the build to pass.

      A customer wrote a groovy script that simulates requesting repomd.xml and primary.xml.gz in a tight loop. During an NXRM yum metadata rebuild, this script will fail with a 404 response for the primary.xml.gz.

      def download(String remoteUrl, String localUrl) {
          new File("$localUrl").withOutputStream { out ->
              new URL(remoteUrl).withInputStream { from ->  out << from; }
          }
      }
      
      def String readFileString(String filePath) {
          File file = new File(filePath)
          String fileContent = file.text
          return fileContent
      }
      
      def baseUrl = 'http://localhost:8081/repository/yum-hosted/A/'
      def repomdUrl = 'repodata/repomd.xml'
      
      for (int i = 0; i < 100000; i++) {
          new File("repomd.xml").delete()
          download(baseUrl+repomdUrl, "repomd.xml")
          String xml = readFileString("repomd.xml")
      
          def rootNode = new XmlSlurper().parseText(xml)
          assert rootNode.name() == 'repomd'
          def primary = rootNode.children().find { node ->
              node.name() == 'data' && node.@type == 'primary'
          }
          def primaryUrl = primary.location.@href
          new File("primary.xml.gz").delete()
          download(baseUrl+primaryUrl, "primary.xml.gz")
      }
      
      System.out.println('DONE!')
      
      

      Expected

      Come up with a more tolerant metadata rebuild algorithm that avoids the chances of 404 and build failures.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              plynch Peter Lynch
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Maksym Lukaretkyi Maksym Lukaretkyi
              Team:
              NXRM - Sentinels
              Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title