Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-25793

PyPI proxy repository returns 404 for a package that exists upstream


    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.27.0, 3.28.1, 3.29.2
    • Fix Version/s: None
    • Component/s: PyPI
    • Labels:
    • Environment:
      Ubuntu 18.04
    • Notability:


      I have just recently upgraded an installation of Nexus OSS  3.20.0-04 to 3.28.1-01 (on Ubuntu 18.04).

      Since the upgrade, the Nexus PyPi proxy repository has been returning 404 for a specific package that exists on the upstream server: pkg_resources. It seems to work fine for every other package I have tested.

      We first noticed the issue when various CI pipelines began to fail when creating Python virtual environments:

      Collecting pkg_resources
      Traceback (most recent call last):
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main
          status = self.run(options, args)
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 353, in run
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 749, in build
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link
          self.link = finder.find_requirement(self, upgrade)
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement
          all_candidates = self.find_all_candidates(req.name)
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates
          for page in self._get_pages(url_locations, project_name):
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages
          page = self._get_page(location)
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page
          return HTMLPage.get_page(link, session=self.session)
        File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 795, in get_page
        File "/scratch/ben/nexus_test/test/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
          raise HTTPError(http_error_msg, response=self)
      HTTPError: 404 Client Error: Not Found for url: https://nexus/repository/pypi-proxy/simple/pkg-resources/

      I tried a bunch of things to attempt to resolve this but had no luck:

      • Invalidating the cache for the pypi-proxy repo
      • Disabling negative caching for the pypi-proxy repo
      • Deleting and recreating the pypi-proxy repo
      • Clearing /opt/sonatype-work/cache and /opt/sonatype-work/tmp and restarting Nexus
      • Probably more that I forgot

      I captured the attached log from our production Nexus server. This shows the client request to Nexus, Nexus making a request to https://pypi.org (which is successful, and receives a 200 status code) and the the resulting 404 sent back to the client.

      I have been forced to rollback the update in order to circumvent this issue.


      I have been able to reproduce this issue in a test environment with a fresh installation of 3.28.1-01. It works fine with a fresh installation of 3.20.0-04.

      Steps to reproduce: 
      1. Fresh install of Nexus OSS and clear any existing data directory (/opt/sonatype-work)

      2. Create a PyPi proxy repo in the web UI

       Name: pypi-proxy
       Online: yes
       Remote Storage: https://pypi.org/
       Use the Nexus truststore: no
       Blocked: no
       Auto blocking: yes
       Max component age: 1440
       Max metadata age: 1440
       Blob store: default
       Strict Content Type Validation: yes
       Routing table: none
       Not found cache enable: yes
       Not found cache TTL: 1440
       Cleanup policies: none
       HTTP Authentication: no
       HTTP request settings: none

      3. Request simple/pkg-resources

      $ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources

      On Nexus OSS 3.20.0-04 this works as expected:

      $ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources
       <html lang="en">
       <head><title>Links for pkg-resources</title>
       <meta name="api-version" value="2"/>
       <body><h1>Links for pkg-resources</h1>

      This mostly matches what I get from the upstream server

      $ curl https://pypi.org/simple/pkg-resources/
       <!DOCTYPE html>
       <title>Links for pkg-resources</title>
       <h1>Links for pkg-resources</h1>

      However, on Nexus OSS 3.28.1-01 I get a 404

      $ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources
       <!DOCTYPE html>
       <html lang="en">
       <title>404 - Nexus Repository Manager</title>
       <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
      <!--[if lt IE 9]>
       <script>(new Image).src="http://endepv001:8081/favicon.ico?3.28.1-01"</script>
       <link rel="icon" type="image/png" href="http://endepv001:8081/favicon-32x32.png?3.28.1-01" sizes="32x32">
       <link rel="mask-icon" href="http://endepv001:8081/safari-pinned-tab.svg?3.28.1-01" color="#5bbad5">
       <link rel="icon" type="image/png" href="http://endepv001:8081/favicon-16x16.png?3.28.1-01" sizes="16x16">
       <link rel="shortcut icon" href="http://endepv001:8081/favicon.ico?3.28.1-01">
       <meta name="msapplication-TileImage" content="http://endepv001:8081/mstile-144x144.png?3.28.1-01">
       <meta name="msapplication-TileColor" content="#00a300">
      <link rel="stylesheet" type="text/css" href="http://endepv001:8081/static/css/nexus-content.css?3.28.1-01"/>
       <div class="nexus-header">
       <a href="http://endepv001:8081">
       <div class="product-logo">
       <img src="http://endepv001:8081/static/rapture/resources/icons/x32/nexus-white.png?3.28.1-01" alt="Product logo"/>
       <div class="product-id">
       <div class="product-id__line-1">
       <span class="product-name">Nexus Repository Manager</span>
       <div class="product-id__line-2">
       <span class="product-spec">OSS 3.28.1-01</span>
      <div class="nexus-body">
       <div class="content-header">
       <img src="http://endepv001:8081/static/rapture/resources/icons/x32/exclamation.png?3.28.1-01" alt="Exclamation point" aria-role="presentation"/>
       <span class="title">Error 404</span>
       <span class="description">Not Found</span>
       <div class="content-body">
       <div class="content-section">
       Not Found

      While pypi.org returns what is essentially an empty page, the Python virtualenv module seem to expect it to be available (I have no idea what purpose this serves). In any case, this appears to be a regression in Nexus.

      Please let me know if you need any more information from me.




            Unassigned Unassigned
            bcrisp Ben Crisp
            Last Updated By:
            Rich Seddon Rich Seddon
            3 Vote for this issue
            9 Start watching this issue


              Date of First Response: