Details
Major Description
I have just recently upgraded an installation of Nexus OSS 3.20.0-04 to 3.28.1-01 (on Ubuntu 18.04).
Since the upgrade, the Nexus PyPi proxy repository has been returning 404 for a specific package that exists on the upstream server: pkg_resources. It seems to work fine for every other package I have tested.
We first noticed the issue when various CI pipelines began to fail when creating Python virtual environments:
Collecting pkg_resources Exception: Traceback (most recent call last): File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 353, in run wb.build(autobuilding=True) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/wheel.py", line 749, in build self.requirement_set.prepare_files(self.finder) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_set.py", line 554, in _prepare_file require_hashes File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/req/req_install.py", line 278, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 465, in find_requirement all_candidates = self.find_all_candidates(req.name) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 423, in find_all_candidates for page in self._get_pages(url_locations, project_name): File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 568, in _get_pages page = self._get_page(location) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 683, in _get_page return HTMLPage.get_page(link, session=self.session) File "/usr/share/python-wheels/pip-9.0.1-py2.py3-none-any.whl/pip/index.py", line 795, in get_page resp.raise_for_status() File "/scratch/ben/nexus_test/test/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status raise HTTPError(http_error_msg, response=self) HTTPError: 404 Client Error: Not Found for url: https://nexus/repository/pypi-proxy/simple/pkg-resources/
I tried a bunch of things to attempt to resolve this but had no luck:
- Invalidating the cache for the pypi-proxy repo
- Disabling negative caching for the pypi-proxy repo
- Deleting and recreating the pypi-proxy repo
- Clearing /opt/sonatype-work/cache and /opt/sonatype-work/tmp and restarting Nexus
- Probably more that I forgot
I captured the attached log from our production Nexus server. This shows the client request to Nexus, Nexus making a request to https://pypi.org (which is successful, and receives a 200 status code) and the the resulting 404 sent back to the client.
I have been forced to rollback the update in order to circumvent this issue.
I have been able to reproduce this issue in a test environment with a fresh installation of 3.28.1-01. It works fine with a fresh installation of 3.20.0-04.
Steps to reproduce:
1. Fresh install of Nexus OSS and clear any existing data directory (/opt/sonatype-work)
2. Create a PyPi proxy repo in the web UI
Settings: Name: pypi-proxy Online: yes Remote Storage: https://pypi.org/ Use the Nexus truststore: no Blocked: no Auto blocking: yes Max component age: 1440 Max metadata age: 1440 Blob store: default Strict Content Type Validation: yes Routing table: none Not found cache enable: yes Not found cache TTL: 1440 Cleanup policies: none HTTP Authentication: no HTTP request settings: none
3. Request simple/pkg-resources
$ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources
On Nexus OSS 3.20.0-04 this works as expected:
$ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources <html lang="en"> <head><title>Links for pkg-resources</title> <meta name="api-version" value="2"/> </head> <body><h1>Links for pkg-resources</h1> </body> </html>
This mostly matches what I get from the upstream server
$ curl https://pypi.org/simple/pkg-resources/ <!DOCTYPE html> <html> <head> <title>Links for pkg-resources</title> </head> <body> <h1>Links for pkg-resources</h1> </body> </html>
However, on Nexus OSS 3.28.1-01 I get a 404
$ curl http://localhost:8081/repository/pypi-proxy/simple/pkg-resources <!DOCTYPE html> <html lang="en"> <head> <title>404 - Nexus Repository Manager</title> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <!--[if lt IE 9]> <script>(new Image).src="http://endepv001:8081/favicon.ico?3.28.1-01"</script> <![endif]--> <link rel="icon" type="image/png" href="http://endepv001:8081/favicon-32x32.png?3.28.1-01" sizes="32x32"> <link rel="mask-icon" href="http://endepv001:8081/safari-pinned-tab.svg?3.28.1-01" color="#5bbad5"> <link rel="icon" type="image/png" href="http://endepv001:8081/favicon-16x16.png?3.28.1-01" sizes="16x16"> <link rel="shortcut icon" href="http://endepv001:8081/favicon.ico?3.28.1-01"> <meta name="msapplication-TileImage" content="http://endepv001:8081/mstile-144x144.png?3.28.1-01"> <meta name="msapplication-TileColor" content="#00a300"> <link rel="stylesheet" type="text/css" href="http://endepv001:8081/static/css/nexus-content.css?3.28.1-01"/> </head> <body> <div class="nexus-header"> <a href="http://endepv001:8081"> <div class="product-logo"> <img src="http://endepv001:8081/static/rapture/resources/icons/x32/nexus-white.png?3.28.1-01" alt="Product logo"/> </div> <div class="product-id"> <div class="product-id__line-1"> <span class="product-name">Nexus Repository Manager</span> </div> <div class="product-id__line-2"> <span class="product-spec">OSS 3.28.1-01</span> </div> </div> </a> </div> <div class="nexus-body"> <div class="content-header"> <img src="http://endepv001:8081/static/rapture/resources/icons/x32/exclamation.png?3.28.1-01" alt="Exclamation point" aria-role="presentation"/> <span class="title">Error 404</span> <span class="description">Not Found</span> </div> <div class="content-body"> <div class="content-section"> Not Found </div> </div> </div> </body> </html>
While pypi.org returns what is essentially an empty page, the Python virtualenv module seem to expect it to be available (I have no idea what purpose this serves). In any case, this appears to be a regression in Nexus.
Please let me know if you need any more information from me.