Details
Description
Description
Given that the Nexus Firewall Quarantining feature is enabled for a proxy repository in Nexus 2, if these events occur, then an artifact can appear permanently stuck in PENDING quarantine status and will not be served by Nexus:
- cache an artifact file in proxy repo - Nexus serves this file when requested
- enable quarantine for proxy repo
- Nexus downloads a new version of the proxied artifact from remote
- Nexus permanently reports the file as quarantined ( status=PENDING ) and returns 403 for all future requests as long as quarantine is enabled. Nexus will never check the artifact against IQ Server repository policies.
Example Reproduce
Setup
Configure IQ Server 1.50 - http://localhost:8070
No repository policies are needed for this test.
Repository Manager 2.14.4 ( or any Nexus 2.x version ) - http://localhost:8081/nexus
1) Configure a working IQ Server connection from NXRM to IQ server.
2) Create Repositories in Nexus 2
Maven 2 Proxy with default settings for RELEASE policy, including artifact max age -1
example-proxy - (http://localhost:8081/nexus/content/repositories/example-proxy/) proxies http://localhost:8081/nexus/content/repositories/example-hosted/
Maven 2 Hosted with Deployment Policy Allow Redeploy
example-hosted - (http://localhost:8081/nexus/content/repositories/example-hosted/)
3) Set the following loggers in NXRM to help verify outbound activity.
remote.storage.outbound to level DEBUG
org.apache.http to level DEBUG
4) Obtain an artifact file to test with, that would be identified by IQ server if it were evaluated:
curl -v -L http://jcenter.bintray.com/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o spring-security-web-3.1.0.RELEASE.jar
spring-security-web-3.1.0.RELEASE.jar
Reproduce
1. upload a file into example-hosted
curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar
2. Download the file from example-proxy
curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
3) Enable quarantine on example-proxy
Administration -> Capabilities
Add new
Audit and Quarantine: example-proxy
By design this action by itself will not quarantine already cache jars.
By design, new requests for a proxy cached file should never trigger a remote check of the jar, unless
- someone expires cache on the proxy repository ( or group containing it )
- someone deletes the cached jar file out-of-band from disk storage
4) Verify download is still successful (200) and does not trigger any remote checks for the new file.
curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
5) Upload new file into hosted repo at same coordinates
First check the last modified header on example-hosted for the jar download
curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
Wait at least 1 minute- Then upload a replacement:
curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE-new.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar
Verify the last modified header has changed from the initial value.
curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
5) Expire cache on example-proxy repo to force all new inbound requests to check remote for updates. This example curl command is the same operation that happens when a user right-clicks the repository in the UI and chooses "Expire Cache"
curl -v -u admin:admin123 -X DELETE http://localhost:8081/nexus/service/local/data_cache/repositories/example-proxy/content
Cache can also be expired implicitly for a repo when the remote URL of the repository is changed by an administrator.
6) Request the file from example-proxy again.
curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
Since the remote changed the file, it should be downloaded again to example-proxy local storage.
The response will be 403 quarantined - and the jar will permanently stay in this state
2018-11-08 09:57:11,252-0400 INFO [qtp685447609-454] anonymous com.sonatype.nexus.clm.internal.QuarantineRequestStrategy - Blocked serving of quarantined item example-proxy:/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar because quarantineStatus=PENDING
Mitigation of this Issue
We currently have no plans to change this in any version of Nexus 2. It is considered unusual that a remote would serve the same file at the same coordinates but report a different last modified time, forcing Nexus to resave the file.
If Nexus 2 is using an HTTP proxy for outbound requests, it is important to verify that the this HTTP proxy server is not manipulating the Last-Modified headers being returned from the remote. If it does, this could trigger the problem.
Long term: Upgrade to Nexus 3 where this problem does not happen if the remote reports a newer file.
Short term do either:
a) avoid explicitly or implicitly Expiring Cache on a proxy repository with quarantine enabled.
b) avoid configuring a RELEASE policy repository with artifact max age or item max age not equal to -1 .
Emergency Fix for Failing Builds Receiving 403
An administrator can navigate to the affected artifact under Browse Storage tab for the repository. On the Artifact tab, use the Delete button to delete the artifact from storage. Upon the next request for the item, the item will be evaluated by IQ Server and if allowed by policy, will no longer be quarantined.
Attachments
Issue Links
- relates
-
NEXUS-18742 Cached proxy repository artifacts will become permanently quarantined if a cache is expired on a proxy in Nexus Repo 2.x
-
- Closed
-