cached proxy repository artifacts will become permanently quarantined if a newer file is downloaded from remote

Description

Given that the Nexus Firewall Quarantining feature is enabled for a proxy repository in Nexus 2, if these events occur, then an artifact can appear permanently stuck in PENDING quarantine status and will not be served by Nexus:

• cache an artifact file in proxy repo - Nexus serves this file when requested
• enable quarantine for proxy repo
• Nexus downloads a new version of the proxied artifact from remote
• Nexus permanently reports the file as quarantined ( status=PENDING ) and returns 403 for all future requests as long as quarantine is enabled. Nexus will never check the artifact against IQ Server repository policies.

Example Reproduce

Setup

Configure IQ Server 1.50 - http://localhost:8070
No repository policies are needed for this test.

Repository Manager 2.14.4 ( or any Nexus 2.x version ) - http://localhost:8081/nexus

1) Configure a working IQ Server connection from NXRM to IQ server.

2) Create Repositories in Nexus 2

Maven 2 Proxy with default settings for RELEASE policy, including artifact max age -1
example-proxy - (http://localhost:8081/nexus/content/repositories/example-proxy/) proxies http://localhost:8081/nexus/content/repositories/example-hosted/

Maven 2 Hosted with Deployment Policy Allow Redeploy
example-hosted - (http://localhost:8081/nexus/content/repositories/example-hosted/)

3) Set the following loggers in NXRM to help verify outbound activity.
remote.storage.outbound to level DEBUG
org.apache.http to level DEBUG

4) Obtain an artifact file to test with, that would be identified by IQ server if it were evaluated:

curl -v -L http://jcenter.bintray.com/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o spring-security-web-3.1.0.RELEASE.jar

spring-security-web-3.1.0.RELEASE.jar

Reproduce

1. upload a file into example-hosted

curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar

2. Download the file from example-proxy

curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null

3) Enable quarantine on example-proxy

Administration -> Capabilities
Add new
Audit and Quarantine: example-proxy

By design this action by itself will not quarantine already cache jars.
By design, new requests for a proxy cached file should never trigger a remote check of the jar, unless

• someone expires cache on the proxy repository ( or group containing it )
• someone deletes the cached jar file out-of-band from disk storage

4) Verify download is still successful (200) and does not trigger any remote checks for the new file.

curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null

5) Upload new file into hosted repo at same coordinates
First check the last modified header on example-hosted for the jar download

curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null

Wait at least 1 minute- Then upload a replacement:

curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE-new.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar

Verify the last modified header has changed from the initial value.

curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null

5) Expire cache on example-proxy repo to force all new inbound requests to check remote for updates. This example curl command is the same operation that happens when a user right-clicks the repository in the UI and chooses "Expire Cache"

curl -v -u admin:admin123 -X DELETE http://localhost:8081/nexus/service/local/data_cache/repositories/example-proxy/content

Cache can also be expired implicitly for a repo when the remote URL of the repository is changed by an administrator.

6) Request the file from example-proxy again.

curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null

Since the remote changed the file, it should be downloaded again to example-proxy local storage.

The response will be 403 quarantined - and the jar will permanently stay in this state

2018-11-08 09:57:11,252-0400 INFO  [qtp685447609-454] anonymous com.sonatype.nexus.clm.internal.QuarantineRequestStrategy - Blocked serving of quarantined item example-proxy:/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar because quarantineStatus=PENDING

Mitigation of this Issue

We currently have no plans to change this in any version of Nexus 2. It is considered unusual that a remote would serve the same file at the same coordinates but report a different last modified time, forcing Nexus to resave the file.

If Nexus 2 is using an HTTP proxy for outbound requests, it is important to verify that the this HTTP proxy server is not manipulating the Last-Modified headers being returned from the remote. If it does, this could trigger the problem.

Long term: Upgrade to Nexus 3 where this problem does not happen if the remote reports a newer file.

Short term do either:
a) avoid explicitly or implicitly Expiring Cache on a proxy repository with quarantine enabled.
b) avoid configuring a RELEASE policy repository with artifact max age or item max age not equal to -1 .

Emergency Fix for Failing Builds Receiving 403

An administrator can navigate to the affected artifact under Browse Storage tab for the repository. On the Artifact tab, use the Delete button to delete the artifact from storage. Upon the next request for the item, the item will be evaluated by IQ Server and if allowed by policy, will no longer be quarantined.