Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-18352

cached proxy repository artifacts will become permanently quarantined if a newer file is downloaded from remote

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.13, 2.14.4, 2.14.10
    • Fix Version/s: 3.5.0
    • Component/s: Firewall
    • Labels:

      Description

      Description

      Given that the Nexus Firewall Quarantining feature is enabled for a proxy repository in Nexus 2, if these events occur, then an artifact can appear permanently stuck in PENDING quarantine status and will not be served by Nexus:

      • cache an artifact file in proxy repo - Nexus serves this file when requested
      • enable quarantine for proxy repo
      • Nexus downloads a new version of the proxied artifact from remote
      • Nexus permanently reports the file as quarantined ( status=PENDING ) and returns 403 for all future requests as long as quarantine is enabled. Nexus will never check the artifact against IQ Server repository policies.

      Example Reproduce

      Setup

      Configure IQ Server 1.50 - http://localhost:8070
      No repository policies are needed for this test.

      Repository Manager 2.14.4 ( or any Nexus 2.x version ) - http://localhost:8081/nexus

      1) Configure a working IQ Server connection from NXRM to IQ server.

      2) Create Repositories in Nexus 2

      Maven 2 Proxy with default settings for RELEASE policy, including artifact max age -1
      example-proxy - (http://localhost:8081/nexus/content/repositories/example-proxy/) proxies http://localhost:8081/nexus/content/repositories/example-hosted/

      Maven 2 Hosted with Deployment Policy Allow Redeploy
      example-hosted - (http://localhost:8081/nexus/content/repositories/example-hosted/)

      3) Set the following loggers in NXRM to help verify outbound activity.
      remote.storage.outbound to level DEBUG
      org.apache.http to level DEBUG

      4) Obtain an artifact file to test with, that would be identified by IQ server if it were evaluated:

      curl -v -L http://jcenter.bintray.com/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o spring-security-web-3.1.0.RELEASE.jar
      

      spring-security-web-3.1.0.RELEASE.jar

      Reproduce

      1. upload a file into example-hosted

      curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar
      

      2. Download the file from example-proxy

      curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
      

      3) Enable quarantine on example-proxy

      Administration -> Capabilities
      Add new
      Audit and Quarantine: example-proxy

      By design this action by itself will not quarantine already cache jars.
      By design, new requests for a proxy cached file should never trigger a remote check of the jar, unless

      • someone expires cache on the proxy repository ( or group containing it )
      • someone deletes the cached jar file out-of-band from disk storage

      4) Verify download is still successful (200) and does not trigger any remote checks for the new file.

      curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
      

      5) Upload new file into hosted repo at same coordinates
      First check the last modified header on example-hosted for the jar download

      curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
      

      Wait at least 1 minute- Then upload a replacement:

      curl -v -u admin:admin123 --upload-file spring-security-web-3.1.0.RELEASE-new.jar http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar
      

      Verify the last modified header has changed from the initial value.

      curl -v http://localhost:8081/nexus/content/repositories/example-hosted/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
      

      5) Expire cache on example-proxy repo to force all new inbound requests to check remote for updates. This example curl command is the same operation that happens when a user right-clicks the repository in the UI and chooses "Expire Cache"

      curl -v -u admin:admin123 -X DELETE http://localhost:8081/nexus/service/local/data_cache/repositories/example-proxy/content
      

      Cache can also be expired implicitly for a repo when the remote URL of the repository is changed by an administrator.

      6) Request the file from example-proxy again.

      curl -v http://localhost:8081/nexus/content/repositories/example-proxy/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar -o /dev/null
      

      Since the remote changed the file, it should be downloaded again to example-proxy local storage.

      The response will be 403 quarantined - and the jar will permanently stay in this state

      2018-11-08 09:57:11,252-0400 INFO  [qtp685447609-454] anonymous com.sonatype.nexus.clm.internal.QuarantineRequestStrategy - Blocked serving of quarantined item example-proxy:/org/springframework/security/spring-security-web/3.1.0.RELEASE/spring-security-web-3.1.0.RELEASE.jar because quarantineStatus=PENDING
      

      Mitigation of this Issue

      We currently have no plans to change this in any version of Nexus 2. It is considered unusual that a remote would serve the same file at the same coordinates but report a different last modified time, forcing Nexus to resave the file.

      If Nexus 2 is using an HTTP proxy for outbound requests, it is important to verify that the this HTTP proxy server is not manipulating the Last-Modified headers being returned from the remote. If it does, this could trigger the problem.

      Long term: Upgrade to Nexus 3 where this problem does not happen if the remote reports a newer file.

      Short term do either:
      a) avoid explicitly or implicitly Expiring Cache on a proxy repository with quarantine enabled.
      b) avoid configuring a RELEASE policy repository with artifact max age or item max age not equal to -1 .

      Emergency Fix for Failing Builds Receiving 403

      An administrator can navigate to the affected artifact under Browse Storage tab for the repository. On the Artifact tab, use the Delete button to delete the artifact from storage. Upon the next request for the item, the item will be evaluated by IQ Server and if allowed by policy, will no longer be quarantined.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Rich Seddon
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title