Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-29433

fetching identical group npm package metadata concurrently can result in uncooperative threads and increased blobstore and database load

Details

    • 2
    • 2

    Description

      NEXUS-28277 was a report where different cooperation keys for the same concurrent npm group package matadata requests could cause mis-cooperation when trying to merge the same npm package metadata. That issue was reported and fixed against an OrientDB backend.

      The implementation for databases such as PostgreSQL and H2 have a similar problem, where differing cooperation keys can still be used. Not only will this have overlapping threads trying to update the same package metadata, in the cases where the assets may be stored in an S3 blobstore, the additional threads used may inadvertently lead to exhaustion of the S3 HTTP connection pool.

      Symptoms

      Occurrences of INFO log messages such as

      com.sonatype.nexus.repository.content.npm.internal.NpmGroupDataFacet - Missing blob example-npm-group@9B37DE7A-4F9AEDEE-9347A374-D6DD7C75-7C7BD1D3:a14c56c3-fc81-4b5b-825d-99415aecd02e containing cached metadata /example, deleting asset and triggering rebuild.

      S3 connection pool exhaustion DEBUG log messages showing S3 connection pool exhausted, such as:

      org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route:

      Unknown macro: {s}

      ->https://example.aws.s3.endpoint:443][total available: 0; route allocated: 201 of 201; total allocated: 201 of 201]

      Expected

      Align the NPM group package metadata cooperation keys for identical npm group level package metadata to reduce the number of concurrent threads that may be trying to access and therefore merge the same metadata. Basically implement similar changes that NEXUS-28277 made, but for the new databases.

      Attachments

        Issue Links

          Activity

            People

              vhashock Vitalii Hashock
              plynch Peter Lynch
              Michael Oliverio Michael Oliverio
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                tigCommentSecurity.panel-title