Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-28266

make S3 Connection Time to Live (TTL) setting configurable to help avoid socket connect timeouts

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.0.0, 3.30.0
    • Fix Version/s: 3.34.0
    • Component/s: S3
    • Notability:
      2

      Description

      Problem

      If a connection is cached inside the S3 connection pool for re-use, then the IP address mapped to an S3 bucket name is also cached. Since last use of the connection the IP address mapped to the S3 bucket name may have changed ( expected in AWS environment). When the connection is attempted to be re-used by NXRM, a socket connect timeout error may occur trying to establish a connection to the now defunct IP address associated with the re-used connection. These socket connect timeout errors lead to instability accessing blobs stored in the S3 blobstore and potentially build failures.

      Caused by: java.net.SocketTimeoutException: connect timed out
       at java.net.PlainSocketImpl.socketConnect(Native Method)
       at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
       at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:589)
       at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:368)
       at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:142)
       at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
       ... 167 common frames omitted
      

      Detail

      NXRM 3 presently uses AWS SDK v1 to communicate with any configured S3 bucket.

      AWS recommends that the the Java system property networkaddress.cache.ttl be set on the JVM of clients to possibly a lower than default value to help deal with DNS name resolution of S3 bucket names changing IP address.

      https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html

      The Java virtual machine (JVM) caches DNS name lookups. When the JVM resolves a hostname to an IP address, it caches the IP address for a specified period of time, known as the time-to-live (TTL).

      Because AWS resources use DNS name entries that occasionally change, we recommend that you configure your JVM with a TTL value of no more than 60 seconds. This ensures that when a resource’s IP address changes, your application will be able to receive and use the resource’s new IP address by requerying the DNS.

      NXRM presently has this property set to 3600 seconds inside <app-dir>/etc/karaf/system.properties

      To override this value, NXRM administrators can add this line

      -Dnetworkaddress.cache.ttl=60
      

      to <app-dir>/bin/nexus.vmoptions file.

      However, the HTTP connections made to S3 are configurable via Java APIs of the SDK. A connection pool is used to re-use connections. One of the connection options is called "Connection Time to Live (TTL)" described here:

      https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html#http-transport-configuration

      Connection Time to Live (TTL)

      By default, the SDK will attempt to reuse HTTP connections as long as possible. In failure situations where a connection is established to a server that has been brought out of service, having a finite TTL can help with application recovery. For example, setting a 15 minute TTL will ensure that even if you have a connection established to a server that is experiencing issues, you’ll reestablish a connection to a new server within 15 minutes.

      To set the HTTP connection TTL, use the ClientConfiguration.setConnectionTTL method.

      The JavaDoc states:

      Returns the expiration time (in milliseconds) for a connection in the connection pool. When
       * retrieving a connection from the pool to make a request, the total time that the connection
       * has been open is compared against this value. Connections which have been open for longer are
       * discarded, and if needed a new connection is created.
       * <p>
       * Tuning this setting down (together with an appropriately-low setting for Java's DNS cache
       * TTL) ensures that your application will quickly rotate over to new IP addresses when the
       * service begins announcing them through DNS, at the cost of having to re-establish new
       * connections more frequently.
      
      When a connection is
       * retrieved from the connection pool, this parameter is checked to see if the connection can be
       * reused.
      

      NXRM does not expose a way to have the AWS SDK v1 Connection TTL customized from its default value of ( -1 ) never expire.

      Expected

      Provide an NXRM administrator option to adjust the Connection Time to Live (TTL) value for S3 blobstore connections.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Joe Tom Joe Tom
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title