If a connection is cached inside the S3 connection pool for re-use, then the IP address mapped to an S3 bucket name is also cached. Since last use of the connection the IP address mapped to the S3 bucket name may have changed ( expected in AWS environment). When the connection is attempted to be re-used by NXRM, a socket connect timeout error may occur trying to establish a connection to the now defunct IP address associated with the re-used connection. These socket connect timeout errors lead to instability accessing blobs stored in the S3 blobstore and potentially build failures.
NXRM 3 presently uses AWS SDK v1 to communicate with any configured S3 bucket.
AWS recommends that the the Java system property networkaddress.cache.ttl be set on the JVM of clients to possibly a lower than default value to help deal with DNS name resolution of S3 bucket names changing IP address.
The Java virtual machine (JVM) caches DNS name lookups. When the JVM resolves a hostname to an IP address, it caches the IP address for a specified period of time, known as the time-to-live (TTL).
Because AWS resources use DNS name entries that occasionally change, we recommend that you configure your JVM with a TTL value of no more than 60 seconds. This ensures that when a resource’s IP address changes, your application will be able to receive and use the resource’s new IP address by requerying the DNS.
NXRM presently has this property set to 3600 seconds inside <app-dir>/etc/karaf/system.properties
To override this value, NXRM administrators can add this line
to <app-dir>/bin/nexus.vmoptions file.
However, the HTTP connections made to S3 are configurable via Java APIs of the SDK. A connection pool is used to re-use connections. One of the connection options is called "Connection Time to Live (TTL)" described here:
Connection Time to Live (TTL)
By default, the SDK will attempt to reuse HTTP connections as long as possible. In failure situations where a connection is established to a server that has been brought out of service, having a finite TTL can help with application recovery. For example, setting a 15 minute TTL will ensure that even if you have a connection established to a server that is experiencing issues, you’ll reestablish a connection to a new server within 15 minutes.
To set the HTTP connection TTL, use the ClientConfiguration.setConnectionTTL method.
The JavaDoc states:
NXRM does not expose a way to have the AWS SDK v1 Connection TTL customized from its default value of ( -1 ) never expire.
Provide an NXRM administrator option to adjust the Connection Time to Live (TTL) value for S3 blobstore connections.