Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-32449

blobstore outbound http connection or file handle leak while rebuilding Helm index

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.38.0, 3.38.1
    • Fix Version/s: 3.42.0
    • Component/s: helm, S3
    • Notability:
      3

      Description

      A customer had an S3 blobstore configured for their helm repositories:

      Only requests related to helm repositories were causing failures, other repo formats with their own blobstore were not experiencing problems.

      There is a leak of either HTTP connections for S3 / Azure blobstores, or file handles for File based blobstores.

      Example for S3 blobstore:

      2022-05-25 10:41:47,881+0000 ERROR [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Error while rebuilding Helm index for repository helm-public
      com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      

      Full messages:

      2022-05-25 10:40:04,747+0000 INFO  [event-27-thread-12208]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:05,747+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:06,330+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:07,248+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:08,248+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:08,796+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:09,630+0000 INFO  [event-27-thread-12208]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:10,630+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:11,267+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:12,065+0000 INFO  [event-27-thread-12209]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:13,065+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:13,698+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:14,220+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:15,220+0000 INFO  [event-27-thread-12208]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:15,833+0000 INFO  [event-27-thread-12208]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:16,321+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:17,321+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:17,849+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:18,401+0000 INFO  [event-27-thread-12209]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:19,402+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:20,004+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:20,449+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:21,450+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:22,025+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:23,980+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:24,980+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:40:25,600+0000 INFO  [event-27-thread-12205]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Finished rebuilding Helm index for repository helm-public
      2022-05-25 10:40:25,900+0000 INFO  [event-27-thread-12207]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Scheduling rebuild of Helm metadata to start in 1 seconds
      2022-05-25 10:40:26,900+0000 INFO  [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Rebuilding Helm index for repository helm-public
      2022-05-25 10:41:07,921+0000 ERROR [periodic-4-thread-1]  *SYSTEM org.sonatype.nexus.blobstore.s3.internal.S3BlobStoreMetricsStore - Cannot write blob store metrics
      com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
      	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:394)
      	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5950)
      	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1812)
      	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1772)
      	at com.amazonaws.services.s3.EncryptingAmazonS3Client.putObject(EncryptingAmazonS3Client.java:157)
      	at com.amazonaws.services.s3.EncryptingAmazonS3Client.putObject(EncryptingAmazonS3Client.java:150)
      	at org.sonatype.nexus.blobstore.s3.internal.S3PropertiesFile.write(S3PropertiesFile.java:84)
      	at org.sonatype.nexus.blobstore.s3.internal.S3PropertiesFile.write(S3PropertiesFile.java:1)
      	at org.sonatype.nexus.blobstore.CloudBlobPropertiesSupport.store(CloudBlobPropertiesSupport.java:55)
      	at org.sonatype.nexus.blobstore.BlobStoreMetricsStoreSupport.flushProperties(BlobStoreMetricsStoreSupport.java:173)
      	at org.sonatype.nexus.blobstore.BlobStoreMetricsStoreSupport.lambda$0(BlobStoreMetricsStoreSupport.java:134)
      	at org.sonatype.nexus.scheduling.internal.PeriodicJobServiceImpl.lambda$1(PeriodicJobServiceImpl.java:90)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
      	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
      	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
      	at com.amazonaws.http.conn.$Proxy343.get(Unknown Source)
      	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
      	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
      	... 28 common frames omitted
      2022-05-25 10:41:47,881+0000 ERROR [event-27-thread-12206]  svc.rit-nexus-all com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl - Error while rebuilding Helm index for repository helm-public
      com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
      	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
      	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
      	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
      	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
      	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:692)
      	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
      	at org.sonatype.nexus.repository.mime.DefaultContentValidator.determineContentType(DefaultContentValidator.java:71)
      	at org.sonatype.nexus.repository.content.facet.AssetBlobValidators.lambda$0(AssetBlobValidators.java:61)
      	at org.sonatype.nexus.repository.content.facet.ContentFacetSupport.determineContentType(ContentFacetSupport.java:326)
      	at org.sonatype.nexus.repository.content.facet.ContentFacetSupport.checkContentType(ContentFacetSupport.java:311)
      	at org.sonatype.nexus.repository.content.fluent.internal.FluentAssetBuilderImpl.makePermanent(FluentAssetBuilderImpl.java:211)
      	at org.sonatype.nexus.repository.content.fluent.internal.FluentAssetBuilderImpl.lambda$0(FluentAssetBuilderImpl.java:106)
      	at org.sonatype.nexus.repository.content.fluent.internal.FluentAssetBuilderImpl.save(FluentAssetBuilderImpl.java:136)
      	at com.sonatype.repository.helm.datastore.internal.recipe.HelmContentFacetImpl.putIndex(HelmContentFacetImpl.java:104)
      	at com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl.createIndexYaml(CreateIndexFacetImpl.java:180)
      	at com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl.updateIndexYaml(CreateIndexFacetImpl.java:173)
      	at com.sonatype.repository.helm.datastore.internal.createindex.CreateIndexFacetImpl.on(CreateIndexFacetImpl.java:158)
      	at sun.reflect.GeneratedMethodAccessor637.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
      	at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:144)
      	at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
      		at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
      		at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
      		at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
      		at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
      		at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1338)
      		at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1312)
      		at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1393)
      		at org.sonatype.nexus.blobstore.s3.internal.S3PropertiesFile.exists(S3PropertiesFile.java:94)
      		at org.sonatype.nexus.blobstore.s3.internal.S3BlobAttributes.load(S3BlobAttributes.java:44)
      		at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.exists(S3BlobStore.java:827)
      		at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:26)
      		at org.sonatype.nexus.blobstore.group.BlobStoreGroup.lambda$10(BlobStoreGroup.java:268)
      		at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
      		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
      		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
      		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      		at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
      		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      		at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
      		at org.sonatype.nexus.blobstore.group.BlobStoreGroup.deleteHard(BlobStoreGroup.java:269)
      		at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      		at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      		at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
      		at org.sonatype.nexus.blobstore.api.BlobStore.deleteIfTemp(BlobStore.java:342)
      		at org.sonatype.nexus.repository.view.payloads.TempBlob.close(TempBlob.java:99)
      		at com.sonatype.repository.helm.datastore.internal.recipe.HelmContentFacetImpl.putIndex(HelmContentFacetImpl.java:108)
      		... 15 common frames omitted
      	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
      		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
      		at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
      		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		at java.lang.reflect.Method.invoke(Method.java:498)
      		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
      		at com.amazonaws.http.conn.$Proxy343.get(Unknown Source)
      		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
      		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
      		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330)
      		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
      		... 46 common frames omitted
      Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
      	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
      	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
      	at com.amazonaws.http.conn.$Proxy343.get(Unknown Source)
      	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
      	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
      	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1330)
      	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
      	... 37 common frames omitted
      
      

      Expected

      Do not leak S3 blobstore connections or file handles for Helm Repositories.

      Workaround

      To clear the leak, a restart of Nexus Repository is required.

      If the leak surfaces after a long period of time, then more frequent restarts might help avoid the issue, along with increasing the S3 default connection pool size.

      Until the problem is fixed, then not using S3 blobstores for Helm format repositories might be preferred.

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Eugene Bulatnikov Eugene Bulatnikov
            Team:
            NXRM - Sentinels
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title