Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-20152

S3 Blob Store: Timeout waiting for connection from pool

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.19.1, 3.29.2
    • Fix Version/s: 3.31.0
    • Component/s: S3
    • Labels:

      Description

      Storing Docker images in the s3 blob store and requesting them occasionally results in a ConnectionPoolTimeoutException. 

       

      Stacktrace:

      2019-06-12 00:42:46,660+0000 WARN [qtp1142135953-29965] anonymous org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/tubular-docker-private/v2/spark/spark-16-04/blobs/sha256:e8f8d183f54eaf2574ad1657bdce26d5d75502c37c8e727487b387c37ccb9f62
      com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1113)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1063)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
       at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247)
       at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194)
       at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1398)
       at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1259)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.getInputStream(S3BlobStore.java:532)
       at org.sonatype.nexus.repository.view.payloads.BlobPayload.openInputStream(BlobPayload.java:44)
       at org.sonatype.nexus.repository.view.Content.openInputStream(Content.java:95)
       at org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender.send(DefaultHttpResponseSender.java:80)
       at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:228)
       at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:174)
       at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
       at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
       at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
       at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
       at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
       at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
       at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
       at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
       at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
       at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
       at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
       at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
       at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
       at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
       at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
       at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
       at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
       at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
       at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
       at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
       at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
       at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
       at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
       at org.eclipse.jetty.server.Server.handle(Server.java:502)
       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
       at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:313)
       at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
       at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
       at com.amazonaws.http.conn.$Proxy229.get(Unknown Source)
       at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
       at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
       at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
       at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
       at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1235)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)
       ... 100 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              c4urself Christian
              Last Updated By:
              Joe Tom Joe Tom
              Votes:
              22 Vote for this issue
              Watchers:
              28 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title