[NEXUS-18648] npm requests to proxy repositories using S3BlobStorage may start permanently failing with 500 - Sonatype JIRA

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Duplicate
Affects Version/s: 3.14.0
Fix Version/s: 3.15.0
Component/s: NPM, S3
Labels:
- supportant
- t-morpheus

Description

Standard NPM package metadata and tgz requests to a group repository containing an npm proxy repo using S3 storage against the official registry may start returning 500 status responses. The initial caching of the npm packages works fine, then later the blobs go missing and nexus responds with 500 for every npm package metadata request and related package tgz requests.

The log messages suggest the the database is referencing a blob which can't be found in the S3 bucket and can't be fetched from the remote anymore either.

Workaround

For users running 3.14.0-04 there is a patch to recover automatically from the suddenly missing npm package roots

https://issues.sonatype.org/browse/NEXUS-17808?focusedCommentId=570003&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-570003

The exact cause of the missing package root and the resulting problems retrieving the related tar files is not clear yet, but one customer has found the deleting the entire npm proxy repo and adding it back to the group repo will also resolve the issue temporarily as the packages are not initially cached. Then later when the same packages are requested, those requests will start failing.

If you have Nexus firewall enabled for your NPM proxy repository, we do not recommend deleting it as a workaround unless you are prepared to lose waiver information in IQ server for it.

Symptoms

2018-11-28 14:20:48,547+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:48,581+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:48,662+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:48,725+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:48,840+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:49,073+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:49,405+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:50,142+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:51,935+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:51,936+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8
2018-11-28 14:20:51,937+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl - Unable to find blob default-s3-blob-store@2E30A8FD-268BE9C9-D375BC72-A9508E1A-B6D08961:2a0cda89-fd0c-42bc-a02f-59319e649680 for nodemon, will check remote
2018-11-28 14:20:51,992+0000 INFO  [qtp1298682466-402]  admin org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for npm-proxy changed from READY to AVAILABLE - reason n/a for n/a
2018-11-28 14:20:52,307+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,332+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,374+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,432+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,535+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,704+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:52,974+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:53,619+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:54,621+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties})
2018-11-28 14:20:54,622+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8
2018-11-28 14:20:54,665+0000 WARN  [qtp1298682466-402]  admin org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/npm-repo/nodemon
org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
 at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:226)
 at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
 at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
 at com.sun.proxy.$Proxy201.allowRetry(Unknown Source)
 at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
 at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
 at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.putPackageRoot(NpmProxyFacetImpl.java:232)
 at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.store(NpmProxyFacetImpl.java:134)
 at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:247)
 at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.lambda$1(ProxyFacetSupport.java:234)
 at org.sonatype.nexus.common.io.CooperatingFuture.performCall(CooperatingFuture.java:122)
 at com.sonatype.nexus.hazelcast.internal.io.DistributedCooperatingFuture.performCall(DistributedCooperatingFuture.java:50)
 at org.sonatype.nexus.common.io.CooperatingFuture.call(CooperatingFuture.java:64)
 at org.sonatype.nexus.common.io.ScopedCooperationFactorySupport$ScopedCooperation.cooperate(ScopedCooperationFactorySupport.java:99)
 at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:225)
 at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:50)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.Context$proceed$1.call(Unknown Source)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
 at org.sonatype.nexus.repository.npm.internal.NpmProxyRecipe$_closure1.doCall(NpmProxyRecipe.groovy:186)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
 at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1099)
 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
 at groovy.lang.Closure.call(Closure.java:418)
 at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
 at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
 at com.sun.proxy.$Proxy161.handle(Unknown Source)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:56)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.npm.internal.NpmHandlers$1.handle(NpmHandlers.java:128)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
 at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64)
 at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
 at org.sonatype.nexus.repository.group.GroupHandler.getAll(GroupHandler.java:181)
 at org.sonatype.nexus.repository.group.GroupHandler.getAll(GroupHandler.java:149)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
 at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
 at org.sonatype.nexus.repository.npm.internal.NpmGroupPackageHandler.doGet(NpmGroupPackageHandler.groovy:75)
 at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:93)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
 at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
 at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
 at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64)
 at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
 at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
 at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:210)
 at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:172)
 at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
 at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
 at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
 at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
 at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
 at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
 at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
 at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
 at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
 at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
 at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
 at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
 at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
 at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
 at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
 at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
 at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
 at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
 at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.Server.handle(Server.java:531)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
 at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default-s3-blob-store@2E30A8FD-268BE9C9-D375BC72-A9508E1A-B6D08961:2a0cda89-fd0c-42bc-a02f-59319e649680 exists in metadata, but is missing from the blobstore
 at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:950)
 at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
 at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
 at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
 at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
 at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
 at com.sun.proxy.$Proxy201.requireBlob(Unknown Source)
 at org.sonatype.nexus.repository.npm.internal.NpmFacetUtils.loadPackageRoot(NpmFacetUtils.java:318)
 at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.mergeNewRootWithExistingRoot(NpmProxyFacetImpl.java:271)
 at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.doPutPackageRoot(NpmProxyFacetImpl.java:251)
 at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
 ... 159 common frames omitted

Attachments

Issue Links

duplicates

NEXUS-20197 Requests to npm to proxy repositories backed by s3 blob stores in HA clusters sometimes start failing with 500 responses

Closed

testing discovered

NEXUS-18675 log blob deleted reason for s3 based deleted blobs

Closed

Activity

People

Assignee:: Andrew Taylor [X] (Inactive)

Reporter:: Peter Lynch

Last Updated By:: Peter Lynch

Team:: NXRM - Morpheus

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 12/04/18 09:16 PM

Updated:: 06/14/19 08:24 PM

Resolved:: 03/26/19 07:38 PM

Date of First Response:: 10/Dec/18 6:12 PM