Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: 3.14.0
-
Fix Version/s: 3.15.0
-
Labels:
Description
Standard NPM package metadata and tgz requests to a group repository containing an npm proxy repo using S3 storage against the official registry may start returning 500 status responses. The initial caching of the npm packages works fine, then later the blobs go missing and nexus responds with 500 for every npm package metadata request and related package tgz requests.
The log messages suggest the the database is referencing a blob which can't be found in the S3 bucket and can't be fetched from the remote anymore either.
Workaround
For users running 3.14.0-04 there is a patch to recover automatically from the suddenly missing npm package roots
The exact cause of the missing package root and the resulting problems retrieving the related tar files is not clear yet, but one customer has found the deleting the entire npm proxy repo and adding it back to the group repo will also resolve the issue temporarily as the packages are not initially cached. Then later when the same packages are requested, those requests will start failing.
If you have Nexus firewall enabled for your NPM proxy repository, we do not recommend deleting it as a workaround unless you are prepared to lose waiver information in IQ server for it.
Symptoms
2018-11-28 14:20:48,547+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:48,581+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:48,662+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:48,725+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:48,840+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:49,073+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:49,405+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:50,142+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:51,935+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:51,936+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8 2018-11-28 14:20:51,937+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl - Unable to find blob default-s3-blob-store@2E30A8FD-268BE9C9-D375BC72-A9508E1A-B6D08961:2a0cda89-fd0c-42bc-a02f-59319e649680 for nodemon, will check remote 2018-11-28 14:20:51,992+0000 INFO [qtp1298682466-402] admin org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for npm-proxy changed from READY to AVAILABLE - reason n/a for n/a 2018-11-28 14:20:52,307+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,332+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,374+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,432+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,535+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,704+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:52,974+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:53,619+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:54,621+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.blobstore.s3.internal.S3BlobStore - Attempt to access soft-deleted blob 2a0cda89-fd0c-42bc-a02f-59319e649680 (S3PropertiesFile{bucket=sc-default-repo, key=content/vol-28/chap-10/2a0cda89-fd0c-42bc-a02f-59319e649680.properties}) 2018-11-28 14:20:54,622+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8 2018-11-28 14:20:54,665+0000 WARN [qtp1298682466-402] admin org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/npm-repo/nodemon org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8 at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:226) at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69) at com.sun.proxy.$Proxy201.allowRetry(Unknown Source) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81) at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.putPackageRoot(NpmProxyFacetImpl.java:232) at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.store(NpmProxyFacetImpl.java:134) at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:247) at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.lambda$1(ProxyFacetSupport.java:234) at org.sonatype.nexus.common.io.CooperatingFuture.performCall(CooperatingFuture.java:122) at com.sonatype.nexus.hazelcast.internal.io.DistributedCooperatingFuture.performCall(DistributedCooperatingFuture.java:50) at org.sonatype.nexus.common.io.CooperatingFuture.call(CooperatingFuture.java:64) at org.sonatype.nexus.common.io.ScopedCooperationFactorySupport$ScopedCooperation.cooperate(ScopedCooperationFactorySupport.java:99) at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:225) at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:50) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.Context$proceed$1.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at org.sonatype.nexus.repository.npm.internal.NpmProxyRecipe$_closure1.doCall(NpmProxyRecipe.groovy:186) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1099) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) at groovy.lang.Closure.call(Closure.java:418) at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) at com.sun.proxy.$Proxy161.handle(Unknown Source) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:56) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.npm.internal.NpmHandlers$1.handle(NpmHandlers.java:128) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.Context.start(Context.java:114) at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) at org.sonatype.nexus.repository.group.GroupHandler.getAll(GroupHandler.java:181) at org.sonatype.nexus.repository.group.GroupHandler.getAll(GroupHandler.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185) at org.sonatype.nexus.repository.npm.internal.NpmGroupPackageHandler.doGet(NpmGroupPackageHandler.groovy:75) at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:93) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) at org.sonatype.nexus.repository.view.Context.start(Context.java:114) at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:210) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:172) at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default-s3-blob-store@2E30A8FD-268BE9C9-D375BC72-A9508E1A-B6D08961:2a0cda89-fd0c-42bc-a02f-59319e649680 exists in metadata, but is missing from the blobstore at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:950) at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272) at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) at com.sun.proxy.$Proxy201.requireBlob(Unknown Source) at org.sonatype.nexus.repository.npm.internal.NpmFacetUtils.loadPackageRoot(NpmFacetUtils.java:318) at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.mergeNewRootWithExistingRoot(NpmProxyFacetImpl.java:271) at org.sonatype.nexus.repository.npm.internal.NpmProxyFacetImpl.doPutPackageRoot(NpmProxyFacetImpl.java:251) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) ... 159 common frames omitted
Attachments
Issue Links
- duplicates
-
NEXUS-20197 Requests to npm to proxy repositories backed by s3 blob stores in HA clusters sometimes start failing with 500 responses
-
- Closed
-
- testing discovered
-
NEXUS-18675 log blob deleted reason for s3 based deleted blobs
-
- Closed
-