Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-12798

Attempt to access soft-deleted blob

    XMLWordPrintable

    Details

    • Sprint:
      Formats/Core Team - Sprint 93, Core Team - Sprint 94, Core Team - Sprint 95, Core Team - Sprint 96, Core Team - Sprint 97

      Description

      This is started happening on the 2017-03-27. This caused "Internal Server 500" error when requesting a components. Removing “deleted=true” from the blob property file does resolve the issue as compact blobstore has not been run.

      This jira is for determining root cause of why Nexus is trying to access soft-deleted blobs.

      2017-03-27 18:50:33,943-0400 WARN  [qtp1318557382-437] RC87740 org.sonatype.nexus.blobstore.file.internal.FileBlobStore - Attempt to access soft-deleted blob 1739f6c0-179d-4d91-8a68-680cebdb733a (/tech/appl/default/nexusservice/software/nexusrepo/sonatype-work/nexus3/blobs/default/content/vol-26/chap-31/1739f6c0-179d-4d91-8a68-680cebdb733a.properties)
      2017-03-27 18:50:33,944-0400 WARN  [qtp1318557382-437] RC87740 org.sonatype.nexus.repository.storage.StorageTxImpl - Reached max retries: 8/8
      2017-03-27 18:50:33,957-0400 WARN  [qtp1318557382-437] RC87740 org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Service failure
      org.sonatype.nexus.repository.storage.RetryDeniedException: Reached max retries: 8/8
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:212) [na:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.8.0_102]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [na:1.8.0_102]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [na:1.8.0_102]
      	at java.lang.reflect.Method.invoke(Method.java:498) [na:1.8.0_102]
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) [na:na]
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69) [na:na]
      	at com.sun.proxy.$Proxy174.allowRetry(Unknown Source) [na:na]
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81) [na:na]
      	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54) [na:na]
      	at org.sonatype.nexus.repository.maven.internal.proxy.MavenProxyFacet.getCachedContent(MavenProxyFacet.java:58) [na:na]
      	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:162) [na:na]
      	at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:49) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.maven.internal.VersionPolicyHandler.handle(VersionPolicyHandler.java:60) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:50) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at com.sonatype.nexus.clm.internal.QuarantineContributedHandler.handle(QuarantineContributedHandler.java:62) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:43) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.Context.start(Context.java:114) [na:na]
      	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63) [na:na]
      	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) [na:na]
      	at org.sonatype.nexus.repository.group.GroupHandler.getFirst(GroupHandler.java:118) [na:na]
      	at org.sonatype.nexus.repository.group.GroupHandler.doGet(GroupHandler.java:95) [na:na]
      	at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:79) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:55) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at com.sonatype.nexus.clm.internal.QuarantineContributedHandler.handle(QuarantineContributedHandler.java:62) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:43) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46) [na:na]
      	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80) [na:na]
      	at org.sonatype.nexus.repository.view.Context.start(Context.java:114) [na:na]
      	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:63) [na:na]
      	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52) [na:na]
      	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43) [na:na]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:197) [na:na]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:159) [na:na]
      	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:116) [na:na]
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api:3.1.0]
      	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [com.google.inject:4.1.0]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112) [org.apache.shiro.web:1.3.2]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [org.apache.shiro.web:1.3.2]
      	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85) [org.sonatype.nexus.security:3.2.1.01]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [org.apache.shiro.web:1.3.2]
      	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [org.apache.shiro.core:1.3.2]
      	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [org.apache.shiro.core:1.3.2]
      	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [org.apache.shiro.core:1.3.2]
      	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [org.apache.shiro.web:1.3.2]
      	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101) [org.sonatype.nexus.security:3.2.1.01]
      	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [org.apache.shiro.web:1.3.2]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71) [org.sonatype.nexus.plugins.nexus-repository-httpbridge:3.2.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:112) [com.sonatype.nexus.plugins.nexus-licensing-plugin:3.2.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97) [com.codahale.metrics.servlet:3.0.2]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68) [org.sonatype.nexus.base:3.2.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102) [org.sonatype.nexus.base:3.2.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98) [org.sonatype.nexus.base:3.2.1.01]
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) [com.google.inject:4.1.0]
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [com.google.inject:4.1.0]
      	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) [org.sonatype.nexus.bootstrap:3.2.1.01]
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [org.eclipse.jetty.security:9.3.7.v20160115]
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) [org.eclipse.jetty.servlet:9.3.7.v20160115]
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175) [com.codahale.metrics.jetty9:3.0.2]
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.Server.handle(Server.java:517) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242) [org.eclipse.jetty.server:9.3.7.v20160115]
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75) [org.eclipse.jetty.io:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) [org.eclipse.jetty.util:9.3.7.v20160115]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
      Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@C531886F-C150FBC7-7D02F53E-B15BA592-10937A64:1739f6c0-179d-4d91-8a68-680cebdb733a exists in metadata, but is missing from the blobstore
      	at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:746) [na:na]
      	at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) [na:na]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [na:1.8.0_102]
      	at java.lang.reflect.Method.invoke(Method.java:498) [na:1.8.0_102]
      	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53) [na:na]
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [na:na]
      	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270) [na:na]
      	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53) [na:na]
      	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63) [na:na]
      	at com.sun.proxy.$Proxy174.requireBlob(Unknown Source) [na:na]
      	at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.get(MavenFacetImpl.java:182) [na:na]
      	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) [na:na]
      	... 126 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mprescott Michael Prescott
              Reporter:
              msurani Mahendra Surani
              Last Updated By:
              Peter Lynch
              Votes:
              3 Vote for this issue
              Watchers:
              17 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title