Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16993

not running Compact Blobstore may lead to noisy 'IOException An attempt was made to move the file pointer before the beginning of the file'

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.10.0
    • Fix Version/s: None
    • Component/s: Blobstore
    • Labels:
    • Environment:
      Windows
    • Story Points:
      3
    • Notability:
      3

      Description

      A customer had 9 custom blobstores, no Compact blobstore tasks scheduled and after upgrading to 3.10.0 started noticing 1GB sized application log files ( nexus.log) filled with these messages:

      2018-05-04 13:16:16,980+0200 WARN  [qtp308051675-71869]  *UNKNOWN org.sonatype.nexus.repository.storage.BlobTx - Unable to delete old blob MavenInternal@21FE55D8-20A18D19-9FAA6F72-94D23602-D21ED468:3e775ec3-7c72-48c6-aebb-d1ad4aada2da while committing transaction
org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: 3e775ec3-7c72-48c6-aebb-d1ad4aada2da, java.io.IOException: An attempt was made to move the file pointer before the beginning of the file
	at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:492)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:270)
	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
	at org.sonatype.nexus.repository.storage.BlobTx.commit(BlobTx.java:156)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:182)
	at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
	at com.sun.proxy.$Proxy201.commit(Unknown Source)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
	at org.sonatype.nexus.repository.maven.internal.MavenFacetUtils.deleteWithHashes(MavenFacetUtils.java:267)
	at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.evictCache(MavenGroupFacet.java:211)
	at org.sonatype.nexus.repository.maven.internal.group.MavenGroupFacet.on(MavenGroupFacet.java:225)
	at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:91)
	at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:76)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
	at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:71)
	at com.google.common.eventbus.Dispatcher$ImmediateDispatcher.dispatch(Dispatcher.java:200)
	at com.google.common.eventbus.EventBus.post(EventBus.java:217)
	at org.sonatype.nexus.internal.event.EventManagerImpl.post(EventManagerImpl.java:126)
	at org.sonatype.nexus.orient.entity.EntityHook.postEvents(EntityHook.java:269)
	at org.sonatype.nexus.orient.entity.EntityHook.flushEvents(EntityHook.java:245)
	at org.sonatype.nexus.orient.entity.EntityHook.onAfterTxCommit(EntityHook.java:166)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2963)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2884)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.commit(StorageTxImpl.java:181)
	at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:66)
	at com.sun.proxy.$Proxy201.commit(Unknown Source)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:67)
	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
	at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.doGet(HostedHandler.java:75)
	at org.sonatype.nexus.repository.maven.internal.hosted.HostedHandler.handle(HostedHandler.java:61)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:44)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.maven.internal.VersionPolicyHandler.handle(VersionPolicyHandler.java:61)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.view.handlers.ExceptionHandler.handle(ExceptionHandler.java:44)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:52)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:46)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:80)
	at org.sonatype.nexus.repository.view.Context.start(Context.java:114)
	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:64)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:211)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:173)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:102)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:530)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:289)
	at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:149)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: An attempt was made to move the file pointer before the beginning of the file
	at java.io.RandomAccessFile.setLength(Native Method)
	at com.squareup.tape.QueueFile.setLength(QueueFile.java:406)
	at com.squareup.tape.QueueFile.expandIfNecessary(QueueFile.java:375)
	at com.squareup.tape.QueueFile.add(QueueFile.java:305)
	at com.squareup.tape.QueueFile.add(QueueFile.java:287)
	at org.sonatype.nexus.blobstore.file.FileBlobStore.delete(FileBlobStore.java:486)
	... 151 common frames omitted

      Expected

      Handling this better in the future - ie. continuing but with potentially reduced features (such as read-only mode) to avoid further corruption + clear logging so customer knows how to fix it. Be smarter about reclaiming disk with compact blobstore tasks especially when a system contains many blobstores.

        Attachments

          Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-16875 deletions.index file limited to max 1G filesize

          • Medium - Medium
          • Open

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Rich Seddon Rich Seddon
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title