Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19437

a blobstore in any other state than STARTED cannot be removed

    XMLWordPrintable

    Details

    • Notability:
      2

      Description

      A blobstore may enter a Lifecycle state other than STARTED and permanently be stuck in that state.

      An end user would like to remove the broken blobstore completely from the configuration database, provided that the following constraints are not in violation:

      • it is referenced by a repository
      • it is the only member of a group blob store
      • the database is frozen/read-only

      However lifecycle state of a blobstore is in memory only, not in the database. Once stuck in a bad state, it may still be visible in the UI and exposed for removal, but it cannot be deleted using normal procedures. The removal will error with a stack trace similar to:

      2019-03-19 13:38:22,585+0100 ERROR [qtp1557116923-415]  admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Blobstore.remove, java-method: org.sonatype.nexus.coreui.BlobStoreComponent.remove
      org.sonatype.nexus.common.stateguard.InvalidStateException: Invalid state: STOPPED; allowed: [STARTED]
       at org.sonatype.nexus.common.stateguard.StateGuard._ensure(StateGuard.java:115)
       at org.sonatype.nexus.common.stateguard.StateGuard.access$1(StateGuard.java:108)
       at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:187)
       at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
       at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.forceDelete(BlobStoreManagerImpl.java:272)

      2020-01-21 14:16:59,225+0000 ERROR [qtp224338229-1090] admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Blobstore.remove, java-method: org.sonatype.nexus.coreui.BlobStoreComponent.remove
      org.sonatype.nexus.common.stateguard.InvalidStateException: Invalid state: FAILED; allowed: [STARTED]
      at org.sonatype.nexus.common.stateguard.StateGuard._ensure(StateGuard.java:115)
      at org.sonatype.nexus.common.stateguard.StateGuard.access$1(StateGuard.java:108)
      at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:187)
      at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.forceDelete(BlobStoreManagerImpl.java:296)
      at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
      at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
      at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.delete(BlobStoreManagerImpl.java:281)

      Workaround

      Blobstore removal can only happen if the blobstore lifecycle state is STARTED. It is preferred an admin understand why a blobstore could not be started and correct that, however if the goal is to just delete the blobstore and not have concern over why a blobstore is in a non-STARTED state, one may use an Admin - Execute script task with this source to force delete a blobstore as long as other referential integrity rules are not violated.

      // customize with your own blobstore name
      def blobstoreName = 'MyBlobStore'
      def manager = container.lookup(org.sonatype.nexus.blobstore.api.BlobStoreManager.class.name)
      manager.get(blobstoreName).stateGuard.transition('STARTED').run({})
      manager.forceDelete(blobstoreName)
      

      Expected

      Provide a supported method to remove a blobstore that is not in STARTED state, and where removing it would not create its own integrity issues.
      Always log at default levels the name of the blobstore that is being attempted to be deleted and that failed removal ( currently logged at DEBUG )

      Example Reproduce Case

      A permission issue causes the S3 blobstore to be in a stopped state.  At this point, the blob store cannot be removed because it ends up in STOPPED state.  

      Expected:  A broken blob store should be removable, provided it is not used by any repositories.

      Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxx; S3 Extended Request ID: xxxxxxxxxx)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
       at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
       at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
       at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247)
       at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194)
       at com.amazonaws.services.s3.AmazonS3Client.deleteBucket(AmazonS3Client.java:1586)
       at com.amazonaws.services.s3.AmazonS3Client.deleteBucket(AmazonS3Client.java:1571)
       at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore.remove(S3BlobStore.java:578)
       ... 113 common frames omitted
      2019-03-19 13:38:22,585+0100 ERROR [qtp1557116923-415]  admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Blobstore.remove, java-method: org.sonatype.nexus.coreui.BlobStoreComponent.remove
      org.sonatype.nexus.common.stateguard.InvalidStateException: Invalid state: STOPPED; allowed: [STARTED]
       at org.sonatype.nexus.common.stateguard.StateGuard._ensure(StateGuard.java:115)
       at org.sonatype.nexus.common.stateguard.StateGuard.access$1(StateGuard.java:108)
       at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:187)
       at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
       at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.forceDelete(BlobStoreManagerImpl.java:272)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.repository.internal.blobstore.BlobStoreManagerImpl.delete(BlobStoreManagerImpl.java:257)
       at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
       at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
       at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
       at org.sonatype.nexus.blobstore.api.BlobStoreManager$delete$2.call(Unknown Source)
       at org.sonatype.nexus.coreui.BlobStoreComponent.remove(BlobStoreComponent.groovy:175)
       at com.palominolabs.metrics.guice.ExceptionMeteredInterceptor.invoke(ExceptionMeteredInterceptor.java:49)
       at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47)
       at org.sonatype.nexus.validation.internal.ValidationInterceptor.invoke(ValidationInterceptor.java:53)
       at org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)
       at org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
       at org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mjohnson Matt Johnson
              Reporter:
              msurani Mahendra Surani
              Last Updated By:
              Petr Novak Petr Novak
              Team:
              NXRM - Neo
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title