Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-24088

an S3 blobstore in a blobstore group which references a S3 bucket that is no longer accessible cannot be removed from NXRM and breaks the Blobstore UI

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.22.0, 3.23.0, 3.37.1
    • Fix Version/s: None
    • Component/s: S3
    • Labels:
    • Notability:
      2

      Description

      Problem

      When a single S3 bucket becomes inaccessible which is referenced by NXRM via an S3 blobstore, the Blobstore UI can fail to render any blobstores at all. The List of blobstores in the UI is blank.

      Further, if the S3 blobstore is a member of a group blobstore, there is no way to remove that blobstore reference from NXRM, using groovy scripting, REST API, scheduled task or Blobstore UI.

      Update in 3.37.1, while using Postgres: if there is an S3 blobstore that is broken, by itself will not prevent loading the blobstores UI, however if that S3 blobstore is inside a group blobstore, the Blobstore UI will not load.

      Setup a blobstore group member

      1. Create an S3 blobstore named 's3-blobstore'
      2. Promote the default file blobstore to a group and add the s3 blobstore as a member. The member list should read
        default-promoted
        s3-blobstore
      3. Alter the bucket policy in AWS S3 console of the S3 bucket referenced by the s3-blobstore to deny access to certain critical actions which are normally required. Example policy that could be used. Replace <s3-bucket-name> with actual bucket name and <user-role-arn> with actual role or user arn used to access the bucket
        {
            "Version": "2012-10-17",
            "Id": "NexusS3BlobStorePolicy",
            "Statement": [
                {
                    "Sid": "NexusS3BlobStoreAccess",
                    "Effect": "Deny",
                    "Principal": {
                        "AWS": "<user-role-arn>"
                    },
                    "Action": [
                        "s3:PutObject",
                        "s3:GetObject",
                        "s3:DeleteObject",
                        "s3:ListBucket",
                        "s3:GetLifecycleConfiguration",
                        "s3:PutLifecycleConfiguration",
                        "s3:PutObjectTagging",
                        "s3:GetObjectTagging",
                        "s3:DeleteObjectTagging",
                    ],
                    "Resource": [
                        "arn:aws:s3:::<s3-bucket-name>",
                        "arn:aws:s3:::<s3-bucket-name>/*"
                    ]
                }
            ]
        } 
      4. Delete the metadata.properties and -metrics.properties file in the root of the S3 bucket using the S3 console - this will trigger code eventually where NXRM will try to recreate these files. The intent of this step is to mimic the entire bucket having been deleted.

      Try to delete the blobstore (FAIL)

      1. Run a "admin - Remove a member from a blobstore group" scheduled task trying to remove the 's3-blobstore' from its group. The task will fail trying to write files into the blobstore due to permissions.
      2. Try to remove the s3-blobstore using the Blobstore UI - you cannot because the UI states the blobstore is in use by the group and cannot be removed.
      3. Try to remove the s3-blobstore from the 'default' group blobstore. You cannot, there will be an error telling you to use the task instead.
      4. Reboot NXRM. On startup NXRM tries to init the s3-blobstore, moving it from state NEW to STARTED. This init fails due to permissions, leaving the state of the blobstore in NEW state.
      5. After possibly several restarts, or clearing the browser cache, a new web browser session to the NXRM UI will try to load the Blobstore UI list. The list will be empty showing no blobstores at all to edit all because the metrics files stored in the s3-blobstore were attempted to be read as part of loading the metrics for the 'default' group blobstore and this fails due to permissions of one of its members:
        2020-05-27 11:23:56,901-0300 ERROR [qtp1577682986-55] admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Blobstore.read, java-method: org.sonatype.nexus.coreui.BlobStoreComponent.read
        org.sonatype.nexus.common.stateguard.InvalidStateException: Invalid state: FAILED; allowed: [STARTED]
        	at org.sonatype.nexus.common.stateguard.StateGuard._ensure(StateGuard.java:115)
        	at org.sonatype.nexus.common.stateguard.StateGuard.access$1(StateGuard.java:108)
        	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:271)
        	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
        	at org.sonatype.nexus.blobstore.group.BlobStoreGroup.lambda$12(BlobStoreGroup.java:269)
        	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        	at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1359)
        	at java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)
        	at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)
        	at java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)
        	at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)
        	at java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)
        	at org.sonatype.nexus.blobstore.group.internal.BlobStoreGroupMetrics.<init>(BlobStoreGroupMetrics.java:50)
        	at org.sonatype.nexus.blobstore.group.BlobStoreGroup.getMetrics(BlobStoreGroup.java:271)
        	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        	at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        	at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
        	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
        	at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:63)
        	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:299)
        	at org.sonatype.nexus.coreui.BlobStoreComponent.asBlobStoreXO(BlobStoreComponent.groovy:237)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
        	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:352)
        	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
        	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:68)
        	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:177)
        	at org.sonatype.nexus.coreui.BlobStoreComponent$_read_closure3.doCall(BlobStoreComponent.groovy:104)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
        	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
        	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
        
      6. Enable groovy scripting by adding nexus.scripts.allowCreation=true to the nexus.properties and restarting. Try to run a script to force delete the blobstore with this source:
        // customize with your own blobstore name
        def blobstoreName = 's3-blobstore'
        def manager = container.lookup(org.sonatype.nexus.blobstore.api.BlobStoreManager.class.name)
        manager.get(blobstoreName).stateGuard.transition('STARTED').run({})
        manager.forceDelete(blobstoreName)
        

        This task also fails due to permission problems, despite trying to forcefully move the state of the blobstore from NEW to STARTED before delete. There is nothing logged to indicate the transition failed.

      Expected

      There will be cases where the S3 bucket admin knows the bucket has been deleted, or is still present but contains nothing of value they wish to keep any longer. There should be a way to forcefully confirm to NXRM that it should should delete all references to the blobstore it no longer has permission to access.

      External changes to a single S3 bucket which make it no longer a valid reference should not put the entire blobstores UI into an unusuable state.

      Provide a supported way for an NXRM admin to remove a blobstore that points at a bucket that is no longer valid.

      Workaround

      Recreate the S3 bucket in AWS console and restore permissions to the S3 key that NXRM has referenced in its config. Once NXRM has the correct permissions again to think the bucket is valid, then the scheduled task should be able to be used to remove the blobstore reference

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title