Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-32529

Nexus upgrade can fail with NullPointerException at OrientBlobReconciliationLogMigrator stage.

    Details

    • Story Points:
      2
    • Sprint:
      NXRM Immortals Sprint 39
    • Notability:
      3
    • InvestmentLayer:
      support-escalated
    • Aha Concept:
      non-concept

      Description

      Nexus upgrade can fail with NullPointerException as following:

      2022-05-30 14:06:55,948+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1 - Found reconciliation logs for 1 blob stores to migrate
      2022-05-30 14:06:55,960+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Problem upgrading OrientBlobReconciliationLogMigrator from 1.0 to 1.1
      java.lang.NullPointerException: null
      	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.lambda$7(OrientBlobReconciliationLogMigrator_1_1.java:142)
      	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
      	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
      

      Steps to Reproduce:

      Step-1). Start Nexus 3.32.1  (or try using some other version of nexus)

      DEMO_DIR="/Cases/Upgrade_Issue"
      ${DEMO_DIR}/nexus-3.32.1-01/bin/nexus start
      tail -f ${DEMO_DIR}/sonatype-work/nexus3/log/*
      

      Step-2). Creating raw-hosted repo and upload some assets to raw-hosted

      curl -v -X 'POST' \
        'http://localhost:8081/service/rest/v1/repositories/raw/hosted' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d '{
        "name": "raw-hosted",
        "online": true,
        "storage": {
          "blobStoreName": "default",
          "strictContentTypeValidation": true,
          "writePolicy": "allow_once"
        },
        "cleanup": {
          "policyNames": [
          ]
        },
        "component": {
          "proprietaryComponents": true
        },
        "raw": {
          "contentDisposition": "ATTACHMENT"
        }
      }'
      
      
      curl -u "admin:admin123" -T <(echo 'hello-1') "http://localhost:8081/repository/raw-hosted/test/hello-1.txt"
      curl -u "admin:admin123" -T <(echo 'hello-2') "http://localhost:8081/repository/raw-hosted/test/hello-2.txt"
      curl -u "admin:admin123" -T <(echo 'hello-3') "http://localhost:8081/repository/raw-hosted/test/hello-3.txt"
      curl -u "admin:admin123" -T <(echo 'hello-4') "http://localhost:8081/repository/raw-hosted/test/hello-4.txt"
      curl -u "admin:admin123" -T <(echo 'hello-5') "http://localhost:8081/repository/raw-hosted/test/hello-5.txt"
      

      Step-3). Verify if the directories are created inside

      ls -lart ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default
      

      Step-4). Create couple of blobstores so that we can promote "default"

      curl -v -X 'POST' \
        'http://localhost:8081/service/rest/v1/blobstores/file' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d '{
        "softQuota": {
          "type": "spaceRemainingQuota",
          "limit": "104857600"
        },
        "path": "test-blobstore-A",
        "name": "test-blobstore-A"
      }'
      
      
      curl -v -X 'POST' \
        'http://localhost:8081/service/rest/v1/blobstores/file' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d '{
        "softQuota": {
          "type": "spaceRemainingQuota",
          "limit": "104857600"
        },
        "path": "test-blobstore-B",
        "name": "test-blobstore-B"
      }'
      
      curl -v -X 'POST' \
        'http://localhost:8081/service/rest/v1/blobstores/file' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d '{
        "softQuota": {
          "type": "spaceRemainingQuota",
          "limit": "104857600"
        },
        "path": "test-blobstore-C",
        "name": "test-blobstore-C"
      }'
      

      Step-5). Promoting "default" blobstore to group blobstore and add members to it.

      curl -v -X 'POST' \
        'http://localhost:8081/service/rest/v1/blobstores/group/promote/default' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d ''
      
      
      curl -v -X 'PUT' \
        'http://localhost:8081/service/rest/v1/blobstores/group/default' \
        -u admin:admin123 \
        -H 'accept: application/json' \
        -H 'Content-Type: application/json' \
        -d '{
        "softQuota": {
          "type": "spaceRemainingQuota",
          "limit": "104857600"
        },
        "members": [
          "test-blobstore-A",
          "test-blobstore-B",
          "default-promoted"
        ],
        "fillPolicy": "writeToFirst"
      }'
      

      Step-6). Perform Upgrade to nexus nexus-3.39.0-01

      ${DEMO_DIR}/nexus-3.32.1-01/bin/nexus stop
      cd "${DEMO_DIR}"
      
      curl -LO https://download.sonatype.com/nexus/3/nexus-3.39.0-01-unix.tar.gz
      mv ${DEMO_DIR}/sonatype-work  ${DEMO_DIR}/sonatype-work_ORIGINAL
      
      tar -xzvf nexus-3.39.0-01-unix.tar.gz
      rm -rf ${DEMO_DIR}/sonatype-work
      mv ${DEMO_DIR}/sonatype-work_ORIGINAL  ${DEMO_DIR}/sonatype-work
      
      ${DEMO_DIR}/nexus-3.39.0-01/bin/nexus start
      tail -f ${DEMO_DIR}/sonatype-work/nexus3/log/*
      

      Step-7). Notice the error in nexus.log

      2022-05-30 14:06:55,275+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
      2022-05-30 14:06:55,419+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
      - - - - - - - - - - - - - - - - - - - - - - - - -
      Begin upgrade
      - - - - - - - - - - - - - - - - - - - - - - - - -
      2022-05-30 14:06:55,419+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint component
      2022-05-30 14:06:55,642+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint config
      2022-05-30 14:06:55,776+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint security
      2022-05-30 14:06:55,863+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
      - - - - - - - - - - - - - - - - - - - - - - - - -
      Apply upgrade
      - - - - - - - - - - - - - - - - - - - - - - - - -
      2022-05-30 14:06:55,864+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade docker from 1.0 to 1.1
      2022-05-30 14:06:55,875+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade component from 1.15 to 1.16
      2022-05-30 14:06:55,902+1000 INFO  [FelixStartLevel]  *SYSTEM com.orientechnologies.orient.core.index.OIndexRebuildOutputListener - $ANSI{green {db=component}} - Rebuilding index component.asset_bucket_rid_idx (estimated 5 items)...
      2022-05-30 14:06:55,907+1000 INFO  [FelixStartLevel]  *SYSTEM com.orientechnologies.orient.core.index.OIndexRebuildOutputListener - $ANSI{green {db=component}} --> OK, indexed 5 items in 6 ms
      2022-05-30 14:06:55,947+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade OrientBlobReconciliationLogMigrator from 1.0 to 1.1
      2022-05-30 14:06:55,948+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1 - Found reconciliation logs for 1 blob stores to migrate
      2022-05-30 14:06:55,960+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Problem upgrading OrientBlobReconciliationLogMigrator from 1.0 to 1.1
      java.lang.NullPointerException: null
      	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.lambda$7(OrientBlobReconciliationLogMigrator_1_1.java:142)
      	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
      	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
      	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
      	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
      	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
      	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
      	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
      	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.getBlobStorePath(OrientBlobReconciliationLogMigrator_1_1.java:143)
      	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.copyFiles(OrientBlobReconciliationLogMigrator_1_1.java:114)
      	at java.util.Optional.ifPresent(Optional.java:159)
      	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.apply(OrientBlobReconciliationLogMigrator_1_1.java:96)
      	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
      	at java.util.ArrayList.forEach(ArrayList.java:1259)
      	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
      	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
      	at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
      	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$304863509.GUICE$TRAMPOLINE(<generated>)
      	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
      	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
      	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
      	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
      	at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
      	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$304863509.start(<generated>)
      	at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
      	at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
      	at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
      	at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
      	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
      	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
      	at java.lang.Thread.run(Thread.java:748)
      2022-05-30 14:06:55,962+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
      - - - - - - - - - - - - - - - - - - - - - - - - -
      Rollback upgrade
      - - - - - - - - - - - - - - - - - - - - - - - - -
      

      Workaround:

      1. Move the dir "sonatype-work/nexus3/log/blobstore/default" to the original blobstore name (eg default-original).

      # mv ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default   ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default-original
      

      2. Perform the upgrade again (start nexus 3.39.1)

        Attachments

          Activity

            People

            Assignee:
            mpiggott Matthew Piggott
            Reporter:
            jsensharma Jay Kumar SenSharma
            CC:
            Chris Warren
            Last Updated By:
            Rich Seddon Rich Seddon
            Team:
            NXRM - IMMORTALS
            Owner:
            Matthew Piggott Matthew Piggott
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title