[NEXUS-32529] Nexus upgrade can fail with NullPointerException at OrientBlobReconciliationLogMigrator stage. - Sonatype JIRA

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 3.39.0, 3.38.1
Fix Version/s: 3.42.0
Component/s: Blobstore, Upgrade
Labels:
- regression
- supportant

Story Points:
2
Sprint:
NXRM Immortals Sprint 39
Notability:
3
InvestmentLayer:
support-escalated
Aha Concept:
non-concept

Description

Nexus upgrade can fail with NullPointerException as following:

2022-05-30 14:06:55,948+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1 - Found reconciliation logs for 1 blob stores to migrate
2022-05-30 14:06:55,960+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Problem upgrading OrientBlobReconciliationLogMigrator from 1.0 to 1.1
java.lang.NullPointerException: null
	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.lambda$7(OrientBlobReconciliationLogMigrator_1_1.java:142)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)

Steps to Reproduce:

Step-1). Start Nexus 3.32.1 (or try using some other version of nexus)

DEMO_DIR="/Cases/Upgrade_Issue"
${DEMO_DIR}/nexus-3.32.1-01/bin/nexus start
tail -f ${DEMO_DIR}/sonatype-work/nexus3/log/*

Step-2). Creating raw-hosted repo and upload some assets to raw-hosted

curl -v -X 'POST' \
  'http://localhost:8081/service/rest/v1/repositories/raw/hosted' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "raw-hosted",
  "online": true,
  "storage": {
    "blobStoreName": "default",
    "strictContentTypeValidation": true,
    "writePolicy": "allow_once"
  },
  "cleanup": {
    "policyNames": [
    ]
  },
  "component": {
    "proprietaryComponents": true
  },
  "raw": {
    "contentDisposition": "ATTACHMENT"
  }
}'


curl -u "admin:admin123" -T <(echo 'hello-1') "http://localhost:8081/repository/raw-hosted/test/hello-1.txt"
curl -u "admin:admin123" -T <(echo 'hello-2') "http://localhost:8081/repository/raw-hosted/test/hello-2.txt"
curl -u "admin:admin123" -T <(echo 'hello-3') "http://localhost:8081/repository/raw-hosted/test/hello-3.txt"
curl -u "admin:admin123" -T <(echo 'hello-4') "http://localhost:8081/repository/raw-hosted/test/hello-4.txt"
curl -u "admin:admin123" -T <(echo 'hello-5') "http://localhost:8081/repository/raw-hosted/test/hello-5.txt"

Step-3). Verify if the directories are created inside

ls -lart ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default

Step-4). Create couple of blobstores so that we can promote "default"

curl -v -X 'POST' \
  'http://localhost:8081/service/rest/v1/blobstores/file' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "softQuota": {
    "type": "spaceRemainingQuota",
    "limit": "104857600"
  },
  "path": "test-blobstore-A",
  "name": "test-blobstore-A"
}'


curl -v -X 'POST' \
  'http://localhost:8081/service/rest/v1/blobstores/file' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "softQuota": {
    "type": "spaceRemainingQuota",
    "limit": "104857600"
  },
  "path": "test-blobstore-B",
  "name": "test-blobstore-B"
}'

curl -v -X 'POST' \
  'http://localhost:8081/service/rest/v1/blobstores/file' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "softQuota": {
    "type": "spaceRemainingQuota",
    "limit": "104857600"
  },
  "path": "test-blobstore-C",
  "name": "test-blobstore-C"
}'

Step-5). Promoting "default" blobstore to group blobstore and add members to it.

curl -v -X 'POST' \
  'http://localhost:8081/service/rest/v1/blobstores/group/promote/default' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d ''


curl -v -X 'PUT' \
  'http://localhost:8081/service/rest/v1/blobstores/group/default' \
  -u admin:admin123 \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "softQuota": {
    "type": "spaceRemainingQuota",
    "limit": "104857600"
  },
  "members": [
    "test-blobstore-A",
    "test-blobstore-B",
    "default-promoted"
  ],
  "fillPolicy": "writeToFirst"
}'

Step-6). Perform Upgrade to nexus nexus-3.39.0-01

${DEMO_DIR}/nexus-3.32.1-01/bin/nexus stop
cd "${DEMO_DIR}"

curl -LO https://download.sonatype.com/nexus/3/nexus-3.39.0-01-unix.tar.gz
mv ${DEMO_DIR}/sonatype-work  ${DEMO_DIR}/sonatype-work_ORIGINAL

tar -xzvf nexus-3.39.0-01-unix.tar.gz
rm -rf ${DEMO_DIR}/sonatype-work
mv ${DEMO_DIR}/sonatype-work_ORIGINAL  ${DEMO_DIR}/sonatype-work

${DEMO_DIR}/nexus-3.39.0-01/bin/nexus start
tail -f ${DEMO_DIR}/sonatype-work/nexus3/log/*

Step-7). Notice the error in nexus.log

2022-05-30 14:06:55,275+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Start UPGRADE
2022-05-30 14:06:55,419+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
- - - - - - - - - - - - - - - - - - - - - - - - -
Begin upgrade
- - - - - - - - - - - - - - - - - - - - - - - - -
2022-05-30 14:06:55,419+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint component
2022-05-30 14:06:55,642+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint config
2022-05-30 14:06:55,776+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Checkpoint security
2022-05-30 14:06:55,863+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
- - - - - - - - - - - - - - - - - - - - - - - - -
Apply upgrade
- - - - - - - - - - - - - - - - - - - - - - - - -
2022-05-30 14:06:55,864+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade docker from 1.0 to 1.1
2022-05-30 14:06:55,875+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade component from 1.15 to 1.16
2022-05-30 14:06:55,902+1000 INFO  [FelixStartLevel]  *SYSTEM com.orientechnologies.orient.core.index.OIndexRebuildOutputListener - $ANSI{green {db=component}} - Rebuilding index component.asset_bucket_rid_idx (estimated 5 items)...
2022-05-30 14:06:55,907+1000 INFO  [FelixStartLevel]  *SYSTEM com.orientechnologies.orient.core.index.OIndexRebuildOutputListener - $ANSI{green {db=component}} --> OK, indexed 5 items in 6 ms
2022-05-30 14:06:55,947+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade OrientBlobReconciliationLogMigrator from 1.0 to 1.1
2022-05-30 14:06:55,948+1000 INFO  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1 - Found reconciliation logs for 1 blob stores to migrate
2022-05-30 14:06:55,960+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Problem upgrading OrientBlobReconciliationLogMigrator from 1.0 to 1.1
java.lang.NullPointerException: null
	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.lambda$7(OrientBlobReconciliationLogMigrator_1_1.java:142)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.getBlobStorePath(OrientBlobReconciliationLogMigrator_1_1.java:143)
	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.copyFiles(OrientBlobReconciliationLogMigrator_1_1.java:114)
	at java.util.Optional.ifPresent(Optional.java:159)
	at org.sonatype.nexus.repository.upgrade.OrientBlobReconciliationLogMigrator_1_1.apply(OrientBlobReconciliationLogMigrator_1_1.java:96)
	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
	at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$304863509.GUICE$TRAMPOLINE(<generated>)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
	at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
	at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$304863509.start(<generated>)
	at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
	at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
	at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
	at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
	at java.lang.Thread.run(Thread.java:748)
2022-05-30 14:06:55,962+1000 WARN  [FelixStartLevel]  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - 
- - - - - - - - - - - - - - - - - - - - - - - - -
Rollback upgrade
- - - - - - - - - - - - - - - - - - - - - - - - -

Workaround:

1. Move the dir "sonatype-work/nexus3/log/blobstore/default" to the original blobstore name (eg default-original).

# mv ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default   ${DEMO_DIR}/sonatype-work/nexus3/log/blobstore/default-original

2. Perform the upgrade again (start nexus 3.39.1)

Attachments

Activity

People

Assignee:: Matthew Piggott

Reporter:: Jay Kumar SenSharma

CC:: Chris Warren

Last Updated By:: Rich Seddon

Team:: NXRM - IMMORTALS

Owner:: Matthew Piggott

Votes:: 1 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 05/30/22 04:28 AM

Updated:: 09/11/22 03:19 PM

Resolved:: 08/02/22 12:22 AM

Date of First Response:: 20/Jun/22 1:39 AM