Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
3.7.0, 3.7.1, 3.8.0, 3.9.0
-
Yes
Description
Symptom 1: The Export configuration and metadata backup task may fail for some database backups, with this type of error:
2018-03-16 06:23:05,700-0500 WARN [quartz-5-thread-16] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4b33365c-25f9-4c54-a499-592fbe68f392 : 'Export configuration & metadata for backup' [db.backup] execution failure org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: 2 failures at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95) at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:103) at org.sonatype.nexus.internal.backup.DatabaseBackupTask.execute(DatabaseBackupTask.java:122) at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145) at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40) at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120) at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: java.lang.RuntimeException: database backup of component failed at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1) at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ... 4 common frames omitted Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/component/component.50.wal not found at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145) at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127) at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69) at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63) at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.run(Operations.java:155) at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67) ... 8 common frames omitted Suppressed: java.lang.RuntimeException: database backup of audit failed at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1) at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ... 4 common frames omitted Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/audit/audit.5.wal not found at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145) at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127) at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57) at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229) at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69) at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63) at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53) at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64) at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56) at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200) at org.sonatype.nexus.transaction.Operations.run(Operations.java:155) at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63) at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67) ... 8 common frames omitted
Sympton 2: Syncing HA-C node databases may also fail
[1] 2018-03-19 09:35:32,597+0000 WARN [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Moving existent database 'component' in '/var/lib/nexus/sonatype-work/nexus3/db/component' to '/var/lib/nexus/sonatype-work/nexus3/backup/databases/component' and get a fresh copy from a remote node... [1] 2018-03-19 09:35:32,600+0000 INFO [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Copying remote database 'component' to: ../sonatype-work/nexus3/tmp/orientdb/install_component_server0.zip [1] 2018-03-19 09:35:32,602+0000 INFO [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Installing database 'component' to: /var/lib/nexus/sonatype-work/nexus3/db/component... [1] 2018-03-19 09:35:32,607+0000 INFO [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] - writing chunk #1 offset=0 size=8.00MB [1] 2018-03-19 09:35:32,640+0000 INFO [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Downloaded configuration for database 'component' from the cluster [1] 2018-03-19 09:35:32,642+0000 INFO [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Setting new distributed configuration for database: component (version=3) [1] 2018-03-19 09:35:33,463+0000 ERROR [FelixStartLevel] node2 *SYSTEM ROOT - Exception `4514D70E` in storage `plocal:/var/lib/nexus/sonatype-work/nexus3/db/component`: 2.2.31 (build 285537d2767275f460df32c6a3be01bfff6a517c, branch 2.2.x) [1] java.lang.IllegalArgumentException: Directory /var/lib/nexus/sonatype-work/nexus3/db/component/dirty.fl not found [1] at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145) [1] at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127) [1] at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57) [1] at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229) [1] at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.backup(ODistributedStorage.java:1880) [1] at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060) [1] at org.sonatype.nexus.orient.DatabaseExternalizerImpl.backup(DatabaseExternalizerImpl.java:96) [1] at org.sonatype.nexus.orient.DatabaseCheckpointSupport.begin(DatabaseCheckpointSupport.java:72) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.lambda$2(UpgradeServiceImpl.java:179) [1] at java.util.ArrayList.forEach(ArrayList.java:1249) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:134) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:93) [1] at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.CGLIB$start$4(<generated>) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174$$FastClassByGuice$$2ae549d6.invoke(<generated>) [1] at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [1] at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76) [1] at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) [1] at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191) [1] at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56) [1] at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77) [1] at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [1] at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.start(<generated>) [1] at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155) [1] at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95) [1] at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:189) [1] at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429) [1] at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [1] at java.lang.Thread.run(Thread.java:745)
Cause
This is due to an upstream OrientDB bug: https://www.prjhub.com/#/issues/9732
The result of this bug is that some databases may not be backed up correctly or HA nodes may not sync correctly.
Attachments
Issue Links
- fixed by
-
NEXUS-16450 Upgrade to OrientDB 2.2.34 to pick up HA sync and backup fixes
-
- Done
-
- is duplicated by
-
NEXUS-16547 Nexus backup task and database journal file
-
- Closed
-