Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16561

ODatabaseDocumentTx.backup may fail due to IllegalArgumentException Directory not found

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.7.0, 3.8.0, 3.7.1, 3.9.0
    • Fix Version/s: 3.11.0
    • Component/s: Backup, Database, HA
    • Labels:
    • Release Note:
      Yes

      Description

      Symptom 1: The Export configuration and metadata backup task may fail for some database backups, with this type of error:

      2018-03-16 06:23:05,700-0500 WARN  [quartz-5-thread-16]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4b33365c-25f9-4c54-a499-592fbe68f392 : 'Export configuration & metadata for backup' [db.backup] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: 2 failures
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:103)
      	at org.sonatype.nexus.internal.backup.DatabaseBackupTask.execute(DatabaseBackupTask.java:122)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: java.lang.RuntimeException: database backup of component failed
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1)
      		at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41)
      		at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      		at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      		... 4 common frames omitted
      	Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/component/component.50.wal not found
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      		at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      		at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      		at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      		at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67)
      		... 8 common frames omitted
      	Suppressed: java.lang.RuntimeException: database backup of audit failed
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1)
      		at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41)
      		at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      		at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      		... 4 common frames omitted
      	Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/audit/audit.5.wal not found
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      		at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      		at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      		at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      		at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67)
      		... 8 common frames omitted
      

      Sympton 2: Syncing HA-C node databases may also fail

      [1] 2018-03-19 09:35:32,597+0000 WARN  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Moving existent database 'component' in '/var/lib/nexus/sonatype-work/nexus3/db/component' to '/var/lib/nexus/sonatype-work/nexus3/backup/databases/component' and get a fresh copy from a remote node...
      [1] 2018-03-19 09:35:32,600+0000 INFO  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Copying remote database 'component' to: ../sonatype-work/nexus3/tmp/orientdb/install_component_server0.zip
      [1] 2018-03-19 09:35:32,602+0000 INFO  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Installing database 'component' to: /var/lib/nexus/sonatype-work/nexus3/db/component...
      [1] 2018-03-19 09:35:32,607+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] - writing chunk #1 offset=0 size=8.00MB
      [1] 2018-03-19 09:35:32,640+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Downloaded configuration for database 'component' from the cluster
      [1] 2018-03-19 09:35:32,642+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Setting new distributed configuration for database: component (version=3)
      
      [1] 2018-03-19 09:35:33,463+0000 ERROR [FelixStartLevel] node2 *SYSTEM ROOT - Exception `4514D70E` in storage `plocal:/var/lib/nexus/sonatype-work/nexus3/db/component`: 2.2.31 (build 285537d2767275f460df32c6a3be01bfff6a517c, branch 2.2.x)
      [1] java.lang.IllegalArgumentException: Directory /var/lib/nexus/sonatype-work/nexus3/db/component/dirty.fl not found
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      [1] 	at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      [1] 	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.backup(ODistributedStorage.java:1880)
      [1] 	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      [1] 	at org.sonatype.nexus.orient.DatabaseExternalizerImpl.backup(DatabaseExternalizerImpl.java:96)
      [1] 	at org.sonatype.nexus.orient.DatabaseCheckpointSupport.begin(DatabaseCheckpointSupport.java:72)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.lambda$2(UpgradeServiceImpl.java:179)
      [1] 	at java.util.ArrayList.forEach(ArrayList.java:1249)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:134)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:93)
      [1] 	at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.CGLIB$start$4(<generated>)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174$$FastClassByGuice$$2ae549d6.invoke(<generated>)
      [1] 	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
      [1] 	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
      [1] 	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      [1] 	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
      [1] 	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      [1] 	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      [1] 	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.start(<generated>)
      [1] 	at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155)
      [1] 	at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95)
      [1] 	at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:189)
      [1] 	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
      [1] 	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
      [1] 	at java.lang.Thread.run(Thread.java:745)
      

      Cause

      This is due to an upstream OrientDB bug: https://www.prjhub.com/#/issues/9732

      The result of this bug is that some databases may not be backed up correctly or HA nodes may not sync correctly.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mbucher Michael Bucher
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Peter Lynch
              Team:
              Nexus - Platform
              Votes:
              10 Vote for this issue
              Watchers:
              21 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title