Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16561

ODatabaseDocumentTx.backup may fail due to IllegalArgumentException Directory not found

Details

    • Yes

    Description

      Symptom 1: The Export configuration and metadata backup task may fail for some database backups, with this type of error:

      2018-03-16 06:23:05,700-0500 WARN  [quartz-5-thread-16]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4b33365c-25f9-4c54-a499-592fbe68f392 : 'Export configuration & metadata for backup' [db.backup] execution failure
      org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: 2 failures
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
      	at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:103)
      	at org.sonatype.nexus.internal.backup.DatabaseBackupTask.execute(DatabaseBackupTask.java:122)
      	at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:93)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	Suppressed: java.lang.RuntimeException: database backup of component failed
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1)
      		at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41)
      		at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      		at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      		... 4 common frames omitted
      	Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/component/component.50.wal not found
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      		at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      		at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      		at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      		at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67)
      		... 8 common frames omitted
      	Suppressed: java.lang.RuntimeException: database backup of audit failed
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:77)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:1)
      		at org.sonatype.nexus.thread.internal.MDCAwareCallable.call(MDCAwareCallable.java:41)
      		at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      		at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      		... 4 common frames omitted
      	Caused by: java.lang.IllegalArgumentException: Directory /home/crisbld/build/nexus3/sonatype-work/nexus3/db/audit/audit.5.wal not found
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      		at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      		at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      		at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.lambda$1(DatabaseBackupRunner.java:69)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:63)
      		at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
      		at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
      		at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
      		at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
      		at org.sonatype.nexus.transaction.Operations.run(Operations.java:155)
      		at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:63)
      		at org.sonatype.nexus.internal.backup.DatabaseBackupRunner.call(DatabaseBackupRunner.java:67)
      		... 8 common frames omitted
      

      Sympton 2: Syncing HA-C node databases may also fail

      [1] 2018-03-19 09:35:32,597+0000 WARN  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Moving existent database 'component' in '/var/lib/nexus/sonatype-work/nexus3/db/component' to '/var/lib/nexus/sonatype-work/nexus3/backup/databases/component' and get a fresh copy from a remote node...
      [1] 2018-03-19 09:35:32,600+0000 INFO  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Copying remote database 'component' to: ../sonatype-work/nexus3/tmp/orientdb/install_component_server0.zip
      [1] 2018-03-19 09:35:32,602+0000 INFO  [Timer-1] node2 *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB]<-[33E6A1A2-88515E5A-305590A8-0E0A7484-5D79C149] Installing database 'component' to: /var/lib/nexus/sonatype-work/nexus3/db/component...
      [1] 2018-03-19 09:35:32,607+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] - writing chunk #1 offset=0 size=8.00MB
      [1] 2018-03-19 09:35:32,640+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Downloaded configuration for database 'component' from the cluster
      [1] 2018-03-19 09:35:32,642+0000 INFO  [OrientDB installDatabase node=B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB db=component] cxpxnex01box002.cxp.williamhill.plc *SYSTEM ROOT - [B04B1AAD-CE70147A-9654C698-58577E49-B9D57FFB] Setting new distributed configuration for database: component (version=3)
      
      [1] 2018-03-19 09:35:33,463+0000 ERROR [FelixStartLevel] node2 *SYSTEM ROOT - Exception `4514D70E` in storage `plocal:/var/lib/nexus/sonatype-work/nexus3/db/component`: 2.2.31 (build 285537d2767275f460df32c6a3be01bfff6a517c, branch 2.2.x)
      [1] java.lang.IllegalArgumentException: Directory /var/lib/nexus/sonatype-work/nexus3/db/component/dirty.fl not found
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:145)
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.addFolder(OZIPCompressionUtil.java:127)
      [1] 	at com.orientechnologies.orient.core.compression.impl.OZIPCompressionUtil.compressDirectory(OZIPCompressionUtil.java:57)
      [1] 	at com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage.backup(OLocalPaginatedStorage.java:229)
      [1] 	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.backup(ODistributedStorage.java:1880)
      [1] 	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.backup(ODatabaseDocumentTx.java:3060)
      [1] 	at org.sonatype.nexus.orient.DatabaseExternalizerImpl.backup(DatabaseExternalizerImpl.java:96)
      [1] 	at org.sonatype.nexus.orient.DatabaseCheckpointSupport.begin(DatabaseCheckpointSupport.java:72)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.lambda$2(UpgradeServiceImpl.java:179)
      [1] 	at java.util.ArrayList.forEach(ArrayList.java:1249)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:134)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:93)
      [1] 	at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:67)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.CGLIB$start$4(<generated>)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174$$FastClassByGuice$$2ae549d6.invoke(<generated>)
      [1] 	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
      [1] 	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
      [1] 	at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
      [1] 	at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:191)
      [1] 	at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:56)
      [1] 	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
      [1] 	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
      [1] 	at org.sonatype.nexus.upgrade.internal.UpgradeServiceImpl$$EnhancerByGuice$$dbe05174.start(<generated>)
      [1] 	at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:155)
      [1] 	at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:95)
      [1] 	at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:189)
      [1] 	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
      [1] 	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
      [1] 	at java.lang.Thread.run(Thread.java:745)
      

      Cause

      This is due to an upstream OrientDB bug: https://www.prjhub.com/#/issues/9732

      The result of this bug is that some databases may not be backed up correctly or HA nodes may not sync correctly.

      Attachments

        Issue Links

          Activity

            People

              mbucher Michael Bucher
              plynch Peter Lynch
              Joe Tom Joe Tom
              Nexus - Platform
              Votes:
              10 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                tigCommentSecurity.panel-title