Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-20035

support zip generation may fail or create empty support zips if there is an error exporting a database

    XMLWordPrintable

    Details

    • Notability:
      n/a

      Description

      In an HA-C environment, a customer was experiencing database sync issues for the security database. When asked to generate support zips from all three nodes, one node produced a normal support zip, while the other two nodes in the cluster produced zero byte/empty support zips, despite having all support zip options checked.

      Upon investigation, it was learned that the security database export failed on the problem nodes, during support zip generation, and this subsequently resulted in a completely empty support zip file.

      019-05-23 09:10:03,272+0100 INFO  [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.wonderland.DownloadServiceImpl - Downloads directory: /app/example/sonatype-work/nexus3/downloads
      2019-05-23 09:10:03,278+0100 INFO  [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Maximum included file size: 30 megabytes
      2019-05-23 09:10:03,278+0100 INFO  [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Maximum ZIP file size: 20 megabytes
      2019-05-23 09:10:03,280+0100 INFO  [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Generating support ZIP: Request{systemInformation=true, threadDump=true, metrics=true, configuration=true, security=true, log=true, tasklog=true, auditlog=true, jmx=true, limitFileSizes=true, limitZipSize=true}
      
      019-05-23 09:10:15,165+0100 INFO  [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SystemInformationGeneratorImpl - Generating system information report
      2019-05-23 09:10:15,450+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM com.orientechnologies.orient.core.db.tool.ODatabaseExport - Error on exporting database 'security' to: streaming.json
      com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select from index:dictionary
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.executeOnServers(ODistributedStorage.java:487)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:292)
      	at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
      	at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
      	at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:102)
      	at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:756)
      	at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportManualIndexes(ODatabaseExport.java:430)
      	at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:125)
      	at org.sonatype.nexus.orient.DatabaseExternalizerImpl.export(DatabaseExternalizerImpl.java:153)
      	at org.sonatype.nexus.internal.orient.PasswordSanitizedJsonSource.generate(PasswordSanitizedJsonSource.java:68)
      	at org.sonatype.nexus.supportzip.GeneratedContentSourceSupport.prepare(GeneratedContentSourceSupport.java:51)
      	at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare$0.call(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      	at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare.call(Unknown Source)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl$_generate_closure3.doCall(SupportZipGeneratorImpl.groovy:188)
      	at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2125)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2110)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2151)
      	at org.codehaus.groovy.runtime.dgm$163.invoke(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:186)
      	at org.sonatype.nexus.supportzip.SupportZipGenerator$generate.callCurrent(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:153)
      	at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:69)
      	at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:1)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:285)
      	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
      	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      2019-05-23 09:10:15,454+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM com.orientechnologies.orient.core.db.tool.ODatabaseExport - Error during closing of write stream
      com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (PipedInputStream); line: 1, column: 46392])
       at [Source: (PipedInputStream); line: 1, column: 46747]
      	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:588)
      	at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:483)
      	at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:495)
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2928)
      	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:685)
      	at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1884)
      	at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
      	at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
      	at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1878)
      	at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
      	at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
      	at org.sonatype.nexus.common.io.SanitizingJsonOutputStream.lambda$0(SanitizingJsonOutputStream.java:61)
      	at java.lang.Thread.run(Thread.java:748)
      
      2019-05-23 09:10:15,457+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Failed to create support ZIP
      com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on exporting database 'security' to: streaming.json
      	at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:132)
      	at org.sonatype.nexus.orient.DatabaseExternalizerImpl.export(DatabaseExternalizerImpl.java:153)
      	at org.sonatype.nexus.internal.orient.PasswordSanitizedJsonSource.generate(PasswordSanitizedJsonSource.java:68)
      	at org.sonatype.nexus.supportzip.GeneratedContentSourceSupport.prepare(GeneratedContentSourceSupport.java:51)
      	at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare$0.call(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      	at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare.call(Unknown Source)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl$_generate_closure3.doCall(SupportZipGeneratorImpl.groovy:188)
      	at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
      	at groovy.lang.Closure.call(Closure.java:418)
      	at groovy.lang.Closure.call(Closure.java:434)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2125)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2110)
      	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2151)
      	at org.codehaus.groovy.runtime.dgm$163.invoke(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
      	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:186)
      	at org.sonatype.nexus.supportzip.SupportZipGenerator$generate.callCurrent(Unknown Source)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
      	at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:153)
      	at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:69)
      	at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:1)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:285)
      	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
      	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      	Suppressed: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (PipedInputStream); line: 1, column: 46392])
       at [Source: (PipedInputStream); line: 1, column: 46747]
      		at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:588)
      		at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:483)
      		at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:495)
      		at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2928)
      		at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:685)
      		at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1884)
      		at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
      		at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
      		at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1878)
      		at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
      		at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
      		at org.sonatype.nexus.common.io.SanitizingJsonOutputStream.lambda$0(SanitizingJsonOutputStream.java:61)
      		at java.lang.Thread.run(Thread.java:748)
      Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select from index:dictionary
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.executeOnServers(ODistributedStorage.java:487)
      	at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:292)
      	at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
      	at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
      	at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:102)
      	at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
      	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:756)
      	at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportManualIndexes(ODatabaseExport.java:430)
      	at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:125)
      	... 41 common frames omitted
      2019-05-23 09:10:15,466+0100 INFO  [hz.nexus.cached.thread-8] Node2 *SYSTEM com.sonatype.nexus.hazelcast.internal.BlobStoreDownloadServiceImpl - Move: Node2-support-zip-20190523-091003-1.zip
      2019-05-23 09:10:15,586+0100 INFO  [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Created support ZIP file: $blobstore$/default/Node2-support-zip-20190523-091003-1.zip
      
      

      Expected

      Under no circumstances should an error generating one part of a support zip impact the generation and inclusion of other parts of the support zip. The support zip generation should have successfully completed other part inclusions.

        Attachments

          Activity

            People

            Assignee:
            mjohnson Matt Johnson
            Reporter:
            plynch Peter Lynch
            Last Updated By:
            Wes Wannemacher
            Team:
            NXRM - Neo
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title