-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.16.1
-
Fix Version/s: 3.21.0
-
Component/s: HA, Support Tools
-
Labels:
-
Notability:n/a
In an HA-C environment, a customer was experiencing database sync issues for the security database. When asked to generate support zips from all three nodes, one node produced a normal support zip, while the other two nodes in the cluster produced zero byte/empty support zips, despite having all support zip options checked.
Upon investigation, it was learned that the security database export failed on the problem nodes, during support zip generation, and this subsequently resulted in a completely empty support zip file.
019-05-23 09:10:03,272+0100 INFO [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.wonderland.DownloadServiceImpl - Downloads directory: /app/example/sonatype-work/nexus3/downloads
2019-05-23 09:10:03,278+0100 INFO [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Maximum included file size: 30 megabytes
2019-05-23 09:10:03,278+0100 INFO [hz.nexus.generic-operation.thread-0] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Maximum ZIP file size: 20 megabytes
2019-05-23 09:10:03,280+0100 INFO [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Generating support ZIP: Request{systemInformation=true, threadDump=true, metrics=true, configuration=true, security=true, log=true, tasklog=true, auditlog=true, jmx=true, limitFileSizes=true, limitZipSize=true}
019-05-23 09:10:15,165+0100 INFO [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SystemInformationGeneratorImpl - Generating system information report
2019-05-23 09:10:15,450+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM com.orientechnologies.orient.core.db.tool.ODatabaseExport - Error on exporting database 'security' to: streaming.json
com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select from index:dictionary
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.executeOnServers(ODistributedStorage.java:487)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:292)
at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:102)
at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:756)
at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportManualIndexes(ODatabaseExport.java:430)
at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:125)
at org.sonatype.nexus.orient.DatabaseExternalizerImpl.export(DatabaseExternalizerImpl.java:153)
at org.sonatype.nexus.internal.orient.PasswordSanitizedJsonSource.generate(PasswordSanitizedJsonSource.java:68)
at org.sonatype.nexus.supportzip.GeneratedContentSourceSupport.prepare(GeneratedContentSourceSupport.java:51)
at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare.call(Unknown Source)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl$_generate_closure3.doCall(SupportZipGeneratorImpl.groovy:188)
at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at groovy.lang.Closure.call(Closure.java:418)
at groovy.lang.Closure.call(Closure.java:434)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2125)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2110)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2151)
at org.codehaus.groovy.runtime.dgm$163.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:186)
at org.sonatype.nexus.supportzip.SupportZipGenerator$generate.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:153)
at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:69)
at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:285)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
2019-05-23 09:10:15,454+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM com.orientechnologies.orient.core.db.tool.ODatabaseExport - Error during closing of write stream
com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (PipedInputStream); line: 1, column: 46392])
at [Source: (PipedInputStream); line: 1, column: 46747]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:588)
at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:483)
at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:495)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2928)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:685)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1884)
at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1878)
at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream.lambda$0(SanitizingJsonOutputStream.java:61)
at java.lang.Thread.run(Thread.java:748)
2019-05-23 09:10:15,457+0100 ERROR [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Failed to create support ZIP
com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on exporting database 'security' to: streaming.json
at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:132)
at org.sonatype.nexus.orient.DatabaseExternalizerImpl.export(DatabaseExternalizerImpl.java:153)
at org.sonatype.nexus.internal.orient.PasswordSanitizedJsonSource.generate(PasswordSanitizedJsonSource.java:68)
at org.sonatype.nexus.supportzip.GeneratedContentSourceSupport.prepare(GeneratedContentSourceSupport.java:51)
at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.sonatype.nexus.supportzip.SupportBundle$ContentSource$prepare.call(Unknown Source)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl$_generate_closure3.doCall(SupportZipGeneratorImpl.groovy:188)
at sun.reflect.GeneratedMethodAccessor1325.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at groovy.lang.Closure.call(Closure.java:418)
at groovy.lang.Closure.call(Closure.java:434)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2125)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2110)
at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2151)
at org.codehaus.groovy.runtime.dgm$163.invoke(Unknown Source)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:186)
at org.sonatype.nexus.supportzip.SupportZipGenerator$generate.callCurrent(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:157)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl.generate(SupportZipGeneratorImpl.groovy:153)
at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:69)
at com.sonatype.nexus.hazelcast.internal.executor.SupportZipHazelcastJob.call(SupportZipHazelcastJob.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.hazelcast.executor.impl.DistributedExecutorService$CallableProcessor.run(DistributedExecutorService.java:285)
at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
Suppressed: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Array (start marker at [Source: (PipedInputStream); line: 1, column: 46392])
at [Source: (PipedInputStream); line: 1, column: 46747]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:588)
at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:483)
at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:495)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2928)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:685)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1884)
at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:1878)
at com.fasterxml.jackson.core.util.JsonGeneratorDelegate.copyCurrentStructure(JsonGeneratorDelegate.java:431)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream$SanitizingJsonGenerator.copyCurrentStructure(SanitizingJsonOutputStream.java:122)
at org.sonatype.nexus.common.io.SanitizingJsonOutputStream.lambda$0(SanitizingJsonOutputStream.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select from index:dictionary
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.executeOnServers(ODistributedStorage.java:487)
at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:292)
at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:102)
at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.query(ODatabaseDocumentTx.java:756)
at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportManualIndexes(ODatabaseExport.java:430)
at com.orientechnologies.orient.core.db.tool.ODatabaseExport.exportDatabase(ODatabaseExport.java:125)
... 41 common frames omitted
2019-05-23 09:10:15,466+0100 INFO [hz.nexus.cached.thread-8] Node2 *SYSTEM com.sonatype.nexus.hazelcast.internal.BlobStoreDownloadServiceImpl - Move: Node2-support-zip-20190523-091003-1.zip
2019-05-23 09:10:15,586+0100 INFO [hz.nexus.cached.thread-8] Node2 *SYSTEM org.sonatype.nexus.internal.atlas.SupportZipGeneratorImpl - Created support ZIP file: $blobstore$/default/Node2-support-zip-20190523-091003-1.zip
Expected
Under no circumstances should an error generating one part of a support zip impact the generation and inclusion of other parts of the support zip. The support zip generation should have successfully completed other part inclusions.
