Startup hangs for a long time if configured IQ Server cannot be reached.

If the configured IQ Server is not available it blocks the jetty startup thread.  This can cause startup to hang for a long time. The thread interaction is posted below.

Note that these connection attempts are made even if the IQ Server capability is disabled!

Expected: Long running operations Outbound network activity should not be performed on the jetty main thread during startup - it potentially can block the entire server from being ready. Startup needs to be as quick as possible and one cannot predict how slow network I/O will be.

"jetty-main-1" #63 daemon prio=5 os_prio=31 tid=0x00007fcc91a74800 nid=0x6d1b in Object.wait() [0x000070000989c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.osgi.util.tracker.ServiceTracker.waitForService(ServiceTracker.java:502)

• locked <0x000000077a1740a0> (a org.osgi.util.tracker.ServiceTracker$Tracked)
  at org.sonatype.nexus.bootstrap.osgi.BootstrapListener.contextInitialized(BootstrapListener.java:128)
  at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:953)
  at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:558)
  at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:918)
  at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:370)
  at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497)
  at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459)
  at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:848)
  at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:287)
  at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
• locked <0x0000000775add6f8> (a java.lang.Object)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
  at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
  at com.codahale.metrics.jetty9.InstrumentedHandler.doStart(InstrumentedHandler.java:92)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
• locked <0x00000007764cb2c0> (a java.lang.Object)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
  at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
• locked <0x00000007764cb4a0> (a java.lang.Object)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
  at org.eclipse.jetty.server.Server.start(Server.java:419)
  at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
  at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
  at org.eclipse.jetty.server.Server.doStart(Server.java:386)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
• locked <0x00000007759e9798> (a java.lang.Object)
  at org.sonatype.nexus.bootstrap.jetty.JettyServer$JettyMainThread.run(JettyServer.java:274)

"FelixStartLevel" #20 daemon prio=5 os_prio=31 tid=0x00007fcc92936800 nid=0x5f03 runnable [0x0000700008b74000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

• locked <0x00000007b7e418e8> (a java.net.SocksSocketImpl)
  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:589)
  at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
  at org.sonatype.nexus.internal.httpclient.NexusSSLConnectionSocketFactory.connectSocket(NexusSSLConnectionSocketFactory.java:89)
  at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
  at zz.n$a.a(SourceFile:157)
  at zz.n$a.a(SourceFile:102)
  at zz.a.a(SourceFile:28)
  at zz.c.a(SourceFile:55)
  at com.sonatype.insight.rm.rest.RestClientFactory$RepositorySpecificClient.setEnabled(SourceFile:190)
  at com.sonatype.nexus.clm.internal.IQErrorManager.sendRepositoryState(IQErrorManager.java:79)
  at com.sonatype.nexus.clm.internal.IQErrorManager.verifyConnection(IQErrorManager.java:54)
  at com.sonatype.nexus.clm.internal.capability.FirewallAuditCapability.onActivate(FirewallAuditCapability.java:110)
  at com.sonatype.nexus.clm.internal.capability.FirewallAuditCapability.onActivate(FirewallAuditCapability.java:1)
  at org.sonatype.nexus.capability.CapabilitySupport.onActivate(CapabilitySupport.java:183)
  at org.sonatype.nexus.internal.capability.DefaultCapabilityReference$EnabledState.activate(DefaultCapabilityReference.java:587)
  at org.sonatype.nexus.internal.capability.DefaultCapabilityReference.activate(DefaultCapabilityReference.java:217)
  at org.sonatype.nexus.internal.capability.ActivationConditionHandler.handle(ActivationConditionHandler.java:65)
  at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
  at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
  at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
  at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)
  at com.google.common.eventbus.Dispatcher$ImmediateDispatcher.dispatch(Dispatcher.java:186)
  at com.google.common.eventbus.EventBus.post(EventBus.java:212)
  at org.sonatype.nexus.internal.event.EventManagerImpl.post(EventManagerImpl.java:127)
  at org.sonatype.nexus.capability.condition.ConditionSupport.setSatisfied(ConditionSupport.java:126)
  at org.sonatype.nexus.capability.condition.internal.CompositeConditionSupport.handle(CompositeConditionSupport.java:76)
  at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
  at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
  at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
  at com.google.common.eventbus.Subscriber.dispatchEvent(Subscriber.java:67)
  at com.google.common.eventbus.Dispatcher$ImmediateDispatcher.dispatch(Dispatcher.java:186)
  at com.google.common.eventbus.EventBus.post(EventBus.java:212)
  at org.sonatype.nexus.internal.event.EventManagerImpl.post(EventManagerImpl.java:127)
  at org.sonatype.nexus.capability.condition.ConditionSupport.setSatisfied(ConditionSupport.java:126)
  at org.sonatype.nexus.capability.condition.internal.NexusIsActiveCondition.start(NexusIsActiveCondition.java:57)
  at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:168)
  at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:100)
• locked <0x0000000776d6e110> (a org.eclipse.sisu.inject.DefaultBeanLocator)
  at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:195)
  at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1429)
  at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
  at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:

• <0x00000007b6230f70> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)