Deadlock in repository health check

Observed in a thread dump...

61 threads trying to get run repository health check but are blocked trying to get the task lock.

They are unable to do so, because this thread is trying to get the global configuration lock while creating a health check task:

esh-1-thread-1 id=64 state=BLOCKED

• waiting to lock <0x35187b88> (a org.sonatype.nexus.configuration.application.DefaultNexusConfiguration)
  owned by qtp211318608-297 id=297
  at org.sonatype.nexus.scheduling.internal.DefaultTaskConfigManager.addTask(DefaultTaskConfigManager.java:156)
  at org.sonatype.scheduling.DefaultScheduler.updateSchedule(DefaultScheduler.java:240)
  at org.sonatype.nexus.scheduling.internal.DefaultNexusScheduler.updateSchedule(DefaultNexusScheduler.java:85)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.updateTaskSchedule(HealthCheckTaskManagerImpl.java:215)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.updateTaskSchedule(HealthCheckTaskManagerImpl.java:206)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.scheduleTask(HealthCheckTaskManagerImpl.java:156)
  at com.sonatype.nexus.plugins.healthcheck.task.HealthCheckEventInspector.inspect(HealthCheckEventInspector.java:77)
  at com.sonatype.nexus.plugins.healthcheck.task.HealthCheckEventInspector.on(HealthCheckEventInspector.java:56)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
  at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
• locked com.google.common.eventbus.SynchronizedEventSubscriber@46365cba
  at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
  at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
  at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:117)
  at org.sonatype.nexus.threads.MDCAwareRunnable.run(MDCAwareRunnable.java:41)
  at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
  at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

That thread can't get the configuration lock because it is held by this thread, which is trying to create a health check task:

qtp211318608-297 id=297 state=BLOCKED

• waiting to lock <0x6025840f> (a java.util.HashMap)
  owned by esh-1-thread-1 id=64
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.getTask(HealthCheckTaskManagerImpl.java:83)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.isCandidateRepository(HealthCheckTaskManagerImpl.java:294)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.maybeScheduleTask(HealthCheckTaskManagerImpl.java:307)
  at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.on(HealthCheckTaskManagerImpl.java:273)
  at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
  at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:49)
  at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
  at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
  at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
  at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
  at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
  at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.insertRepository(DefaultRepositoryRegistry.java:235)
  at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.addRepository(DefaultRepositoryRegistry.java:76)
  at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.instantiateRepository(DefaultNexusConfiguration.java:687)
  at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.instantiateRepository(DefaultNexusConfiguration.java:665)
  at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.createRepository(DefaultNexusConfiguration.java:846)
• locked org.sonatype.nexus.configuration.application.DefaultNexusConfiguration@35187b88
  at org.sonatype.nexus.templates.repository.AbstractRepositoryTemplateProvider.createRepository(AbstractRepositoryTemplateProvider.java:66)
  at org.sonatype.nexus.templates.repository.AbstractRepositoryTemplate.create(AbstractRepositoryTemplate.java:101)
  at org.sonatype.nexus.rest.repositories.RepositoryListPlexusResource.post(RepositoryListPlexusResource.java:145)
  at org.sonatype.plexus.rest.resource.RestletResource.acceptRepresentation(RestletResource.java:262)
  at org.sonatype.nexus.rest.NexusRestletResource.acceptRepresentation(NexusRestletResource.java:70)
  at org.restlet.resource.Resource.post(Resource.java:688)
  at org.restlet.resource.Resource.handlePost(Resource.java:537)

Game over.

Full thread dump attached.