Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19249

Deadlock in repository health check

    XMLWordPrintable

    Details

      Description

      Observed in a thread dump...

      61 threads trying to get run repository health check but are blocked trying to get the task lock.

      They are unable to do so, because this thread is trying to get the global configuration lock while creating a health check task:

      esh-1-thread-1 id=64 state=BLOCKED

      • waiting to lock <0x35187b88> (a org.sonatype.nexus.configuration.application.DefaultNexusConfiguration)
        owned by qtp211318608-297 id=297
        at org.sonatype.nexus.scheduling.internal.DefaultTaskConfigManager.addTask(DefaultTaskConfigManager.java:156)
        at org.sonatype.scheduling.DefaultScheduler.updateSchedule(DefaultScheduler.java:240)
        at org.sonatype.nexus.scheduling.internal.DefaultNexusScheduler.updateSchedule(DefaultNexusScheduler.java:85)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.updateTaskSchedule(HealthCheckTaskManagerImpl.java:215)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.updateTaskSchedule(HealthCheckTaskManagerImpl.java:206)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.scheduleTask(HealthCheckTaskManagerImpl.java:156)
        at com.sonatype.nexus.plugins.healthcheck.task.HealthCheckEventInspector.inspect(HealthCheckEventInspector.java:77)
        at com.sonatype.nexus.plugins.healthcheck.task.HealthCheckEventInspector.on(HealthCheckEventInspector.java:56)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
        at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
      • locked com.google.common.eventbus.SynchronizedEventSubscriber@46365cba
        at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
        at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
        at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:117)
        at org.sonatype.nexus.threads.MDCAwareRunnable.run(MDCAwareRunnable.java:41)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

      That thread can't get the configuration lock because it is held by this thread, which is trying to create a health check task:

      qtp211318608-297 id=297 state=BLOCKED

      • waiting to lock <0x6025840f> (a java.util.HashMap)
        owned by esh-1-thread-1 id=64
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.getTask(HealthCheckTaskManagerImpl.java:83)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.isCandidateRepository(HealthCheckTaskManagerImpl.java:294)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.maybeScheduleTask(HealthCheckTaskManagerImpl.java:307)
        at com.sonatype.nexus.plugins.healthcheck.service.impl.HealthCheckTaskManagerImpl.on(HealthCheckTaskManagerImpl.java:273)
        at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventHandler.handleEvent(EventHandler.java:80)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:49)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.dispatch(EventBus.java:329)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultGuavaEventBus.dispatch(DefaultGuavaEventBus.java:34)
        at org.sonatype.sisu.goodies.eventbus.internal.ReentrantGuavaEventBus.dispatchQueuedEvents(ReentrantGuavaEventBus.java:57)
        at org.sonatype.sisu.goodies.eventbus.internal.guava.EventBus.post(EventBus.java:281)
        at org.sonatype.sisu.goodies.eventbus.internal.DefaultEventBus.post(DefaultEventBus.java:78)
        at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.insertRepository(DefaultRepositoryRegistry.java:235)
        at org.sonatype.nexus.proxy.registry.DefaultRepositoryRegistry.addRepository(DefaultRepositoryRegistry.java:76)
        at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.instantiateRepository(DefaultNexusConfiguration.java:687)
        at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.instantiateRepository(DefaultNexusConfiguration.java:665)
        at org.sonatype.nexus.configuration.application.DefaultNexusConfiguration.createRepository(DefaultNexusConfiguration.java:846)
      • locked org.sonatype.nexus.configuration.application.DefaultNexusConfiguration@35187b88
        at org.sonatype.nexus.templates.repository.AbstractRepositoryTemplateProvider.createRepository(AbstractRepositoryTemplateProvider.java:66)
        at org.sonatype.nexus.templates.repository.AbstractRepositoryTemplate.create(AbstractRepositoryTemplate.java:101)
        at org.sonatype.nexus.rest.repositories.RepositoryListPlexusResource.post(RepositoryListPlexusResource.java:145)
        at org.sonatype.plexus.rest.resource.RestletResource.acceptRepresentation(RestletResource.java:262)
        at org.sonatype.nexus.rest.NexusRestletResource.acceptRepresentation(NexusRestletResource.java:70)
        at org.restlet.resource.Resource.post(Resource.java:688)
        at org.restlet.resource.Resource.handlePost(Resource.java:537)

      Game over.

       

      Full thread dump attached.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            rseddon Rich Seddon
            Last Updated By:
            Peter Lynch
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title