Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-12780

task scheduler threads may deadlock at QuartzTaskJob.mayBlock() when more than 20 blocking tasks are encountered

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Release Note:
      Yes
    • Sprint:
      Formats/Core Team - Sprint 92, Formats/Core Team - Sprint 93, Core Team - Sprint 94

      Description

      During some stress testing earlier, I added 100 Maven2 proxy repos and enabled "IQ: Audit and Quarantine" on them. Upon restarting Nexus, I was presented with 100 "Repository Audit" tasks that wouldn't move forward.

      From the thread dump:

      "quartz-4-thread-1" #211 prio=5 os_prio=0 tid=0x000000001b323800 nid=0xba0 waiting on condition [0x000000004a9ce000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000ba350ea0> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskFuture.get(QuartzTaskFuture.java:203)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.mayBlock(QuartzTaskJob.java:222)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:139)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Per default configuration, 20 threads are allocated for Quartz, all of them are stuck like that in QuartzTaskJob.mayBlock().

        Attachments

        1. nexus.log
          297 kB
        2. tasks.png
          tasks.png
          95 kB
        3. thread-dump.log
          119 kB

          Issue Links

            Activity

              People

              Assignee:
              bradbeck Brad Beck
              Reporter:
              bentmann Benjamin Bentmann
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title