Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-12780

task scheduler threads may deadlock at QuartzTaskJob.mayBlock() when more than 20 blocking tasks are encountered

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Release Note:
      Yes
    • Sprint:
      Formats/Core Team - Sprint 92, Formats/Core Team - Sprint 93, Core Team - Sprint 94

      Description

      During some stress testing earlier, I added 100 Maven2 proxy repos and enabled "IQ: Audit and Quarantine" on them. Upon restarting Nexus, I was presented with 100 "Repository Audit" tasks that wouldn't move forward.

      From the thread dump:

      "quartz-4-thread-1" #211 prio=5 os_prio=0 tid=0x000000001b323800 nid=0xba0 waiting on condition [0x000000004a9ce000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000ba350ea0> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskFuture.get(QuartzTaskFuture.java:203)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.mayBlock(QuartzTaskJob.java:222)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:139)
      	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
      	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
      	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Per default configuration, 20 threads are allocated for Quartz, all of them are stuck like that in QuartzTaskJob.mayBlock().

        Attachments

        1. nexus.log
          297 kB
          Benjamin Bentmann
        2. tasks.png
          95 kB
          Benjamin Bentmann
        3. thread-dump.log
          119 kB
          Benjamin Bentmann

          Issue Links

            Activity

              People

              Assignee:
              bradbeck Brad Beck
              Reporter:
              bentmann Benjamin Bentmann
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title