Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-12780

task scheduler threads may deadlock at QuartzTaskJob.mayBlock() when more than 20 blocking tasks are encountered

    XMLWordPrintable

    Details

    • Story Points:
      3
    • Release Note:
      Yes
    • Sprint:
      Formats/Core Team - Sprint 92, Formats/Core Team - Sprint 93, Core Team - Sprint 94

      Description

      During some stress testing earlier, I added 100 Maven2 proxy repos and enabled "IQ: Audit and Quarantine" on them. Upon restarting Nexus, I was presented with 100 "Repository Audit" tasks that wouldn't move forward.

      From the thread dump:

      "quartz-4-thread-1" #211 prio=5 os_prio=0 tid=0x000000001b323800 nid=0xba0 waiting on condition [0x000000004a9ce000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ba350ea0> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
	at org.sonatype.nexus.quartz.internal.task.QuartzTaskFuture.get(QuartzTaskFuture.java:203)
	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.mayBlock(QuartzTaskJob.java:222)
	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:139)
	at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
	at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
	at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

      Per default configuration, 20 threads are allocated for Quartz, all of them are stuck like that in QuartzTaskJob.mayBlock().

        Attachments

        1. nexus.log
          297 kB
        2. tasks.png
          tasks.png
          95 kB
        3. thread-dump.log
          119 kB

          Issue Links

          relates

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-13121 tasks may appear as Starting or Cancelling indefinitely and cannot be stopped cancelled or deleted

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          testing discovered

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-12828 submitting more than 20 tasks at once causes ERROR QuartzSchedulerThread - ThreadPool.runInThread() return false! for some tasks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

            Activity

              People

              Assignee:
              bradbeck Brad Beck
              Reporter:
              bentmann Benjamin Bentmann
              Last Updated By:
              Peter Lynch
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Date of First Response:

                  tigCommentSecurity.panel-title