Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-22855

If too many tasks are scheduled to run simultaneously Nexus Repo will get into a livelock

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: New
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.20.1
    • Fix Version/s: None
    • Component/s: Scheduled Tasks
    • Labels:
    • Notability:
      n/a

      Description

       

      If a very large number of tasks are scheduled to run simultaneously Nexus Repo will get into a state where they are all in a livelock. The bottleneck is occurs because the "mayBlock()" method lists all running tasks. This is an expensive operation. Once in this state, the tasks will never run.

      See NEXUS-22853 for an example of how this can happen, but note that is just one example, there are likely other ways for this to occur.

      Expected: The listing of running tasks is an expensive operation, which as caused other performance problems (see NEXUS-22760). We should consider some way to reduce the impact of this call so that it does not block common operations like scheduling new tasks to run.

      "quartz-3-thread-1" id=214 state=BLOCKED
          - waiting to lock <0x7941cdc6> (a org.sonatype.goodies.common.Mutex)
          owned by quartz-3-thread-20 id=726
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.mayBlock(QuartzTaskJob.java:198)
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:139)
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
          at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
          at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
          at org.sonatype.nexus.quartz.internal.QuartzThreadPool$$Lambda$541/1136420661.run(Unknown Source)
          at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
          at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
          at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@6b20914d
      
      "quartz-3-thread-20" id=726 state=RUNNABLE
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:348)
          at com.fasterxml.jackson.databind.type.TypeFactory.classForName(TypeFactory.java:306)
          at com.fasterxml.jackson.databind.type.TypeFactory.findClass(TypeFactory.java:288)
          at com.fasterxml.jackson.databind.DeserializationContext.findClass(DeserializationContext.java:557)
          at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer$Std._deserialize(FromStringDeserializer.java:243)
          at com.fasterxml.jackson.databind.deser.std.FromStringDeserializer.deserialize(FromStringDeserializer.java:145)
          at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
          at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:251)
          at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1575)
          at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:965)
          at org.sonatype.nexus.quartz.internal.orient.JacksonMarshaller.unmarshall(JacksonMarshaller.java:75)
          at org.sonatype.nexus.quartz.internal.orient.MarshalledEntityAdapter.readFields(MarshalledEntityAdapter.java:71)
          at org.sonatype.nexus.quartz.internal.orient.JobDetailEntityAdapter.readFields(JobDetailEntityAdapter.java:109)
          at org.sonatype.nexus.quartz.internal.orient.JobDetailEntityAdapter.readFields(JobDetailEntityAdapter.java:1)
          at org.sonatype.nexus.orient.entity.EntityAdapter.readEntity(EntityAdapter.java:286)
          at org.sonatype.nexus.orient.entity.IterableEntityAdapter.transformEntity(IterableEntityAdapter.java:63)
          at org.sonatype.nexus.orient.entity.IterableEntityAdapter$$Lambda$312/1020833709.apply(Unknown Source)
          at com.google.common.collect.Iterators$6.transform(Iterators.java:788)
          at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
          at com.google.common.collect.Iterators$5.computeNext(Iterators.java:638)
          at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
          at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
          at com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
          at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
          at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.getJobKeys(JobStoreImpl.java:436)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.lambda$9(JobStoreImpl.java:426)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl$$Lambda$377/604177624.execute(Unknown Source)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl$$Lambda$378/25765727.apply(Unknown Source)
          at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$1(OrientOperations.java:55)
          at org.sonatype.nexus.orient.transaction.OrientOperations$$Lambda$288/1920159934.call(Unknown Source)
          at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
          at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
          at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
          at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
          at org.sonatype.nexus.transaction.Operations.call(Operations.java:166)
          at org.sonatype.nexus.orient.transaction.OrientOperations.call(OrientOperations.java:55)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.execute(JobStoreImpl.java:204)
          at org.sonatype.nexus.quartz.internal.orient.JobStoreImpl.getJobKeys(JobStoreImpl.java:426)
          at org.quartz.core.QuartzScheduler.getJobKeys(QuartzScheduler.java:1433)
          at org.quartz.impl.StdScheduler.getJobKeys(StdScheduler.java:470)
          at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI.allTasks(QuartzSchedulerSPI.java:710)
          at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI.listsTasks(QuartzSchedulerSPI.java:534)
          at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI$$EnhancerByGuice$$be403ec6.CGLIB$listsTasks$8(<generated>)
          at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI$$EnhancerByGuice$$be403ec6$$FastClassByGuice$$4d608bb2.invoke(<generated>)
          at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
          at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
          at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
          at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
          at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:77)
          at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55)
          at org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI$$EnhancerByGuice$$be403ec6.listsTasks(<generated>)
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.blockedBy(QuartzTaskJob.java:264)
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.mayBlock(QuartzTaskJob.java:200)
            - locked org.sonatype.goodies.common.Mutex@7941cdc6
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:139)
          at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
          at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
          at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
          at org.sonatype.nexus.quartz.internal.QuartzThreadPool$$Lambda$541/1136420661.run(Unknown Source)
          at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
          at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
          at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      
          Locked synchronizers: count = 1
            - java.util.concurrent.ThreadPoolExecutor$Worker@464b3500
      

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              rseddon Rich Seddon
              Last Updated By:
              Wes Wannemacher Wes Wannemacher
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title