Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16912

WARN QuartzSchedulerSPI Job missing listener omitting from results

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.10.0
    • Fix Version/s: None
    • Component/s: HA, Scheduled Tasks
    • Labels:
    • Story Points:
      5
    • Notability:
      2

      Description

      While the exact cause is still undetermined, QuartzSchedulerSPI WARN level messages have been noticed inside Nexus 3 application logs. Examples

      Example WARN messages
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.2d62e436-d3e3-4628-9a90-fe31f4172c74
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.d45283d9-fb96-42d5-9d51-551838befdcd
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.280db4c4-9aab-43de-a0b8-065f9a7095ce
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.6ad857f0-f5bf-41ba-ad90-d9a0ce090d3f
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.e6043eee-7fc5-404f-b858-61c9a9d70d06
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.249a2184-7779-4d78-8d7b-21d004f92ef2
      2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.df3f95c0-9ace-48ea-8c70-fd07a6af628a
      2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.9aba2886-7efa-4976-95a9-7e934446b439
      2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.c2329509-c5bb-43e6-afc5-2d6014e2acec
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.2d62e436-d3e3-4628-9a90-fe31f4172c74
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.d45283d9-fb96-42d5-9d51-551838befdcd
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.280db4c4-9aab-43de-a0b8-065f9a7095ce
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.6ad857f0-f5bf-41ba-ad90-d9a0ce090d3f
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.e6043eee-7fc5-404f-b858-61c9a9d70d06
      2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.249a2184-7779-4d78-8d7b-21d004f92ef2
      

      At the same time, other unexplained or broken behaviour is appearing in the same Nexus instances, which usually can be tied back to having some sort of requirement on scheduled task processing. Possibly related are the fact these messages have recently been detected in HA-C environments where the connectivity between nodes in a cluster was suspect.

      Diagnosis

      Reliable steps to reproduce the runtime state where these messages start appearing is under review. What seems consistent is that when nodes in an HA cluster cannot successfully sync, particularly at abrupt shutdown of a node, tasks may lose proper state and hence their triggers. Most commonly the situation arises with Repository Health Check Scheduled tasks.

      Short Term Mitigation for Large Log Files: Reduce Noisy Log Messages

      To prevent the verbose log messages related to this problem from filling disk, short term an administrator can set the logger level of org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI to ERROR.

      While this step silences the logging, we recommend taking additional steps in *Repairing Broken Tasks" to fix the problems suggested by the symptoms.

      A proper adjustment is delivered via NEXUS-19224 made in 3.17.0 as well as a system status check that warns of the condition.

      Temporary Fix: Repairing Broken Tasks (Nexus version 3.10-3.15)

      This step is repair broken tasks. Running this script may not prevent the problem from re-occuring.

      Download the attached script source, NEXUS-16912-repair_missing_triggers-3.10-to-3.15.groovy create an Admin - Execute Script scheduled task and paste the contents of the script into the task source. After saving the task, manually run the task. After the task completes, it will be safe to return the org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI logger back to INFO level.

      Temporary Fix: Repairing Broken Tasks ( NXRM version 3.16 and newer )

      As outcome of NEXUS-20019, download the attached script source, NEXUS-16912-repair_missing_triggers-3.16-and-newer.groovy create an Admin - Execute Script scheduled task and paste the contents of the script into the task source. After saving the task, manually run the task. After the task completes, it will be safe to return the org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI logger back to INFO level.

      Expected

      Tasks should not lose internal triggers - take steps to prevent this condition from being possible, or if not possible to prevent in all circumstances - the condition should be self healing.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Mahendra Surani Mahendra Surani
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title