Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-16912

WARN QuartzSchedulerSPI Job missing listener omitting from results

    XMLWordPrintable

Details

    • Bug
    • Resolution: Parked
    • Major
    • None
    • 3.10.0
    • HA, Scheduled Tasks
    • 5
    • 2
    • 2

    Description

      While the exact cause is still undetermined, QuartzSchedulerSPI WARN level messages have been noticed inside Nexus 3 application logs. Examples

      Example WARN messages
      2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.2d62e436-d3e3-4628-9a90-fe31f4172c74
2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.d45283d9-fb96-42d5-9d51-551838befdcd
2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.280db4c4-9aab-43de-a0b8-065f9a7095ce
2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.6ad857f0-f5bf-41ba-ad90-d9a0ce090d3f
2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.e6043eee-7fc5-404f-b858-61c9a9d70d06
2018-03-06 15:57:05,962-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.249a2184-7779-4d78-8d7b-21d004f92ef2
2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.df3f95c0-9ace-48ea-8c70-fd07a6af628a
2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.9aba2886-7efa-4976-95a9-7e934446b439
2018-03-06 15:57:05,963-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.c2329509-c5bb-43e6-afc5-2d6014e2acec
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.2d62e436-d3e3-4628-9a90-fe31f4172c74
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.d45283d9-fb96-42d5-9d51-551838befdcd
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.280db4c4-9aab-43de-a0b8-065f9a7095ce
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.6ad857f0-f5bf-41ba-ad90-d9a0ce090d3f
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.e6043eee-7fc5-404f-b858-61c9a9d70d06
2018-03-06 15:57:05,965-0500 WARN  [pool-23-thread-1] NODE3 anonymous org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI - Job missing listener; omitting from results: nexus.249a2184-7779-4d78-8d7b-21d004f92ef2

      At the same time, other unexplained or broken behaviour is appearing in the same Nexus instances, which usually can be tied back to having some sort of requirement on scheduled task processing. Possibly related are the fact these messages have recently been detected in HA-C environments where the connectivity between nodes in a cluster was suspect.

      Diagnosis

      Reliable steps to reproduce the runtime state where these messages start appearing is under review. What seems consistent is that when nodes in an HA cluster cannot successfully sync, particularly at abrupt shutdown of a node, tasks may lose proper state and hence their triggers. Most commonly the situation arises with Repository Health Check Scheduled tasks.

      Short Term Mitigation for Large Log Files: Reduce Noisy Log Messages

      To prevent the verbose log messages related to this problem from filling disk, short term an administrator can set the logger level of org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI to ERROR.

      While this step silences the logging, we recommend taking additional steps in *Repairing Broken Tasks" to fix the problems suggested by the symptoms.

      A proper adjustment is delivered via NEXUS-19224 made in 3.17.0 as well as a system status check that warns of the condition.

      Temporary Fix: Repairing Broken Tasks (Nexus version 3.10-3.15)

      This step is repair broken tasks. Running this script may not prevent the problem from re-occuring.

      Download the attached script source, NEXUS-16912-repair_missing_triggers-3.10-to-3.15.groovy create an Admin - Execute Script scheduled task and paste the contents of the script into the task source. After saving the task, manually run the task. After the task completes, it will be safe to return the org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI logger back to INFO level.

      Temporary Fix: Repairing Broken Tasks ( NXRM version 3.16 and newer )

      As outcome of NEXUS-20019, download the attached script source, NEXUS-16912-repair_missing_triggers-3.16-and-newer.groovy create an Admin - Execute Script scheduled task and paste the contents of the script into the task source. After saving the task, manually run the task. After the task completes, it will be safe to return the org.sonatype.nexus.quartz.internal.QuartzSchedulerSPI logger back to INFO level.

      Expected

      Tasks should not lose internal triggers - take steps to prevent this condition from being possible, or if not possible to prevent in all circumstances - the condition should be self healing.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-19224 Reduce log spam for missing listeners

          • Major - Major loss of function.
          • Closed
          relates

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-18983 If NXRM is read-only or lacks quorum, then run now triggers make startup fail.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. NEXUS-20019 repair script needed for Job missing listener log warning

          • Major - Major loss of function.
          • Closed

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. NEXUS-18185 Database out of sync in HA cluster - mitigation and recovery

          • Major - Major loss of function.
          • Done

          Activity

            People

              Unassigned Unassigned
              plynch Peter Lynch
              Michael Oliverio Michael Oliverio
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                tigCommentSecurity.panel-title