Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19851

'Task already running' warnings for 'Log database table record counts'

    XMLWordPrintable

    Details

      Description

      Customer is seeing repeated errors on two of the three cluster nodes indicating that the task "Admin - Log database table record counts" is failing to start because it is already running.

       

      2019-04-30 00:00:00,010+0000 WARN  [hz.nexus.event-1] Repo_Mgr3 *SYSTEM com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor - Failed to START task 7bc2f2ea-b4a0-4e35-b4dd-9a9115d5bda1 as requested by remote node Member [10.xx.xx.xx]:5701 - d0065ea2-1d01-4c15-9163-a23c20f5e4d5
      java.lang.IllegalStateException: Task already running
       at com.google.common.base.Preconditions.checkState(Preconditions.java:504)
       at org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo.runNow(QuartzTaskInfo.java:249)
       at com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor.onMessage(HazelcastTaskDistributor.java:135)
       at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)
       at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61)
       at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47)
       at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
       at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

      Disabling the task while the cluster is running is not possible, attempting to do so fails because the task is marked as running.

      Initial Triage

      • The task prints out useful information about what is going on in the cluster to nexus_cluster.log file, but if it's causing problems it should be safe to delete it.
      • It gets created on startup unless the following is in nexus.properties: nexus.log.cluster.enabled=false
      • The task cannot be stopped while it's running so if its stuck the server will have to be restarted to stop it.

      Mitigation

      Upgrade to 3.16.2 or newer

      Version 3.16.2 has been released with the property nexus.log.cluster.enabled=false set by default, disabling the verbose logging.

      1. Upgrade to 3.16.2
      2. Go to the tasks view and delete any tasks named: Admin - Log database table record counts

      Versions 3.16.0 or 3.16.1

      1. Add nexus.log.cluster.enabled=false to nexus.properties file
      2. Go to the tasks view and wait at least 5 minutes, then refresh task list, to see if the task named Admin - Log database table record counts ever changes from Running to anything else and if so delete it.
      3. Otherwise restart the server and immediately delete the task.

        Attachments

          Activity

            People

            • Assignee:
              mjohnson Matt Johnson
              Reporter:
              cseney Cassandra Seney
              CC:
              Jack Griffith
              Last Updated By:
              Rich Seddon
              Team:
              NXRM - Morpheus
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Date of First Response: