Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-19851

'Task already running' warnings for 'Log database table record counts'

    XMLWordPrintable

    Details

      Description

      Customer is seeing repeated errors on two of the three cluster nodes indicating that the task "Admin - Log database table record counts" is failing to start because it is already running.

       

      2019-04-30 00:00:00,010+0000 WARN  [hz.nexus.event-1] Repo_Mgr3 *SYSTEM com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor - Failed to START task 7bc2f2ea-b4a0-4e35-b4dd-9a9115d5bda1 as requested by remote node Member [10.xx.xx.xx]:5701 - d0065ea2-1d01-4c15-9163-a23c20f5e4d5
      java.lang.IllegalStateException: Task already running
       at com.google.common.base.Preconditions.checkState(Preconditions.java:504)
       at org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo.runNow(QuartzTaskInfo.java:249)
       at com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor.onMessage(HazelcastTaskDistributor.java:135)
       at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)
       at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61)
       at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47)
       at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
       at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

      Disabling the task while the cluster is running is not possible, attempting to do so fails because the task is marked as running.

      Initial Triage

      • The task prints out useful information about what is going on in the cluster to nexus_cluster.log file, but if it's causing problems it should be safe to delete it.
      • It gets created on startup unless the following is in nexus.properties: nexus.log.cluster.enabled=false
      • The task cannot be stopped while it's running so if its stuck the server will have to be restarted to stop it.

      Mitigation

      There is a system property which can disable the task that triggers this error.

      Upgrade to 3.16.2 or newer

      Version 3.16.2 has been released with the property nexus.log.cluster.enabled=false set by default, disabling the verbose logging.

      1. Upgrade to 3.16.2
      2. Go to the tasks view and delete any tasks named: Admin - Log database table record counts

      Versions 3.16.0 or 3.16.1

      1. Add nexus.log.cluster.enabled=false to nexus.properties file
      2. Go to the tasks view and wait at least 5 minutes, then refresh task list, to see if the task named Admin - Log database table record counts ever changes from Running to anything else and if so delete it.
      3. Otherwise restart the server and immediately delete the task.

        Attachments

          Activity

            People

            Assignee:
            mjohnson Matt Johnson
            Reporter:
            cseney Cassandra Seney
            CC:
            Jack Griffith
            Last Updated By:
            Peter Lynch
            Team:
            NXRM - Morpheus
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Date of First Response:

                tigCommentSecurity.panel-title