'Task already running' warnings for 'Log database table record counts'

Customer is seeing repeated errors on two of the three cluster nodes indicating that the task "Admin - Log database table record counts" is failing to start because it is already running.

2019-04-30 00:00:00,010+0000 WARN  [hz.nexus.event-1] Repo_Mgr3 *SYSTEM com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor - Failed to START task 7bc2f2ea-b4a0-4e35-b4dd-9a9115d5bda1 as requested by remote node Member [10.xx.xx.xx]:5701 - d0065ea2-1d01-4c15-9163-a23c20f5e4d5
java.lang.IllegalStateException: Task already running
 at com.google.common.base.Preconditions.checkState(Preconditions.java:504)
 at org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo.runNow(QuartzTaskInfo.java:249)
 at com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor.onMessage(HazelcastTaskDistributor.java:135)
 at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)
 at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61)
 at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47)
 at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
 at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

Disabling the task while the cluster is running is not possible, attempting to do so fails because the task is marked as running.

Initial Triage

• The task prints out useful information about what is going on in the cluster to nexus_cluster.log file, but if it's causing problems it should be safe to delete it.
• It gets created on startup unless the following is in nexus.properties: nexus.log.cluster.enabled=false
• The task cannot be stopped while it's running so if its stuck the server will have to be restarted to stop it.

Mitigation

There is a system property which can disable the task that triggers this error.

Upgrade to 3.16.2 or newer

Version 3.16.2 has been released with the property nexus.log.cluster.enabled=false set by default, disabling the verbose logging.

1. Upgrade to 3.16.2
2. Go to the tasks view and delete any tasks named: Admin - Log database table record counts

Versions 3.16.0 or 3.16.1

1. Add nexus.log.cluster.enabled=false to nexus.properties file
2. Go to the tasks view and wait at least 5 minutes, then refresh task list, to see if the task named Admin - Log database table record counts ever changes from Running to anything else and if so delete it.
3. Otherwise restart the server and immediately delete the task.