'Task already running' warnings for 'Log database table record counts'

Customer is seeing repeated errors on two of the three cluster nodes indicating that the task "Admin - Log database table record counts" is failing to start because it is already running.

2019-04-30 00:00:00,010+0000 WARN  [hz.nexus.event-1] Repo_Mgr3 *SYSTEM com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor - Failed to START task 7bc2f2ea-b4a0-4e35-b4dd-9a9115d5bda1 as requested by remote node Member [10.xx.xx.xx]:5701 - d0065ea2-1d01-4c15-9163-a23c20f5e4d5
java.lang.IllegalStateException: Task already running
 at com.google.common.base.Preconditions.checkState(Preconditions.java:504)
 at org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo.runNow(QuartzTaskInfo.java:249)
 at com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor.onMessage(HazelcastTaskDistributor.java:135)
 at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)
 at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61)
 at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47)
 at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
 at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)

Disabling the task while the cluster is running is not possible, attempting to do so fails because the task is marked as running.

Initial Triage

• The task prints out useful information about what is going on in the cluster to nexus_cluster.log file, but if it's causing problems it should be safe to delete it.
• It gets created on startup unless the following is in nexus.properties: nexus.log.cluster.enabled=false
• The task cannot be stopped while it's running so if its stuck the server will have to be restarted to stop it.

Mitigation

Upgrade to 3.16.2 or newer

Version 3.16.2 has been released with the property nexus.log.cluster.enabled=false set by default, disabling the verbose logging.

1. Upgrade to 3.16.2
2. Go to the tasks view and delete any tasks named: Admin - Log database table record counts

Versions 3.16.0 or 3.16.1

1. Add nexus.log.cluster.enabled=false to nexus.properties file
2. Go to the tasks view and wait at least 5 minutes, then refresh task list, to see if the task named Admin - Log database table record counts ever changes from Running to anything else and if so delete it.
3. Otherwise restart the server and immediately delete the task.