Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.16.0, 3.16.1
-
Component/s: HA
-
Labels:
Description
Customer is seeing repeated errors on two of the three cluster nodes indicating that the task "Admin - Log database table record counts" is failing to start because it is already running.
2019-04-30 00:00:00,010+0000 WARN [hz.nexus.event-1] Repo_Mgr3 *SYSTEM com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor - Failed to START task 7bc2f2ea-b4a0-4e35-b4dd-9a9115d5bda1 as requested by remote node Member [10.xx.xx.xx]:5701 - d0065ea2-1d01-4c15-9163-a23c20f5e4d5
java.lang.IllegalStateException: Task already running
at com.google.common.base.Preconditions.checkState(Preconditions.java:504)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo.runNow(QuartzTaskInfo.java:249)
at com.sonatype.nexus.hazelcast.internal.scheduling.HazelcastTaskDistributor.onMessage(HazelcastTaskDistributor.java:135)
at com.hazelcast.topic.impl.TopicService.dispatchEvent(TopicService.java:136)
at com.hazelcast.spi.impl.eventservice.impl.EventProcessor.process(EventProcessor.java:61)
at com.hazelcast.spi.impl.eventservice.impl.RemoteEventProcessor.run(RemoteEventProcessor.java:47)
at com.hazelcast.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:226)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:209)
Disabling the task while the cluster is running is not possible, attempting to do so fails because the task is marked as running.
Initial Triage
- The task prints out useful information about what is going on in the cluster to nexus_cluster.log file, but if it's causing problems it should be safe to delete it.
- It gets created on startup unless the following is in nexus.properties: nexus.log.cluster.enabled=false
- The task cannot be stopped while it's running so if its stuck the server will have to be restarted to stop it.
Mitigation
There is a system property which can disable the task that triggers this error.
Upgrade to 3.16.2 or newer
Version 3.16.2 has been released with the property nexus.log.cluster.enabled=false set by default, disabling the verbose logging.
- Upgrade to 3.16.2
- Go to the tasks view and delete any tasks named: Admin - Log database table record counts
Versions 3.16.0 or 3.16.1
- Add nexus.log.cluster.enabled=false to nexus.properties file
- Go to the tasks view and wait at least 5 minutes, then refresh task list, to see if the task named Admin - Log database table record counts ever changes from Running to anything else and if so delete it.
- Otherwise restart the server and immediately delete the task.