Uploaded image for project: 'Dev - Nexus Repo'
  1. Dev - Nexus Repo
  2. NEXUS-8487

low disk watermark INFO messages should be adjusted

    Details

    • Type: Bug
    • Status: New
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-m4, 3.0.2, 3.1.0, 3.5.0, 3.21.0, 3.22.0, 3.24.0, 3.25.0, 3.28.0, 3.30.0, 3.31.0, 3.34.0, 3.35.0
    • Fix Version/s: None
    • Component/s: Logging, Search
    • Labels:
    • Notability:
      4

      Description

      Spammy messages every 30 seconds printed with seemingly plenty of space.

      2015-04-23 13:34:18,970-0300 INFO  [elasticsearch[nexus][management][T#2]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:34:48,969-0300 INFO  [elasticsearch[nexus][management][T#5]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:35:18,972-0300 INFO  [elasticsearch[nexus][management][T#2]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:35:48,973-0300 INFO  [elasticsearch[nexus][management][T#3]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:36:18,976-0300 INFO  [elasticsearch[nexus][management][T#2]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:36:48,979-0300 INFO  [elasticsearch[nexus][management][T#1]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:37:18,983-0300 INFO  [elasticsearch[nexus][management][T#3]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:37:48,986-0300 INFO  [elasticsearch[nexus][management][T#1]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:38:18,988-0300 INFO  [elasticsearch[nexus][management][T#4]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      2015-04-23 13:38:48,987-0300 INFO  [elasticsearch[nexus][management][T#2]] *SYSTEM org.elasticsearch.cluster.routing.allocation.decider - [nexus] low disk watermark [15%] exceeded on [c2my0fRGTB-llv9PlQN_Ow][nexus] free: 66gb[14.2%], replicas will not be assigned to this node
      

      Workaround

      Out-of-the box the default is set to warn when the disk gets to 90% - of course for huge disks this could still mean terabytes of storage is left.

      You can adjust the watermark by adding the following line to elasticsearch.yml:

      cluster.routing.allocation.disk.watermark.high: 99%
      

      You can also set it to a specific size if you prefer that instead of a percentage, but note:

      Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around.

      https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

      You can also tune how often the check occurs, and if you want you can turn it off completely:

      cluster.routing.allocation.disk.threshold_enabled: false
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              plynch Peter Lynch
              Last Updated By:
              Hardeep Nagra Hardeep Nagra
              Votes:
              6 Vote for this issue
              Watchers:
              15 Start watching this issue

                Dates

                Created:
                Updated:
                Date of First Response:

                  tigCommentSecurity.panel-title