{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14194","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14194/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14194/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14194/events","html_url":"https://github.com/elastic/elasticsearch/issues/14194","id":112145255,"number":14194,"title":"Low Disk Watermark Logging Oddness","user":{"login":"skearns64","id":1406684,"avatar_url":"https://avatars.githubusercontent.com/u/1406684?v=3","gravatar_id":"","url":"https://api.github.com/users/skearns64","html_url":"https://github.com/skearns64","followers_url":"https://api.github.com/users/skearns64/followers","following_url":"https://api.github.com/users/skearns64/following{/other_user}","gists_url":"https://api.github.com/users/skearns64/gists{/gist_id}","starred_url":"https://api.github.com/users/skearns64/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skearns64/subscriptions","organizations_url":"https://api.github.com/users/skearns64/orgs","repos_url":"https://api.github.com/users/skearns64/repos","events_url":"https://api.github.com/users/skearns64/events{/privacy}","received_events_url":"https://api.github.com/users/skearns64/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Allocation","name":":Allocation","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Logging","name":":Logging","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"}],"state":"open","locked":false,"assignee":{"login":"dakrone","id":19060,"avatar_url":"https://avatars.githubusercontent.com/u/19060?v=3","gravatar_id":"","url":"https://api.github.com/users/dakrone","html_url":"https://github.com/dakrone","followers_url":"https://api.github.com/users/dakrone/followers","following_url":"https://api.github.com/users/dakrone/following{/other_user}","gists_url":"https://api.github.com/users/dakrone/gists{/gist_id}","starred_url":"https://api.github.com/users/dakrone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakrone/subscriptions","organizations_url":"https://api.github.com/users/dakrone/orgs","repos_url":"https://api.github.com/users/dakrone/repos","events_url":"https://api.github.com/users/dakrone/events{/privacy}","received_events_url":"https://api.github.com/users/dakrone/received_events","type":"User","site_admin":false},"milestone":null,"comments":3,"created_at":"2015-10-19T13:33:34Z","updated_at":"2015-10-21T19:53:17Z","closed_at":null,"body":"Using a 2.0.0-snapshot build, I see odd logging behavior for the low disk watermark. \r\n\r\nThe free-disk calculation is working fine, but the logging for this is quite odd. After starting a single-node cluster with 5 shards (each with 1 shard, and 1 unassigned replica), I see several log messages about low disk watermark all at once. 30 seconds later, I see another group of low disk watermark messages, but each time there are more messages than previously. First time was 4 identical log lines, then 7, then 10, then 13.\r\n\r\nI was able to reproduce several times after restarting the nodes. I inserted newlines into the excerpted logs below. This could become a real problem if the trend continues. \r\n\r\n\r\n\r\n```\r\n[2015-10-19 09:21:12,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:12,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:12,203][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:12,203][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n\r\n\r\n[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,201][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:21:42,202][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n\r\n\r\n[2015-10-19 09:22:12,206][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,207][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:12,208][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n\r\n\r\n[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,211][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,212][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:22:42,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n\r\n\r\n[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,213][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,214][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:12,215][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.3gb[14.7%], replicas will not be assigned to this node\r\n\r\n\r\n[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,217][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,218][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,219][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n[2015-10-19 09:23:42,220][INFO ][cluster.routing.allocation.decider] [Solara] low disk watermark [85%] exceeded on [XpVtZKlVQFu80lnBEGdh9A][Solara][/skearns/es/elasticsearch-2.0.0-SNAPSHOT/data/elasticsearch/nodes/0] free: 34.4gb[14.7%], replicas will not be assigned to this node\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/150007165","html_url":"https://github.com/elastic/elasticsearch/issues/14194#issuecomment-150007165","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14194","id":150007165,"user":{"login":"dakrone","id":19060,"avatar_url":"https://avatars.githubusercontent.com/u/19060?v=3","gravatar_id":"","url":"https://api.github.com/users/dakrone","html_url":"https://github.com/dakrone","followers_url":"https://api.github.com/users/dakrone/followers","following_url":"https://api.github.com/users/dakrone/following{/other_user}","gists_url":"https://api.github.com/users/dakrone/gists{/gist_id}","starred_url":"https://api.github.com/users/dakrone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dakrone/subscriptions","organizations_url":"https://api.github.com/users/dakrone/orgs","repos_url":"https://api.github.com/users/dakrone/repos","events_url":"https://api.github.com/users/dakrone/events{/privacy}","received_events_url":"https://api.github.com/users/dakrone/received_events","type":"User","site_admin":false},"created_at":"2015-10-21T19:53:17Z","updated_at":"2015-10-21T19:53:17Z","body":"More info on this. The multiple logging lines is caused by `DiskThresholdDecider` not being bound as a singleton. Usually this wouldn't cause any side effects for Elasticsearch, however, Marvel injects the DTD into its `NodesStatsCollecter` with a `Provider<...>`, which was creating a new instance of `DiskThresholdDecider` every time it collected the stats. The additional instances were then logging, which is why the logging increased in number every time it ran."}}