{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14686","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14686/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14686/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14686/events","html_url":"https://github.com/elastic/elasticsearch/issues/14686","id":116346333,"number":14686,"title":"Watcher/License plugins killed Elasticsearch cluster","user":{"login":"rhealitycheck","id":5863615,"avatar_url":"https://avatars.githubusercontent.com/u/5863615?v=3","gravatar_id":"","url":"https://api.github.com/users/rhealitycheck","html_url":"https://github.com/rhealitycheck","followers_url":"https://api.github.com/users/rhealitycheck/followers","following_url":"https://api.github.com/users/rhealitycheck/following{/other_user}","gists_url":"https://api.github.com/users/rhealitycheck/gists{/gist_id}","starred_url":"https://api.github.com/users/rhealitycheck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhealitycheck/subscriptions","organizations_url":"https://api.github.com/users/rhealitycheck/orgs","repos_url":"https://api.github.com/users/rhealitycheck/repos","events_url":"https://api.github.com/users/rhealitycheck/events{/privacy}","received_events_url":"https://api.github.com/users/rhealitycheck/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-11T14:36:06Z","updated_at":"2015-11-18T15:28:45Z","closed_at":null,"body":"So we've been having issues with watcher, which we installed to test out alerting on our Staging cluster. And this week we've had our Staging cluster completely die, unable to connect to any master node for longer than 5 mins at a time. Looking through the logs we found:\r\n\r\n    [2015-11-10 16:48:28,531][INFO ][license.plugin.core      ] [CHIVLXSTG045] license for [watcher] - valid\r\n    [2015-11-10 16:48:28,536][ERROR][watcher.license          ] [CHIVLXSTG045]\r\n    #\r\n    # Watcher license will expire on [Wednesday, December 09, 2015]. All configured actions on\r\n    # all registered watches are throttled (not executed) on Watcher license expiration.\r\n    # Watches will continue be evaluated and watch history will continue being recorded.\r\n    # Have a new license? please update it. Otherwise, please reach out to your support contact.\r\n    #\r\n\r\nBut this shouldn't be an issue for a month, however uninstalling the watcher and license plugins allowed the cluster to get back to a healthy state. Seems like this is a pretty large bug."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/157749204","html_url":"https://github.com/elastic/elasticsearch/issues/14686#issuecomment-157749204","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14686","id":157749204,"user":{"login":"rhealitycheck","id":5863615,"avatar_url":"https://avatars.githubusercontent.com/u/5863615?v=3","gravatar_id":"","url":"https://api.github.com/users/rhealitycheck","html_url":"https://github.com/rhealitycheck","followers_url":"https://api.github.com/users/rhealitycheck/followers","following_url":"https://api.github.com/users/rhealitycheck/following{/other_user}","gists_url":"https://api.github.com/users/rhealitycheck/gists{/gist_id}","starred_url":"https://api.github.com/users/rhealitycheck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhealitycheck/subscriptions","organizations_url":"https://api.github.com/users/rhealitycheck/orgs","repos_url":"https://api.github.com/users/rhealitycheck/repos","events_url":"https://api.github.com/users/rhealitycheck/events{/privacy}","received_events_url":"https://api.github.com/users/rhealitycheck/received_events","type":"User","site_admin":false},"created_at":"2015-11-18T15:28:45Z","updated_at":"2015-11-18T15:28:45Z","body":"I'm pretty sure we had it on all nodes but I honestly don't remember right now. The strangest part of this behavior was that we had the Watcher and License plugins installed for about a month and a half after which we saw this strange behavior."}}