{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9685","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9685/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9685/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9685/events","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9685","id":57554072,"number":9685,"title":"One node stuck, not responding and brings down the entire cluster","user":{"login":"mkliu","id":1469524,"avatar_url":"https://avatars.githubusercontent.com/u/1469524?v=3","gravatar_id":"","url":"https://api.github.com/users/mkliu","html_url":"https://github.com/mkliu","followers_url":"https://api.github.com/users/mkliu/followers","following_url":"https://api.github.com/users/mkliu/following{/other_user}","gists_url":"https://api.github.com/users/mkliu/gists{/gist_id}","starred_url":"https://api.github.com/users/mkliu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkliu/subscriptions","organizations_url":"https://api.github.com/users/mkliu/orgs","repos_url":"https://api.github.com/users/mkliu/repos","events_url":"https://api.github.com/users/mkliu/events{/privacy}","received_events_url":"https://api.github.com/users/mkliu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-02-13T03:18:00Z","updated_at":"2015-02-14T00:59:26Z","closed_at":"2015-02-13T20:18:40Z","body":"Hi, we had two incidents of this now. We had dedicated master and client nodes. So the symptom is:\r\n1. Management APIs such as _node, cat are not returning. Default the default 9200 response. If I directly hit the master node, default 9200 is returning 200. But the other APIs are not working.\r\n2. No out of memory exception yet. We set HEAP at 20GB, the but the usage is about 15GB only. ( Could it be because of this? Machine is 32GB memory)\r\n3. I restarted a couple of high memory nodes, and master too, still not recovering. Until I found on some master node logs pointing to a node saying operation cannot be executed on bad node.\r\n4. The bad node's log is missing an entire time period since a couple of hours ago. And in Marvel, the node stopped reporting status around the same time too. Didn't see anything suspicious on Marvel events though.\r\n     4.a. First time this happens, the machine is doing some GC operation and stuck\r\n     4.b Second time this happens there's no obvious problem exception some index operation failing. And this time I checked the field_data size too, it's not big, around 1GB only. \r\n\r\nAny clue or else I should look at?"},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/comments/74353118","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9685#issuecomment-74353118","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9685","id":74353118,"user":{"login":"mkliu","id":1469524,"avatar_url":"https://avatars.githubusercontent.com/u/1469524?v=3","gravatar_id":"","url":"https://api.github.com/users/mkliu","html_url":"https://github.com/mkliu","followers_url":"https://api.github.com/users/mkliu/followers","following_url":"https://api.github.com/users/mkliu/following{/other_user}","gists_url":"https://api.github.com/users/mkliu/gists{/gist_id}","starred_url":"https://api.github.com/users/mkliu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkliu/subscriptions","organizations_url":"https://api.github.com/users/mkliu/orgs","repos_url":"https://api.github.com/users/mkliu/repos","events_url":"https://api.github.com/users/mkliu/events{/privacy}","received_events_url":"https://api.github.com/users/mkliu/received_events","type":"User","site_admin":false},"created_at":"2015-02-14T00:59:26Z","updated_at":"2015-02-14T00:59:26Z","body":"Sorry about that, I've sent question to mailing list already too."}}