{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/8911","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/8911/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/8911/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/8911/events","html_url":"https://github.com/elasticsearch/elasticsearch/issues/8911","id":51736320,"number":8911,"title":"Replicas can not get initiliased as they never catch up.","user":{"login":"bluelu","id":339893,"avatar_url":"https://avatars.githubusercontent.com/u/339893?v=3","gravatar_id":"","url":"https://api.github.com/users/bluelu","html_url":"https://github.com/bluelu","followers_url":"https://api.github.com/users/bluelu/followers","following_url":"https://api.github.com/users/bluelu/following{/other_user}","gists_url":"https://api.github.com/users/bluelu/gists{/gist_id}","starred_url":"https://api.github.com/users/bluelu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bluelu/subscriptions","organizations_url":"https://api.github.com/users/bluelu/orgs","repos_url":"https://api.github.com/users/bluelu/repos","events_url":"https://api.github.com/users/bluelu/events{/privacy}","received_events_url":"https://api.github.com/users/bluelu/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2014-12-11T20:51:14Z","updated_at":"2015-02-06T14:35:52Z","closed_at":"2015-02-06T14:35:52Z","body":"We had this happening already on 2 occasions now, still we don't know what exactly triggers this. Didn't see this 1.0.*\r\n\r\nWe index about 350 million entries (+updates) per day over 10 servers (including 10 shards) on ssd disks. It's about 75GB per server, thus 35 million entries per shard/server.\r\n\r\nCluster turned completely green except for two shards (seperate indexes) which were stuck in RECOVERING. We checked the replicas and they had over 10000 files in the index directories (mainly small ones). At that time there was basicly no traffic in the cluster, so it could not be a bandwith issue.\r\n\r\nWe had to turn off our indexing in order to get it to go to the next phase: After a few minutes (not directly), the replicas started recovering the transaction log. We checked the transaction log on the main node, it was over 35GB big and streamed very slowly to the replica. We stopped there, turned off the allocation and removed the directories and restarted the nodes. Then it quickly turned green and we turned on indexing again.\r\n\r\nSince we restarted the cluster before, there could have been a lot of traffic copying over all the shards again as we had deleted the old checksum files because of the checksum bug. We will upgrade our indexes to the new ES version to fix this.\r\n\r\nI don't know when our cluster started recovering those 2 node, but it could have started right away when there was more traffic. Still afterwards, no bandwith issues should have been there.\r\n\r\nStill, it should recover after some time, without us having to stop indexing?\r\n"},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/comments/73245086","html_url":"https://github.com/elasticsearch/elasticsearch/issues/8911#issuecomment-73245086","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/8911","id":73245086,"user":{"login":"bluelu","id":339893,"avatar_url":"https://avatars.githubusercontent.com/u/339893?v=3","gravatar_id":"","url":"https://api.github.com/users/bluelu","html_url":"https://github.com/bluelu","followers_url":"https://api.github.com/users/bluelu/followers","following_url":"https://api.github.com/users/bluelu/following{/other_user}","gists_url":"https://api.github.com/users/bluelu/gists{/gist_id}","starred_url":"https://api.github.com/users/bluelu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bluelu/subscriptions","organizations_url":"https://api.github.com/users/bluelu/orgs","repos_url":"https://api.github.com/users/bluelu/repos","events_url":"https://api.github.com/users/bluelu/events{/privacy}","received_events_url":"https://api.github.com/users/bluelu/received_events","type":"User","site_admin":false},"created_at":"2015-02-06T14:35:52Z","updated_at":"2015-02-06T14:35:52Z","body":"We don't have the log files anymore, so I don't know as it never appeared anymore. I will close the issue since I think it's related to that. "}}