{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/15333","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15333/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15333/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15333/events","html_url":"https://github.com/elastic/elasticsearch/issues/15333","id":121213523,"number":15333,"title":"Failure to recover shards after disk is full","user":{"login":"kpcool","id":2868966,"avatar_url":"https://avatars.githubusercontent.com/u/2868966?v=3","gravatar_id":"","url":"https://api.github.com/users/kpcool","html_url":"https://github.com/kpcool","followers_url":"https://api.github.com/users/kpcool/followers","following_url":"https://api.github.com/users/kpcool/following{/other_user}","gists_url":"https://api.github.com/users/kpcool/gists{/gist_id}","starred_url":"https://api.github.com/users/kpcool/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kpcool/subscriptions","organizations_url":"https://api.github.com/users/kpcool/orgs","repos_url":"https://api.github.com/users/kpcool/repos","events_url":"https://api.github.com/users/kpcool/events{/privacy}","received_events_url":"https://api.github.com/users/kpcool/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-12-09T11:11:10Z","updated_at":"2015-12-14T09:08:18Z","closed_at":null,"body":"Today, the disk got full and ElasticSearch is not able to go back again. Isn't there a built-in system that prevents such failures. I agree that we should be monitoring the hard space and not let this happen in first place, but some times things happen.\r\n\r\nMy setup is a single node at present. Using ES 2.1.0, which was supposed to have this fix.\r\n\r\nI don't see a clear way to recover the node. A post at https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html seemed to help, but still few indices got corrupted and I have no way to recovering them.\r\n\r\nAt the end, I ended up deleted the indices, but that's not the way it should be. Such things must be taken care of ultimately. But this is clearly a bug with ES 2.1.0"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/164383710","html_url":"https://github.com/elastic/elasticsearch/issues/15333#issuecomment-164383710","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15333","id":164383710,"user":{"login":"s1monw","id":973334,"avatar_url":"https://avatars.githubusercontent.com/u/973334?v=3","gravatar_id":"","url":"https://api.github.com/users/s1monw","html_url":"https://github.com/s1monw","followers_url":"https://api.github.com/users/s1monw/followers","following_url":"https://api.github.com/users/s1monw/following{/other_user}","gists_url":"https://api.github.com/users/s1monw/gists{/gist_id}","starred_url":"https://api.github.com/users/s1monw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/s1monw/subscriptions","organizations_url":"https://api.github.com/users/s1monw/orgs","repos_url":"https://api.github.com/users/s1monw/repos","events_url":"https://api.github.com/users/s1monw/events{/privacy}","received_events_url":"https://api.github.com/users/s1monw/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T09:08:18Z","updated_at":"2015-12-14T09:08:18Z","body":"alright I think I found the issue here @kpcool your logfiles brought the conclusion thanks you very much. This is actually a serious issue with our transaction log which basically corrupts itself when you hit a disk-full exception. I will keep you posted on this issue. Thanks for baring with me and helping to figure this out."}}