{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/12463","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12463/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12463/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12463/events","html_url":"https://github.com/elastic/elasticsearch/pull/12463","id":97252233,"number":12463,"title":"IndicesStore shouldn't try to delete index after deleting a shard","user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Internal","name":":Internal","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/resiliency","name":"resiliency","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v1.6.2","name":"v1.6.2","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v1.7.1","name":"v1.7.1","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.0.0","name":"v2.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-07-25T20:15:48Z","updated_at":"2015-07-27T12:35:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/12463","html_url":"https://github.com/elastic/elasticsearch/pull/12463","diff_url":"https://github.com/elastic/elasticsearch/pull/12463.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/12463.patch"},"body":"When a node discovers shard content on disk which isn't used, we reach out to all other nodes that supposed to have the shard active. Only once all of those have confirmed the shard active, the shard has no unassigned copies *and* no cluster state change have happened in the mean while, do we go and delete the shard folder.\r\n\r\nCurrently, after removing a shard, the IndicesStores checks the indices services if that has no more shard active for this index and if so, it tries to delete the entire index folder (unless on master node, where we keep the index metadata around). This is wrong as both the check and the protections in IndicesServices.deleteIndexStore make sure that there isn't any shard *in use* from that index. However, it may be the we erroneously delete other unused shard copies on disk, without the proper safety guards described above.\r\n\r\n Normally, this is not a problem as the missing copy will be recovered from another shard copy on another node (although a shame). However, in extremely rare cases involving multiple node failures/restarts where all shard copies are not available (i.e., shard is red) there are race conditions which can cause all shard copies to be deleted.\r\n\r\n Instead, we should change the decision to clean up an index folder to be based on checking the index directory for being empty and containing no shards.\r\n\r\nNote: this PR is against the 1.6 branch."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/125190762","html_url":"https://github.com/elastic/elasticsearch/pull/12463#issuecomment-125190762","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/12463","id":125190762,"user":{"login":"martijnvg","id":580421,"avatar_url":"https://avatars.githubusercontent.com/u/580421?v=3","gravatar_id":"","url":"https://api.github.com/users/martijnvg","html_url":"https://github.com/martijnvg","followers_url":"https://api.github.com/users/martijnvg/followers","following_url":"https://api.github.com/users/martijnvg/following{/other_user}","gists_url":"https://api.github.com/users/martijnvg/gists{/gist_id}","starred_url":"https://api.github.com/users/martijnvg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martijnvg/subscriptions","organizations_url":"https://api.github.com/users/martijnvg/orgs","repos_url":"https://api.github.com/users/martijnvg/repos","events_url":"https://api.github.com/users/martijnvg/events{/privacy}","received_events_url":"https://api.github.com/users/martijnvg/received_events","type":"User","site_admin":false},"created_at":"2015-07-27T12:35:26Z","updated_at":"2015-07-27T12:35:26Z","body":"@bleskes This bug is sneaky. LGTM. Left a question about deleteIndexStore() itself, but that shouldn't block this PR."}}