{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/9226","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9226/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9226/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9226/events","html_url":"https://github.com/elastic/elasticsearch/issues/9226","id":53927068,"number":9226,"title":"Shard stuck in relocating state with recovery stage=translog","user":{"login":"ppf2","id":7216393,"avatar_url":"https://avatars.githubusercontent.com/u/7216393?v=3","gravatar_id":"","url":"https://api.github.com/users/ppf2","html_url":"https://github.com/ppf2","followers_url":"https://api.github.com/users/ppf2/followers","following_url":"https://api.github.com/users/ppf2/following{/other_user}","gists_url":"https://api.github.com/users/ppf2/gists{/gist_id}","starred_url":"https://api.github.com/users/ppf2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppf2/subscriptions","organizations_url":"https://api.github.com/users/ppf2/orgs","repos_url":"https://api.github.com/users/ppf2/repos","events_url":"https://api.github.com/users/ppf2/events{/privacy}","received_events_url":"https://api.github.com/users/ppf2/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v1.5.1","name":"v1.5.1","color":"dddddd"}],"state":"open","locked":false,"assignee":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"milestone":null,"comments":143,"created_at":"2015-01-09T23:10:42Z","updated_at":"2015-04-09T06:55:21Z","closed_at":null,"body":"A primary shard was stuck in RELOCATION state.  Recovery API shows that all files have completed at 100%.  The stage shows that it is TRANSLOG.  But it has been sitting there for > 15 hours.\r\n\r\n```json\r\n  {\r\n        \"id\": 0,\r\n        \"type\": \"RELOCATION\",\r\n        \"stage\": \"TRANSLOG\",\r\n        \"primary\": true,\r\n        \"start_time\": \"16443.3d\",\r\n        \"start_time_in_millis\": 1420704589802,\r\n        \"stop_time\": \"0s\",\r\n        \"stop_time_in_millis\": 0,\r\n        \"total_time\": \"15.1h\",\r\n        \"total_time_in_millis\": 54706836,\r\n        \"source\": {\r\n          \"id\": \"0eltfic0RdmC9v-Yb17Wzw\",\r\n          \"host\": \"elasticsearch1.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch1.localdomain\"\r\n        },\r\n        \"target\": {\r\n          \"id\": \"ctCRm_huQsSBoTobhmqJdg\",\r\n          \"host\": \"elasticsearch8.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch8.localdomain\"\r\n        },\r\n        \"index\": {\r\n          \"files\": {\r\n            \"total\": 318,\r\n            \"reused\": 0,\r\n            \"recovered\": 318,\r\n            \"percent\": \"100.0%\",\r\n            \"details\": [\r\n             ... list of files\r\n            ]\r\n          },\r\n          \"bytes\": {\r\n            \"total\": 58111147927,\r\n            \"reused\": 0,\r\n            \"recovered\": 58111147927,\r\n            \"percent\": \"100.0%\"\r\n          },\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"translog\": {\r\n          \"recovered\": 24685532,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"start\": {\r\n          \"check_index_time\": \"0s\",\r\n          \"check_index_time_in_millis\": 0,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        }\r\n      }\r\n\r\n```\r\n\r\nRunning a reroute command with no post body got it to unstuck:\r\n\r\n```\r\ncurl -XPOST 'localhost:9200/_cluster/reroute?pretty&explain=true'\r\n```\r\n\r\nNot sure how it got into this state (no log entries related to recovery of this shard in the data or master logs) and why we have to run an empty reroute request to get it unstuck."},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/91130460","html_url":"https://github.com/elastic/elasticsearch/issues/9226#issuecomment-91130460","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9226","id":91130460,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2015-04-09T06:55:21Z","updated_at":"2015-04-09T06:55:21Z","body":"That's great news. Thank you for pointing us at the segment count (I.e., many small files). Let's wait until 1.5.1 is out (soon) and see how works for the others\n\nOn Thu, Apr 9, 2015 at 8:45 AM, sylvae <notifications@github.com> wrote:\n\n> @mikemccand @bleskes I modified the code similar to #10463  and did some tests, the issue is no longer reproduce, shards recovered normally . I think this issue has already been fixed.Thank you.\n> ---\n> Reply to this email directly or view it on GitHub:\n> https://github.com/elastic/elasticsearch/issues/9226#issuecomment-91128850"}}