{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/events","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9226","id":53927068,"number":9226,"title":"Shard stuck in relocating state with recovery stage=translog","user":{"login":"ppf2","id":7216393,"avatar_url":"https://avatars.githubusercontent.com/u/7216393?v=3","gravatar_id":"","url":"https://api.github.com/users/ppf2","html_url":"https://github.com/ppf2","followers_url":"https://api.github.com/users/ppf2/followers","following_url":"https://api.github.com/users/ppf2/following{/other_user}","gists_url":"https://api.github.com/users/ppf2/gists{/gist_id}","starred_url":"https://api.github.com/users/ppf2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppf2/subscriptions","organizations_url":"https://api.github.com/users/ppf2/orgs","repos_url":"https://api.github.com/users/ppf2/repos","events_url":"https://api.github.com/users/ppf2/events{/privacy}","received_events_url":"https://api.github.com/users/ppf2/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"milestone":null,"comments":42,"created_at":"2015-01-09T23:10:42Z","updated_at":"2015-02-17T19:34:03Z","closed_at":null,"body":"A primary shard was stuck in RELOCATION state.  Recovery API shows that all files have completed at 100%.  The stage shows that it is TRANSLOG.  But it has been sitting there for > 15 hours.\r\n\r\n```json\r\n  {\r\n        \"id\": 0,\r\n        \"type\": \"RELOCATION\",\r\n        \"stage\": \"TRANSLOG\",\r\n        \"primary\": true,\r\n        \"start_time\": \"16443.3d\",\r\n        \"start_time_in_millis\": 1420704589802,\r\n        \"stop_time\": \"0s\",\r\n        \"stop_time_in_millis\": 0,\r\n        \"total_time\": \"15.1h\",\r\n        \"total_time_in_millis\": 54706836,\r\n        \"source\": {\r\n          \"id\": \"0eltfic0RdmC9v-Yb17Wzw\",\r\n          \"host\": \"elasticsearch1.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch1.localdomain\"\r\n        },\r\n        \"target\": {\r\n          \"id\": \"ctCRm_huQsSBoTobhmqJdg\",\r\n          \"host\": \"elasticsearch8.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch8.localdomain\"\r\n        },\r\n        \"index\": {\r\n          \"files\": {\r\n            \"total\": 318,\r\n            \"reused\": 0,\r\n            \"recovered\": 318,\r\n            \"percent\": \"100.0%\",\r\n            \"details\": [\r\n             ... list of files\r\n            ]\r\n          },\r\n          \"bytes\": {\r\n            \"total\": 58111147927,\r\n            \"reused\": 0,\r\n            \"recovered\": 58111147927,\r\n            \"percent\": \"100.0%\"\r\n          },\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"translog\": {\r\n          \"recovered\": 24685532,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"start\": {\r\n          \"check_index_time\": \"0s\",\r\n          \"check_index_time_in_millis\": 0,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        }\r\n      }\r\n\r\n```\r\n\r\nRunning a reroute command with no post body got it to unstuck:\r\n\r\n```\r\ncurl -XPOST 'localhost:9200/_cluster/reroute?pretty&explain=true'\r\n```\r\n\r\nNot sure how it got into this state (no log entries related to recovery of this shard in the data or master logs) and why we have to run an empty reroute request to get it unstuck."},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/comments/74735987","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9226#issuecomment-74735987","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226","id":74735987,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2015-02-17T19:34:02Z","updated_at":"2015-02-17T19:34:02Z","body":"@drax68 thank you! how long did it take for the cluster to stabilze?"}}