{"action":"created","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226/events","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9226","id":53927068,"number":9226,"title":"Shard stuck in relocating state with recovery stage=translog","user":{"login":"ppf2","id":7216393,"avatar_url":"https://avatars.githubusercontent.com/u/7216393?v=3","gravatar_id":"","url":"https://api.github.com/users/ppf2","html_url":"https://github.com/ppf2","followers_url":"https://api.github.com/users/ppf2/followers","following_url":"https://api.github.com/users/ppf2/following{/other_user}","gists_url":"https://api.github.com/users/ppf2/gists{/gist_id}","starred_url":"https://api.github.com/users/ppf2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ppf2/subscriptions","organizations_url":"https://api.github.com/users/ppf2/orgs","repos_url":"https://api.github.com/users/ppf2/repos","events_url":"https://api.github.com/users/ppf2/events{/privacy}","received_events_url":"https://api.github.com/users/ppf2/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/labels/:Recovery","name":":Recovery","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"milestone":null,"comments":64,"created_at":"2015-01-09T23:10:42Z","updated_at":"2015-03-09T06:03:11Z","closed_at":null,"body":"A primary shard was stuck in RELOCATION state.  Recovery API shows that all files have completed at 100%.  The stage shows that it is TRANSLOG.  But it has been sitting there for > 15 hours.\r\n\r\n```json\r\n  {\r\n        \"id\": 0,\r\n        \"type\": \"RELOCATION\",\r\n        \"stage\": \"TRANSLOG\",\r\n        \"primary\": true,\r\n        \"start_time\": \"16443.3d\",\r\n        \"start_time_in_millis\": 1420704589802,\r\n        \"stop_time\": \"0s\",\r\n        \"stop_time_in_millis\": 0,\r\n        \"total_time\": \"15.1h\",\r\n        \"total_time_in_millis\": 54706836,\r\n        \"source\": {\r\n          \"id\": \"0eltfic0RdmC9v-Yb17Wzw\",\r\n          \"host\": \"elasticsearch1.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch1.localdomain\"\r\n        },\r\n        \"target\": {\r\n          \"id\": \"ctCRm_huQsSBoTobhmqJdg\",\r\n          \"host\": \"elasticsearch8.localdomain\",\r\n          \"transport_address\": \"inet[/IP:9300]\",\r\n          \"ip\": \"127.0.1.1\",\r\n          \"name\": \"elasticsearch8.localdomain\"\r\n        },\r\n        \"index\": {\r\n          \"files\": {\r\n            \"total\": 318,\r\n            \"reused\": 0,\r\n            \"recovered\": 318,\r\n            \"percent\": \"100.0%\",\r\n            \"details\": [\r\n             ... list of files\r\n            ]\r\n          },\r\n          \"bytes\": {\r\n            \"total\": 58111147927,\r\n            \"reused\": 0,\r\n            \"recovered\": 58111147927,\r\n            \"percent\": \"100.0%\"\r\n          },\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"translog\": {\r\n          \"recovered\": 24685532,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        },\r\n        \"start\": {\r\n          \"check_index_time\": \"0s\",\r\n          \"check_index_time_in_millis\": 0,\r\n          \"total_time\": \"0s\",\r\n          \"total_time_in_millis\": 0\r\n        }\r\n      }\r\n\r\n```\r\n\r\nRunning a reroute command with no post body got it to unstuck:\r\n\r\n```\r\ncurl -XPOST 'localhost:9200/_cluster/reroute?pretty&explain=true'\r\n```\r\n\r\nNot sure how it got into this state (no log entries related to recovery of this shard in the data or master logs) and why we have to run an empty reroute request to get it unstuck."},"comment":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/comments/77804545","html_url":"https://github.com/elasticsearch/elasticsearch/issues/9226#issuecomment-77804545","issue_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/9226","id":77804545,"user":{"login":"rtoma","id":2914051,"avatar_url":"https://avatars.githubusercontent.com/u/2914051?v=3","gravatar_id":"","url":"https://api.github.com/users/rtoma","html_url":"https://github.com/rtoma","followers_url":"https://api.github.com/users/rtoma/followers","following_url":"https://api.github.com/users/rtoma/following{/other_user}","gists_url":"https://api.github.com/users/rtoma/gists{/gist_id}","starred_url":"https://api.github.com/users/rtoma/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtoma/subscriptions","organizations_url":"https://api.github.com/users/rtoma/orgs","repos_url":"https://api.github.com/users/rtoma/repos","events_url":"https://api.github.com/users/rtoma/events{/privacy}","received_events_url":"https://api.github.com/users/rtoma/received_events","type":"User","site_admin":false},"created_at":"2015-03-09T06:03:11Z","updated_at":"2015-03-09T06:03:11Z","body":"I thought ES recommands Java 7u55 and higher because of data corruption bug in lower. You are using  u44. Maybe related?\n\nOp 9 mrt. 2015 om 06:46 heeft kadaan <notifications@github.com<mailto:notifications@github.com>> het volgende geschreven:\n\n\n@bleskes<https://github.com/bleskes> I fixed the link to the hot_threads result.\n\n-\nReply to this email directly or view it on GitHub<https://github.com/elasticsearch/elasticsearch/issues/9226#issuecomment-77803339>.\n"}}