{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14149","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14149/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14149/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14149/events","html_url":"https://github.com/elastic/elasticsearch/issues/14149","id":111729647,"number":14149,"title":"Unassigned shards during cluster start since upgrading to 1.7.1","user":{"login":"davidvgalbraith","id":2757844,"avatar_url":"https://avatars.githubusercontent.com/u/2757844?v=3","gravatar_id":"","url":"https://api.github.com/users/davidvgalbraith","html_url":"https://github.com/davidvgalbraith","followers_url":"https://api.github.com/users/davidvgalbraith/followers","following_url":"https://api.github.com/users/davidvgalbraith/following{/other_user}","gists_url":"https://api.github.com/users/davidvgalbraith/gists{/gist_id}","starred_url":"https://api.github.com/users/davidvgalbraith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidvgalbraith/subscriptions","organizations_url":"https://api.github.com/users/davidvgalbraith/orgs","repos_url":"https://api.github.com/users/davidvgalbraith/repos","events_url":"https://api.github.com/users/davidvgalbraith/events{/privacy}","received_events_url":"https://api.github.com/users/davidvgalbraith/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-15T23:45:47Z","updated_at":"2015-10-16T18:37:27Z","closed_at":"2015-10-16T08:57:46Z","body":"Hey! I have a two-node Elasticsearch cluster that I upgraded to version 1.7.1 from 1.5.2. All my indexes have 2 shards and no replicas. Whenever I start Elasticsearch now, I get red cluster health with unassigned shards:\r\n```\r\n{\r\n  \"cluster_name\" : \"campfire.production.local\",\r\n  \"status\" : \"red\",\r\n  \"timed_out\" : false,\r\n  \"number_of_nodes\" : 2,\r\n  \"number_of_data_nodes\" : 2,\r\n  \"active_primary_shards\" : 214,\r\n  \"active_shards\" : 214,\r\n  \"relocating_shards\" : 0,\r\n  \"initializing_shards\" : 0,\r\n  \"unassigned_shards\" : 18,\r\n  \"delayed_unassigned_shards\" : 0,\r\n  \"number_of_pending_tasks\" : 0,\r\n  \"number_of_in_flight_fetch\" : 0\r\n}\r\n```\r\nI tried to use the reroute API to assign them or figure out what's wrong, and this happened:\r\n```\r\ncurl -XPOST -d '{ \"commands\" : [ { \"allocate\" : { \"index\" : \"metadata-jut_internal@16653\", \"shard\" : 1, \"node\" : \"FYEkARYjQm2Bp41qVoxSUg\" } } ] }' http://localhost:9200/_cluster/reroute?pretty\r\n{\r\n  \"error\" : \"ElasticsearchIllegalArgumentException[[allocate] trying to allocate a primary shard [metadata-jut_internal@16653][1], which is disabled]\",\r\n  \"status\" : 400\r\n}\r\n```\r\nI tried adding `\"allow_primary\": true` to the reassign command, and it assigned the shard, but when I tried to query the data from that shard I found that it had been deleted. That wasn't good. After reading https://github.com/elastic/elasticsearch/issues/14001, I reran the command without `allow_primary` but with `explain`, and here's a portion of the response for one of my indexes whose shards are unassigned:\r\n```\r\n        \"events-jut_internal@2015.07.31\" : {\r\n          \"shards\" : {\r\n            \"0\" : [ {\r\n              \"state\" : \"UNASSIGNED\",\r\n              \"primary\" : true,\r\n              \"node\" : null,\r\n              \"relocating_node\" : null,\r\n              \"shard\" : 0,\r\n              \"index\" : \"events-jut_internal@2015.07.31\",\r\n              \"unassigned_info\" : {\r\n                \"reason\" : \"CLUSTER_RECOVERED\",\r\n                \"at\" : \"2015-10-15T23:07:55.600Z\"\r\n              }\r\n            } ],\r\n            \"1\" : [ {\r\n              \"state\" : \"UNASSIGNED\",\r\n              \"primary\" : true,\r\n              \"node\" : null,\r\n              \"relocating_node\" : null,\r\n              \"shard\" : 1,\r\n              \"index\" : \"events-jut_internal@2015.07.31\",\r\n              \"unassigned_info\" : {\r\n                \"reason\" : \"CLUSTER_RECOVERED\",\r\n                \"at\" : \"2015-10-15T23:07:55.600Z\"\r\n              }\r\n            } ]\r\n```\r\nHow am I to interpret that `CLUSTER_RECOVERED` reason? How can I get my cluster back to green, preferably without losing any more data? Thanks!"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/148799777","html_url":"https://github.com/elastic/elasticsearch/issues/14149#issuecomment-148799777","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14149","id":148799777,"user":{"login":"davidvgalbraith","id":2757844,"avatar_url":"https://avatars.githubusercontent.com/u/2757844?v=3","gravatar_id":"","url":"https://api.github.com/users/davidvgalbraith","html_url":"https://github.com/davidvgalbraith","followers_url":"https://api.github.com/users/davidvgalbraith/followers","following_url":"https://api.github.com/users/davidvgalbraith/following{/other_user}","gists_url":"https://api.github.com/users/davidvgalbraith/gists{/gist_id}","starred_url":"https://api.github.com/users/davidvgalbraith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidvgalbraith/subscriptions","organizations_url":"https://api.github.com/users/davidvgalbraith/orgs","repos_url":"https://api.github.com/users/davidvgalbraith/repos","events_url":"https://api.github.com/users/davidvgalbraith/events{/privacy}","received_events_url":"https://api.github.com/users/davidvgalbraith/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T18:37:27Z","updated_at":"2015-10-16T18:37:27Z","body":"That's a brave little theory, but it doesn't seem to be accurate. I did\r\n```\r\nroot@staging-llnw-de0:~$ curl -XPUT 'localhost:9200/_cluster/settings' -d '{\r\n  \"persistent\": {\r\n    \"cluster.routing.allocation.enable\": \"all\"\r\n  }\r\n}'\r\nroot@staging-llnw-de0:~$ curl localhost:9200/_cluster/settings\r\n{\"persistent\":{\"cluster\":{\"routing\":{\"allocation\":{\"enable\":\"all\"}}}},\"transient\":{}}\r\nroot@staging-llnw-de0:/opt/jut# curl localhost:9200/_cluster/health?pretty\r\n{\r\n  \"cluster_name\" : \"campfire.production.local\",\r\n  \"status\" : \"red\",\r\n  \"timed_out\" : false,\r\n  \"number_of_nodes\" : 2,\r\n  \"number_of_data_nodes\" : 2,\r\n  \"active_primary_shards\" : 214,\r\n  \"active_shards\" : 214,\r\n  \"relocating_shards\" : 0,\r\n  \"initializing_shards\" : 0,\r\n  \"unassigned_shards\" : 18,\r\n  \"delayed_unassigned_shards\" : 0,\r\n  \"number_of_pending_tasks\" : 0,\r\n  \"number_of_in_flight_fetch\" : 0\r\n}\r\n```\r\nI then restarted Elasticsearch on both nodes, and the health stayed the same. Did I run the wrong command to enable allocation? Is there another possible cause of these unassigned shards?"}}