{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/14494","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14494/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14494/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14494/events","html_url":"https://github.com/elastic/elasticsearch/pull/14494","id":114892589,"number":14494,"title":"Delayed allocation can miss a reroute","user":{"login":"ywelsch","id":3718355,"avatar_url":"https://avatars.githubusercontent.com/u/3718355?v=3","gravatar_id":"","url":"https://api.github.com/users/ywelsch","html_url":"https://github.com/ywelsch","followers_url":"https://api.github.com/users/ywelsch/followers","following_url":"https://api.github.com/users/ywelsch/following{/other_user}","gists_url":"https://api.github.com/users/ywelsch/gists{/gist_id}","starred_url":"https://api.github.com/users/ywelsch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ywelsch/subscriptions","organizations_url":"https://api.github.com/users/ywelsch/orgs","repos_url":"https://api.github.com/users/ywelsch/repos","events_url":"https://api.github.com/users/ywelsch/events{/privacy}","received_events_url":"https://api.github.com/users/ywelsch/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Allocation","name":":Allocation","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/review","name":"review","color":"0052cc"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.0.1","name":"v2.0.1","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.1.0","name":"v2.1.0","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v2.2.0","name":"v2.2.0","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v3.0.0","name":"v3.0.0","color":"dddddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-11-03T19:35:44Z","updated_at":"2015-11-09T19:33:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/14494","html_url":"https://github.com/elastic/elasticsearch/pull/14494","diff_url":"https://github.com/elastic/elasticsearch/pull/14494.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/14494.patch"},"body":"Closes #14445 \r\nRelates to  #14010 and #14011\r\n\r\nImplementation note:\r\n\r\nThe smallest test case I could come up with follows #14445 :\r\n- Create 4 data nodes (and 1 master node, to make the test simpler).\r\n- Create 2 indices with delayed shard allocation set to 10 seconds. Each index has 2 shards (1 primary and 1 replica). Balancer ensures that each data node gets a shard, either a primary or a replica shard.\r\n- Shutdown one node that has a replica shard of first index.\r\n- Delayed allocation kicks in (a task is scheduled to reroute in 10 seconds).\r\n- Wait a second\r\n- Shutdown second node that has a replica shard of second index (no new task is scheduled to reroute in 10 seconds).\r\n- Make replica shard of first index obsolete by setting number_of_replicas to 0 (the task that was scheduled to reroute is still being scheduled).\r\n- Wait until 3 shards (1 for first index and 2 for second index) are available. Wait at least 30 minutes to give a chance to delayed allocation of replica shard of second index to kick in.\r\n\r\n\r\nSame as in #14010, the issue can be \"fixed\" by waiting until delayed routing of first shard has kicked in and manually calling reroute. In the given test this works by inserting the following lines before the last line in the test:\r\n\r\n```\r\nlogger.info(\"--> waiting after delayed shard assignment should kick in for test1\");\r\nThread.sleep(15000);\r\nlogger.info(\"--> reroute\");\r\nclient().admin().cluster().prepareReroute().execute().actionGet();\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/155167163","html_url":"https://github.com/elastic/elasticsearch/pull/14494#issuecomment-155167163","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/14494","id":155167163,"user":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"created_at":"2015-11-09T19:33:57Z","updated_at":"2015-11-09T19:33:57Z","body":"@ywelsch discussed this further and @ywelsch is exploring options for mocking time for the purpose of unit testing."}}