{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/9421","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9421/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9421/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9421/events","html_url":"https://github.com/elastic/elasticsearch/issues/9421","id":55503103,"number":9421,"title":"[CI Failure] SimpleSortTests.testIssue8226","user":{"login":"brwe","id":4320215,"avatar_url":"https://avatars.githubusercontent.com/u/4320215?v=3","gravatar_id":"","url":"https://api.github.com/users/brwe","html_url":"https://github.com/brwe","followers_url":"https://api.github.com/users/brwe/followers","following_url":"https://api.github.com/users/brwe/following{/other_user}","gists_url":"https://api.github.com/users/brwe/gists{/gist_id}","starred_url":"https://api.github.com/users/brwe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brwe/subscriptions","organizations_url":"https://api.github.com/users/brwe/orgs","repos_url":"https://api.github.com/users/brwe/repos","events_url":"https://api.github.com/users/brwe/events{/privacy}","received_events_url":"https://api.github.com/users/brwe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/bug","name":"bug","color":"9e2c2c"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/jenkins","name":"jenkins","color":"207de5"}],"state":"open","locked":false,"assignee":{"login":"brwe","id":4320215,"avatar_url":"https://avatars.githubusercontent.com/u/4320215?v=3","gravatar_id":"","url":"https://api.github.com/users/brwe","html_url":"https://github.com/brwe","followers_url":"https://api.github.com/users/brwe/followers","following_url":"https://api.github.com/users/brwe/following{/other_user}","gists_url":"https://api.github.com/users/brwe/gists{/gist_id}","starred_url":"https://api.github.com/users/brwe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brwe/subscriptions","organizations_url":"https://api.github.com/users/brwe/orgs","repos_url":"https://api.github.com/users/brwe/repos","events_url":"https://api.github.com/users/brwe/events{/privacy}","received_events_url":"https://api.github.com/users/brwe/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-01-26T16:27:23Z","updated_at":"2015-03-20T21:11:21Z","closed_at":null,"body":"Fails about once a week. Example failure:\r\nhttp://build-us-00.elasticsearch.org/job/es_g1gc_1x_metal/3129/\r\n\r\nI can reproduce it locally (although very rarely) with some additional logging (action.search.type: TRACE). \r\n\r\nHere is a brief analysis of what happened. Would be great if someone could take a look and let me know if this makes sense.\r\n\r\nFailure:\r\n\r\n```\r\n1> REPRODUCE WITH  : mvn clean test -Dtests.seed=774A2866F1B6042D -Dtests.class=org.elasticsearch.search.sort.SimpleSortTests -Dtests.method=\"testIssue8226 {#76 seed=[774A2866F1B6042D:ACB4FF9F8C8CA341]}\" -Des.logger.level=DEBUG -Des.node.mode=network -Dtests.security.manager=true -Dtests.nightly=false -Dtests.client.ratio=0.0 -Dtests.heap.size=512m -Dtests.jvm.argline=\"-server -XX:+UseConcMarkSweepGC -XX:-UseCompressedOops -XX:+AggressiveOpts -Djava.net.preferIPv4Stack=true\" -Dtests.locale=fi_FI -Dtests.timezone=Etc/GMT+9 -Dtests.processors=4\r\n  1> Throwable:\r\n  1> java.lang.AssertionError: One or more shards were not successful but didn't trigger a failure\r\n  1> Expected: <47>\r\n  1>      but: was <46>\r\n```\r\n\r\nHere is an example failure in detail, the relevant parts of the logs are below:\r\n\r\nState\r\n-----\r\n\r\nnode_0 is master.\r\n[test_5][0] is relocating from node_1 to node_0.\r\nCluster state 3673 has the shard as relocating, in cluster state 3674 it is started.\r\nnode_0 is the coordinating node for the search request.\r\n\r\nIn brief, the request fails for shard [test_5][0] because node_0 operates on an older cluster state 3673 when processing the search request, while node_1 is already on 3674.\r\n\r\n\r\nCourse of events:\r\n----\r\n\r\n1. node_0 sends shard started, but the shard is still in state POST_RECOVERY and will remain so until it receives the new cluster state and applies it locally\r\n2. node_0(master) receives the shard started request and publishes the new cluster state 3674 to node_0 and node_1\r\n3. node_1 receives the cluster state 3674 and applies it locally\r\n4. node_0 sends search request for [test_5][0] to node_1 because according to cluster state 3673 the shard is there and relocating\r\n -> request fails with IndexShardMissingException because node_1 already applied cluster state 3674 and deleted the shard.\r\n5. node_0 then sends request for [test_5][0] to node_0 because the shard is there as well (according to cluster state 3673 it is and initializing)\r\n -> request fails with IllegalIndexShardStateException because node_0 has not yet processed cluster state 3674 and therefore the shard is in POST_RECOVERY instead of STARTED\r\nNo shard failure is logged because IndexShardMissingException and IllegalIndexShardStateException are explicitly excluded from shard failures.\r\n6. node_0 finally also gets to process the new cluster state and moves the shard [test_5][0] to STARTED but it is too late\r\n\r\n\r\nThis is a very rare condition and maybe too bad on client side because the information that one shard did not deliver results is there although it is not explicitly listed as shard failure. We can probably make the test pass easily be just waiting for relocations before executing the search request but that seems wrong because any search request can fail this way.\r\n\r\n\r\nSample log\r\n----\r\n\r\n```\r\n[....]\r\n\r\n  1> [2015-01-26 09:27:14,435][DEBUG][indices.recovery         ] [node_0] [test_5][0] recovery completed from [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}], took [84ms]\r\n  1> [2015-01-26 09:27:14,435][DEBUG][cluster.action.shard     ] [node_0] sending shard started for [test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING], indexUUID [E3T8J7CaRkyK533W0hMBPw], reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]\r\n  1> [2015-01-26 09:27:14,435][DEBUG][cluster.action.shard     ] [node_0] received shard started for [test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING], indexUUID [E3T8J7CaRkyK533W0hMBPw], reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]\r\n  1> [2015-01-26 09:27:14,436][DEBUG][cluster.service          ] [node_0] processing [shard-started ([test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]]: execute\r\n  1> [2015-01-26 09:27:14,436][DEBUG][cluster.action.shard     ] [node_0] [test_5][0] will apply shard started [test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING], indexUUID [E3T8J7CaRkyK533W0hMBPw], reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]\r\n\r\n\r\n[....]\r\n\r\n\r\n  1> [2015-01-26 09:27:14,441][DEBUG][cluster.service          ] [node_0] cluster state updated, version [3674], source [shard-started ([test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]]\r\n  1> [2015-01-26 09:27:14,441][DEBUG][cluster.service          ] [node_0] publishing cluster state version 3674\r\n  1> [2015-01-26 09:27:14,442][DEBUG][discovery.zen.publish    ] [node_1] received cluster state version 3674\r\n  1> [2015-01-26 09:27:14,443][DEBUG][cluster.service          ] [node_1] processing [zen-disco-receive(from master [[node_0][GQ6yYxmyRT-sfvT0cmuqQQ][schmusi][inet[/192.168.2.102:9400]]{mode=network, enable_custom_paths=true}])]: execute\r\n  1> [2015-01-26 09:27:14,443][DEBUG][cluster.service          ] [node_1] cluster state updated, version [3674], source [zen-disco-receive(from master [[node_0][GQ6yYxmyRT-sfvT0cmuqQQ][schmusi][inet[/192.168.2.102:9400]]{mode=network, enable_custom_paths=true}])]\r\n  1> [2015-01-26 09:27:14,443][DEBUG][cluster.service          ] [node_1] set local cluster state to version 3674\r\n  1> [2015-01-26 09:27:14,443][DEBUG][indices.cluster          ] [node_1] [test_5][0] removing shard (not allocated)\r\n  1> [2015-01-26 09:27:14,443][DEBUG][index                    ] [node_1] [test_5] [0] closing... (reason: [removing shard (not allocated)])\r\n  1> [2015-01-26 09:27:14,443][INFO ][test.store               ] [node_1] [test_5][0] Shard state before potentially flushing is STARTED\r\n  1> [2015-01-26 09:27:14,453][DEBUG][search.sort              ] cluster state:\r\n  1> version: 3673\r\n  1> meta data version: 2043\r\n  1> nodes:\r\n  1>    [node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}\r\n  1>    [node_0][GQ6yYxmyRT-sfvT0cmuqQQ][schmusi][inet[/192.168.2.102:9400]]{mode=network, enable_custom_paths=true}, local, master\r\n  1> routing_table (version 3006):\r\n  1> -- index [test_4]\r\n  1> ----shard_id [test_4][4]\r\n  1> --------[test_4][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][7]\r\n  1> --------[test_4][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][7], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][0]\r\n  1> --------[test_4][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][3]\r\n  1> --------[test_4][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][1]\r\n  1> --------[test_4][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][5]\r\n  1> --------[test_4][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][6]\r\n  1> --------[test_4][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][6], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][2]\r\n  1> --------[test_4][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_3]\r\n  1> ----shard_id [test_3][2]\r\n  1> --------[test_3][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_3][0]\r\n  1> --------[test_3][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][3]\r\n  1> --------[test_3][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][1]\r\n  1> --------[test_3][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][5]\r\n  1> --------[test_3][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][6]\r\n  1> --------[test_3][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_3][4]\r\n  1> --------[test_3][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_2]\r\n  1> ----shard_id [test_2][2]\r\n  1> --------[test_2][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_2][0]\r\n  1> --------[test_2][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_2][3]\r\n  1> --------[test_2][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][1]\r\n  1> --------[test_2][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][1], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][5]\r\n  1> --------[test_2][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][4]\r\n  1> --------[test_2][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_1]\r\n  1> ----shard_id [test_1][0]\r\n  1> --------[test_1][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_1][1]\r\n  1> --------[test_1][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_0]\r\n  1> ----shard_id [test_0][4]\r\n  1> --------[test_0][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][0]\r\n  1> --------[test_0][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][0], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][7]\r\n  1> --------[test_0][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][7], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][3]\r\n  1> --------[test_0][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][1]\r\n  1> --------[test_0][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][5]\r\n  1> --------[test_0][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][6]\r\n  1> --------[test_0][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][2]\r\n  1> --------[test_0][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_8]\r\n  1> ----shard_id [test_8][0]\r\n  1> --------[test_8][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_8][1]\r\n  1> --------[test_8][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_8][2]\r\n  1> --------[test_8][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_7]\r\n  1> ----shard_id [test_7][2]\r\n  1> --------[test_7][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_7][0]\r\n  1> --------[test_7][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_7][3]\r\n  1> --------[test_7][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_7][1]\r\n  1> --------[test_7][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_7][4]\r\n  1> --------[test_7][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_6]\r\n  1> ----shard_id [test_6][0]\r\n  1> --------[test_6][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][3]\r\n  1> --------[test_6][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][1]\r\n  1> --------[test_6][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][2]\r\n  1> --------[test_6][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_6][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_5]\r\n  1> ----shard_id [test_5][0]\r\n  1> --------[test_5][0], node[G4AEDzbrRae5BC_UD9zItA], relocating [GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[RELOCATING]\r\n  1> ----shard_id [test_5][3]\r\n  1> --------[test_5][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_5][1]\r\n  1> --------[test_5][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_5][2]\r\n  1> --------[test_5][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1>\r\n  1> routing_nodes:\r\n  1> -----node_id[GQ6yYxmyRT-sfvT0cmuqQQ][V]\r\n  1> --------[test_4][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_1][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_8][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_8][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_7][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_7][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_6][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]\r\n  1> --------[test_5][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> -----node_id[G4AEDzbrRae5BC_UD9zItA][V]\r\n  1> --------[test_4][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][7], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][6], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_3][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_3][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_2][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_2][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][1], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_1][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][0], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][7], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_8][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_5][0], node[G4AEDzbrRae5BC_UD9zItA], relocating [GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[RELOCATING]\r\n  1> --------[test_5][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_5][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ---- unassigned\r\n  1>\r\n  1> tasks: (1):\r\n  1> 13638/URGENT/shard-started ([test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]/13ms\r\n  1>\r\n\r\n\r\n\r\n[...]\r\n\r\n\r\n 1> [2015-01-26 09:27:14,459][TRACE][action.search.type       ] [node_0] got first-phase result from [G4AEDzbrRae5BC_UD9zItA][test_3][3]\r\n  1> [2015-01-26 09:27:14,460][TRACE][action.search.type       ] [node_0] [test_5][0], node[G4AEDzbrRae5BC_UD9zItA], relocating [GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[RELOCATING]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1469040a] lastShard [false]\r\n  1> org.elasticsearch.transport.RemoteTransportException: [node_1][inet[/192.168.2.102:9401]][indices:data/read/search[phase/dfs]]\r\n  1> Caused by: org.elasticsearch.index.IndexShardMissingException: [test_5][0] missing\r\n  1>     at org.elasticsearch.index.IndexService.shardSafe(IndexService.java:203)\r\n  1>     at org.elasticsearch.search.SearchService.createContext(SearchService.java:539)\r\n  1>     at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:523)\r\n  1>     at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:208)\r\n  1>     at org.elasticsearch.search.action.SearchServiceTransportAction$SearchDfsTransportHandler.messageReceived(SearchServiceTransportAction.java:757)\r\n  1>     at org.elasticsearch.search.action.SearchServiceTransportAction$SearchDfsTransportHandler.messageReceived(SearchServiceTransportAction.java:748)\r\n  1>     at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:275)\r\n  1>     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)\r\n  1>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n  1>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n  1>     at java.lang.Thread.run(Thread.java:745)\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_6][1]\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_5][2]\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_7][1]\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_3][2]\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_8][2]\r\n  1> [2015-01-26 09:27:14,455][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_1][0]\r\n\r\n\r\n[...]\r\n\r\n\r\n  1> [2015-01-26 09:27:14,463][TRACE][action.search.type       ] [node_0] [test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]: Failed to execute [org.elasticsearch.action.search.SearchRequest@1469040a] lastShard [true]\r\n  1> org.elasticsearch.index.shard.IllegalIndexShardStateException: [test_5][0] CurrentState[POST_RECOVERY] operations only allowed when started/relocated\r\n  1>     at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:839)\r\n  1>     at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:651)\r\n  1>     at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:647)\r\n  1>     at org.elasticsearch.search.SearchService.createContext(SearchService.java:543)\r\n  1>     at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:523)\r\n  1>     at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:208)\r\n  1>     at org.elasticsearch.search.action.SearchServiceTransportAction$3.call(SearchServiceTransportAction.java:197)\r\n  1>     at org.elasticsearch.search.action.SearchServiceTransportAction$3.call(SearchServiceTransportAction.java:194)\r\n  1>     at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)\r\n  1>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n  1>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n  1>     at java.lang.Thread.run(Thread.java:745)\r\n  1> [2015-01-26 09:27:14,459][TRACE][action.search.type       ] [node_0] got first-phase result from [G4AEDzbrRae5BC_UD9zItA][test_2][4]\r\n\r\n\r\n\r\n[...]\r\n\r\n\r\n\r\n  1> [2015-01-26 09:27:14,493][DEBUG][cluster.service          ] [node_0] set local cluster state to version 3674\r\n  1> [2015-01-26 09:27:14,493][DEBUG][index.shard              ] [node_0] [test_5][0] state: [POST_RECOVERY]->[STARTED], reason [global state is [STARTED]]\r\n  1> [2015-01-26 09:27:14,493][DEBUG][river.cluster            ] [node_0] processing [reroute_rivers_node_changed]: execute\r\n  1> [2015-01-26 09:27:14,493][DEBUG][river.cluster            ] [node_0] processing [reroute_rivers_node_changed]: no change in cluster_state\r\n  1> [2015-01-26 09:27:14,493][DEBUG][cluster.service          ] [node_0] processing [shard-started ([test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], relocating [G4AEDzbrRae5BC_UD9zItA], [P], s[INITIALIZING]), reason [after recovery (replica) from node [[node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}]]]: done applying updated cluster_state (version: 3674)\r\n  1> [2015-01-26 09:27:14,456][TRACE][action.search.type       ] [node_0] got first-phase result from [GQ6yYxmyRT-sfvT0cmuqQQ][test_2][3]\r\n\r\n\r\n[...]\r\n\r\n  1> [2015-01-26 09:27:14,527][DEBUG][search.sort              ] cluster state:\r\n  1> version: 3674\r\n  1> meta data version: 2043\r\n  1> nodes:\r\n  1>    [node_1][G4AEDzbrRae5BC_UD9zItA][schmusi][inet[/192.168.2.102:9401]]{mode=network, enable_custom_paths=true}\r\n  1>    [node_0][GQ6yYxmyRT-sfvT0cmuqQQ][schmusi][inet[/192.168.2.102:9400]]{mode=network, enable_custom_paths=true}, local, master\r\n  1> routing_table (version 3007):\r\n  1> -- index [test_4]\r\n  1> ----shard_id [test_4][2]\r\n  1> --------[test_4][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][7]\r\n  1> --------[test_4][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][7], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][0]\r\n  1> --------[test_4][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][3]\r\n  1> --------[test_4][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][1]\r\n  1> --------[test_4][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][5]\r\n  1> --------[test_4][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_4][6]\r\n  1> --------[test_4][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][6], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_4][4]\r\n  1> --------[test_4][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_3]\r\n  1> ----shard_id [test_3][4]\r\n  1> --------[test_3][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_3][0]\r\n  1> --------[test_3][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][3]\r\n  1> --------[test_3][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][1]\r\n  1> --------[test_3][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][5]\r\n  1> --------[test_3][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_3][6]\r\n  1> --------[test_3][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_3][2]\r\n  1> --------[test_3][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_2]\r\n  1> ----shard_id [test_2][4]\r\n  1> --------[test_2][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_2][0]\r\n  1> --------[test_2][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_2][3]\r\n  1> --------[test_2][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][1]\r\n  1> --------[test_2][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][1], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][5]\r\n  1> --------[test_2][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_2][2]\r\n  1> --------[test_2][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_1]\r\n  1> ----shard_id [test_1][0]\r\n  1> --------[test_1][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_1][1]\r\n  1> --------[test_1][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_0]\r\n  1> ----shard_id [test_0][2]\r\n  1> --------[test_0][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][0]\r\n  1> --------[test_0][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][0], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][7]\r\n  1> --------[test_0][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][7], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][3]\r\n  1> --------[test_0][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][1]\r\n  1> --------[test_0][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][5]\r\n  1> --------[test_0][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_0][6]\r\n  1> --------[test_0][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> ----shard_id [test_0][4]\r\n  1> --------[test_0][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_8]\r\n  1> ----shard_id [test_8][0]\r\n  1> --------[test_8][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_8][1]\r\n  1> --------[test_8][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_8][2]\r\n  1> --------[test_8][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_7]\r\n  1> ----shard_id [test_7][4]\r\n  1> --------[test_7][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_7][0]\r\n  1> --------[test_7][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_7][3]\r\n  1> --------[test_7][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_7][1]\r\n  1> --------[test_7][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_7][2]\r\n  1> --------[test_7][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1>\r\n  1> -- index [test_6]\r\n  1> ----shard_id [test_6][0]\r\n  1> --------[test_6][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][3]\r\n  1> --------[test_6][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][1]\r\n  1> --------[test_6][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_6][2]\r\n  1> --------[test_6][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_6][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1>\r\n  1> -- index [test_5]\r\n  1> ----shard_id [test_5][0]\r\n  1> --------[test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> ----shard_id [test_5][3]\r\n  1> --------[test_5][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_5][1]\r\n  1> --------[test_5][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ----shard_id [test_5][2]\r\n  1> --------[test_5][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1>\r\n  1> routing_nodes:\r\n  1> -----node_id[GQ6yYxmyRT-sfvT0cmuqQQ][V]\r\n  1> --------[test_4][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_4][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_4][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_3][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_3][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_2][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_2][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_1][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][7], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][5], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_0][6], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_0][4], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_8][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_8][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_7][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_7][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_6][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][3], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][1], node[GQ6yYxmyRT-sfvT0cmuqQQ], [R], s[STARTED]\r\n  1> --------[test_6][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_5][0], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> --------[test_5][2], node[GQ6yYxmyRT-sfvT0cmuqQQ], [P], s[STARTED]\r\n  1> -----node_id[G4AEDzbrRae5BC_UD9zItA][V]\r\n  1> --------[test_4][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][7], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_4][6], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_4][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_3][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_3][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_3][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_2][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_2][3], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][1], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][5], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_2][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_1][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][0], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][7], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][5], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_0][6], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_0][4], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_8][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][4], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_7][2], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][0], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_6][2], node[G4AEDzbrRae5BC_UD9zItA], [R], s[STARTED]\r\n  1> --------[test_5][3], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> --------[test_5][1], node[G4AEDzbrRae5BC_UD9zItA], [P], s[STARTED]\r\n  1> ---- unassigned\r\n  1>\r\n  1> tasks: (0):\r\n  1>\r\n\r\n[...]\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/84148217","html_url":"https://github.com/elastic/elasticsearch/issues/9421#issuecomment-84148217","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/9421","id":84148217,"user":{"login":"brwe","id":4320215,"avatar_url":"https://avatars.githubusercontent.com/u/4320215?v=3","gravatar_id":"","url":"https://api.github.com/users/brwe","html_url":"https://github.com/brwe","followers_url":"https://api.github.com/users/brwe/followers","following_url":"https://api.github.com/users/brwe/following{/other_user}","gists_url":"https://api.github.com/users/brwe/gists{/gist_id}","starred_url":"https://api.github.com/users/brwe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brwe/subscriptions","organizations_url":"https://api.github.com/users/brwe/orgs","repos_url":"https://api.github.com/users/brwe/repos","events_url":"https://api.github.com/users/brwe/events{/privacy}","received_events_url":"https://api.github.com/users/brwe/received_events","type":"User","site_admin":false},"created_at":"2015-03-20T21:11:20Z","updated_at":"2015-03-20T21:11:20Z","body":"@bleskes ping :)\r\nI finally came back to this and wrote a test that reproduces the failure reliably (#10194) but I did not quite get what you meant by \"capture the original issue\". Can you elaborate?"}}