{"action":"closed","issue":{"url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/6507","labels_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/6507/labels{/name}","comments_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/6507/comments","events_url":"https://api.github.com/repos/elasticsearch/elasticsearch/issues/6507/events","html_url":"https://github.com/elasticsearch/elasticsearch/issues/6507","id":35785252,"number":6507,"title":"Retry if initial contact fails","user":{"login":"magnhaug","id":477436,"avatar_url":"https://avatars.githubusercontent.com/u/477436?v=3","gravatar_id":"","url":"https://api.github.com/users/magnhaug","html_url":"https://github.com/magnhaug","followers_url":"https://api.github.com/users/magnhaug/followers","following_url":"https://api.github.com/users/magnhaug/following{/other_user}","gists_url":"https://api.github.com/users/magnhaug/gists{/gist_id}","starred_url":"https://api.github.com/users/magnhaug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magnhaug/subscriptions","organizations_url":"https://api.github.com/users/magnhaug/orgs","repos_url":"https://api.github.com/users/magnhaug/repos","events_url":"https://api.github.com/users/magnhaug/events{/privacy}","received_events_url":"https://api.github.com/users/magnhaug/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"bleskes","id":1006375,"avatar_url":"https://avatars.githubusercontent.com/u/1006375?v=3","gravatar_id":"","url":"https://api.github.com/users/bleskes","html_url":"https://github.com/bleskes","followers_url":"https://api.github.com/users/bleskes/followers","following_url":"https://api.github.com/users/bleskes/following{/other_user}","gists_url":"https://api.github.com/users/bleskes/gists{/gist_id}","starred_url":"https://api.github.com/users/bleskes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bleskes/subscriptions","organizations_url":"https://api.github.com/users/bleskes/orgs","repos_url":"https://api.github.com/users/bleskes/repos","events_url":"https://api.github.com/users/bleskes/events{/privacy}","received_events_url":"https://api.github.com/users/bleskes/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2014-06-16T10:43:20Z","updated_at":"2015-01-26T08:53:24Z","closed_at":"2015-01-26T08:53:24Z","body":"We have a 5-node cluster. As part of a full application redeploy, we are restarting every node in sequence.\r\nHere the rebooted 2nd node (\"Moondark\", new node, logs attached) tries to reach the old 3rd node (\"Volstagg\", old master) right as it shuts down as part of the restart. The new 3rd node to replace Volstagg has not yet come up.\r\nMoondark then figures out that the master is unreachable, and *gives up*.\r\n\r\nShouldn't a new node trigger another round of pings and a master connect attempt, in the case that the old master dies during initial connect?\r\n\r\nLike this, the new node never joins the cluster. I let the node live for an hour just to confirm.\r\n\r\n```\r\n2014-06-16 10:18:21,114 INFO  node                                      - [Moondark] version[1.1.1], pid[3867], build[f1585f0/2014-04-16T14:27:12Z]\r\n2014-06-16 10:18:21,118 INFO  node                                      - [Moondark] initializing ...\r\n2014-06-16 10:18:21,125 INFO  plugins                                   - [Moondark] loaded [], sites [head, bigdesk, elasticsearch_hq, kibana]\r\n2014-06-16 10:18:23,008 DEBUG discovery.zen.ping.unicast                - [Moondark] using initial hosts [tsl0mag12:12500, tsl0mag14:12500, tsl0mag15:12500, tsl0mag16:12500, tsl0mag17:12500], with concurrent_connects [10]\r\n2014-06-16 10:18:23,022 DEBUG discovery.zen                             - [Moondark] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]\r\n2014-06-16 10:18:23,025 DEBUG discovery.zen.elect                       - [Moondark] using minimum_master_nodes [3]\r\n2014-06-16 10:18:23,026 DEBUG discovery.zen.fd                          - [Moondark] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]\r\n2014-06-16 10:18:23,045 DEBUG discovery.zen.fd                          - [Moondark] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]\r\n2014-06-16 10:18:24,003 INFO  node                                      - [Moondark] initialized\r\n2014-06-16 10:18:24,003 INFO  node                                      - [Moondark] starting ...\r\n2014-06-16 10:18:24,097 INFO  transport                                 - [Moondark] bound_address {inet[/0:0:0:0:0:0:0:0:12500]}, publish_address {inet[/151.187.99.214:12500]}\r\n2014-06-16 10:18:24,119 TRACE discovery                                 - [Moondark] waiting for 30s for the initial state to be set by the discovery\r\n2014-06-16 10:18:24,162 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connecting (light) to [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:24,163 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connecting (light) to [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:24,162 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connecting to [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]\r\n2014-06-16 10:18:24,167 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connecting (light) to [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:24,169 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connecting (light) to [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:24,222 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connected to [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:24,222 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connected to [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:24,222 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:24,222 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:24,224 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connected to [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:24,227 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:24,227 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connected to [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:24,229 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:24,228 TRACE discovery.zen.ping.unicast                - [Moondark] [1] connected to [#zen_unicast_2#][tsl0mag14.skead.no][inet[tsl0mag14/151.187.99.214:12500]]\r\n2014-06-16 10:18:24,230 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]\r\n2014-06-16 10:18:24,274 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1\r\ncG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Sushi][iCNA-wEbSXOSn71Gngr_Qg][tsl0mag16.skead.no][inet[/151.187.99.216:12500]]], master [[Volstagg][PKLH\r\n_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:24,275 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1\r\ncG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Shatterstar][IQoquIjSRjOpbMenBqAv3A][tsl0mag12.skead.no][inet[/151.187.99.212:12500]]], master [[Volstagg\r\n][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:24,274 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Venus Dee Milo][ZogIMgj1R5WedZVR7f2A_w][tsl0mag17.skead.no][inet[/151.187.99.217:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:24,274 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:24,280 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:25,664 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:25,665 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]\r\n2014-06-16 10:18:25,666 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:25,667 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:25,668 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:25,672 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Venus Dee Milo][ZogIMgj1R5WedZVR7f2A_w][tsl0mag17.skead.no][inet[/151.187.99.217:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:25,673 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Sushi][iCNA-wEbSXOSn71Gngr_Qg][tsl0mag16.skead.no][inet[/151.187.99.216:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:25,674 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Shatterstar][IQoquIjSRjOpbMenBqAv3A][tsl0mag12.skead.no][inet[/151.187.99.212:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:25,674 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:25,674 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,175 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:27,176 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]\r\n2014-06-16 10:18:27,177 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:27,178 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:27,178 TRACE discovery.zen.ping.unicast                - [Moondark] [1] sending to [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:27,180 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Shatterstar][IQoquIjSRjOpbMenBqAv3A][tsl0mag12.skead.no][inet[/151.187.99.212:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,180 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Sushi][iCNA-wEbSXOSn71Gngr_Qg][tsl0mag16.skead.no][inet[/151.187.99.216:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,182 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Venus Dee Milo][ZogIMgj1R5WedZVR7f2A_w][tsl0mag17.skead.no][inet[/151.187.99.217:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,182 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,184 TRACE discovery.zen.ping.unicast                - [Moondark] [1] received response from [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]: [ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}, ping_response{target [[Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]], master [null], cluster_name[IRIS-GP2-N-S]}]\r\n2014-06-16 10:18:27,185 TRACE discovery.zen.ping.unicast                - [Moondark] [1] disconnecting from [#zen_unicast_4#][tsl0mag14.skead.no][inet[tsl0mag16/151.187.99.216:12500]]\r\n2014-06-16 10:18:27,194 TRACE discovery.zen.ping.unicast                - [Moondark] [1] disconnecting from [#zen_unicast_5#][tsl0mag14.skead.no][inet[tsl0mag17/151.187.99.217:12500]]\r\n2014-06-16 10:18:27,195 TRACE discovery.zen.ping.unicast                - [Moondark] [1] disconnecting from [Moondark][lJ5VZ7IGQ5C1cG38rxoSZA][tsl0mag14.skead.no][inet[/151.187.99.214:12500]]\r\n2014-06-16 10:18:27,223 TRACE discovery.zen.ping.unicast                - [Moondark] [1] disconnecting from [#zen_unicast_1#][tsl0mag14.skead.no][inet[tsl0mag12/151.187.99.212:12500]]\r\n2014-06-16 10:18:27,223 TRACE discovery.zen.ping.unicast                - [Moondark] [1] disconnecting from [#zen_unicast_3#][tsl0mag14.skead.no][inet[tsl0mag15/151.187.99.215:12500]]\r\n2014-06-16 10:18:27,226 TRACE discovery.zen                             - [Moondark] full ping responses:\r\n        --> target [[Shatterstar][IQoquIjSRjOpbMenBqAv3A][tsl0mag12.skead.no][inet[/151.187.99.212:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Sushi][iCNA-wEbSXOSn71Gngr_Qg][tsl0mag16.skead.no][inet[/151.187.99.216:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Venus Dee Milo][ZogIMgj1R5WedZVR7f2A_w][tsl0mag17.skead.no][inet[/151.187.99.217:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n2014-06-16 10:18:27,227 DEBUG discovery.zen                             - [Moondark] filtered ping responses: (filter_client[true], filter_data[false])\r\n        --> target [[Shatterstar][IQoquIjSRjOpbMenBqAv3A][tsl0mag12.skead.no][inet[/151.187.99.212:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Sushi][iCNA-wEbSXOSn71Gngr_Qg][tsl0mag16.skead.no][inet[/151.187.99.216:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n        --> target [[Venus Dee Milo][ZogIMgj1R5WedZVR7f2A_w][tsl0mag17.skead.no][inet[/151.187.99.217:12500]]], master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]]\r\n2014-06-16 10:18:27,472 DEBUG discovery.zen.fd                          - [Moondark] [master] starting fault detection against master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], reason [initial_join]\r\n2014-06-16 10:18:27,483 DEBUG discovery.zen.fd                          - [Moondark] [master] stopping fault detection against master [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], reason [master failure, failed to perform initial connect [[Volstagg][inet[/151.187.99.215:12500]] connect_timeout[30s]]]\r\n2014-06-16 10:18:27,483 INFO  discovery.zen                             - [Moondark] master_left [[Volstagg][PKLH_LAsQhOBmv7U2vk_5A][tsl0mag15.skead.no][inet[/151.187.99.215:12500]]], reason [failed to perform initial connect [[Volstagg][inet[/151.187.99.215:12500]] connect_timeout[30s]]]\r\n2014-06-16 10:18:27,489 TRACE discovery                                 - [Moondark] initial state set from discovery\r\n2014-06-16 10:18:27,489 INFO  discovery                                 - [Moondark] IRIS-GP2-N-S/lJ5VZ7IGQ5C1cG38rxoSZA\r\n2014-06-16 10:18:27,498 INFO  http                                      - [Moondark] bound_address {inet[/0:0:0:0:0:0:0:0:12400]}, publish_address {inet[/151.187.99.214:12400]}\r\n2014-06-16 10:18:27,510 INFO  node                                      - [Moondark] started\r\n```"}}