{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/13530","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13530/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13530/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13530/events","html_url":"https://github.com/elastic/elasticsearch/issues/13530","id":106128129,"number":13530,"title":"somtimes data node can not find master.","user":{"login":"darjeelings","id":14049892,"avatar_url":"https://avatars.githubusercontent.com/u/14049892?v=3","gravatar_id":"","url":"https://api.github.com/users/darjeelings","html_url":"https://github.com/darjeelings","followers_url":"https://api.github.com/users/darjeelings/followers","following_url":"https://api.github.com/users/darjeelings/following{/other_user}","gists_url":"https://api.github.com/users/darjeelings/gists{/gist_id}","starred_url":"https://api.github.com/users/darjeelings/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/darjeelings/subscriptions","organizations_url":"https://api.github.com/users/darjeelings/orgs","repos_url":"https://api.github.com/users/darjeelings/repos","events_url":"https://api.github.com/users/darjeelings/events{/privacy}","received_events_url":"https://api.github.com/users/darjeelings/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-12T04:19:07Z","updated_at":"2015-12-09T23:56:48Z","closed_at":"2015-09-13T09:15:20Z","body":"My cluster has 9 data nodes / 2 master / 1 search / 112 node client.\r\n\r\nSometimes data node can't find master node.\r\n\r\nThere was no network, cpu load, memory problem.\r\nNo status change log.\r\n\r\nI'm using this on AWS EC2. Therefore I am using elasticsearch-cloud-aws plugin(https://github.com/elastic/elasticsearch-cloud-aws#generic-configuration).\r\n\r\nI set up the ping_timeout 60s.\r\n\r\nIs it normal?\r\n\r\n\r\n\r\nMaster Log\r\n==================================================\r\n[2015-09-11 06:13:31,348][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101517] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}]) \r\n[2015-09-11 06:14:17,137][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101518] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~46s)\r\n[2015-09-11 06:14:50,083][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101519] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~33s)\r\n[2015-09-11 06:15:20,093][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101520] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~40s)\r\n[2015-09-11 06:19:32,729][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101521] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~4m 12s)\r\n[2015-09-11 06:20:02,734][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101522] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~30s)\r\n[2015-09-11 06:21:17,182][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101523] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~1m 15s)\r\n[2015-09-11 06:21:47,192][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101524] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~30s)\r\n[2015-09-11 06:22:17,203][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101525] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~30s)\r\n[2015-09-11 06:22:47,213][WARN ][discovery.zen.publish    ] [es-master-1] timed out waiting for all nodes to process published state [101526] (timeout [30s], pending nodes: [[es-data-03][CY6HkL\r\nlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false}])  (~30s)\r\n[2015-09-11 06:28:32,613][INFO ][cluster.service          ] [es-master-1] removed {[es-data-03][CY6HkLlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_stora\r\nge_nodes=1, master=false},}, reason: zen-disco-node_failed([es-data-03][CY6HkLlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, master=false\r\n}), reason transport disconnected \r\n[2015-09-11 06:29:38,181][INFO ][cluster.service          ] [es-master-1] added {[es-data-03][CY6HkLlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage\r\n_nodes=1, master=false},}, reason: zen-disco-receive(join from node[[es-data-03][CY6HkLlXSwmqH6BqeTRvcQ][ip-x-x-x-x][inet[/xx.xx.xx.xx:9300]]{rack=rack_tone, max_local_storage_nodes=1, mas\r\nter=false}]) \r\n\r\n\r\nDATA LOG\r\n=============================================\r\n[2015-09-11 06:28:30,715][WARN ][discovery.ec2            ] [es-data-03] master left (reason = do not exists on master, act as master failure), current nodes: {[es-xxxxx-xxx-xxx-xxx-xxxxx-3-3\r\n][_oe3CD_LQZuouoTmPk-tCg][xxxxxxx][inet[/xx.xx.xx.xx:9306]]{client=true, data=false},[es-xxxxx-xxx-xxx-xxx-xxxxx-2-2][ENE4uw6zQUedCo-lqrmSkw][xxxxxxxxxx][inet[/xx.xx.xx.xx:9304]]{clie\r\nnt=true, data=false},[es-xxxxx-xxx-xxx-dcl-xxxxx-2-2][wITo5HsPQdWFEnapurGlyQ][xxxxxxxxxx][inet[/xx.xx.xx.xx:9307]]{client=true, data=false},[es-xxxxx-xxx-xxx-xxxx-xxxxx-2-2][g5bwjPizSia\r\n\r\n...\r\n\r\n \r\n\r\n[2015-09-11 06:28:30,716][INFO ][cluster.service          ] [es-data-03] removed {[es-master-1][qtVZhNb2T2uuyuEHgIPBmQ][ip-xx-xx-xx-xx][inet[/xx.xx.xx.xx:9300]]{data=false, rack=rack_tone, max\r\n_local_storage_nodes=1, master=true},}, reason: zen-disco-master_failed ([es-master-1][qtVZhNb2T2uuyuEHgIPBmQ][ip-xx-xx-xx-xx][inet[/xx.xx.xx.xx:9300]]{data=false, rack=rack_tone, max_local_st\r\norage_nodes=1, master=true})\r\n[2015-09-11 06:29:31,881][INFO ][cluster.service          ] [es-data-03] detected_master [es-master-1][qtVZhNb2T2uuyuEHgIPBmQ][ip-xx-xx-xx-xx][inet[/xx.xx.xx.xx:9300]]{data=false, rack=rack_to\r\nne, max_local_storage_nodes=1, master=true}, added {[es-master-1][qtVZhNb2T2uuyuEHgIPBmQ][ip-xx-xx-xx-xx][inet[/xx.xx.xx.xx:9300]]{data=false, rack=rack_tone, max_local_storage_nodes=1, master\r\n=true},}, reason: zen-disco-receive(from master [[es-master-1][qtVZhNb2T2uuyuEHgIPBmQ][ip-xx-xx-xx-xx][inet[/xx.xx.xx.xx:9300]]{data=false, rack=rack_tone, max_local_storage_nodes=1, master=tr\r\nue}])\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/163440890","html_url":"https://github.com/elastic/elasticsearch/issues/13530#issuecomment-163440890","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/13530","id":163440890,"user":{"login":"markwalkom","id":3184718,"avatar_url":"https://avatars.githubusercontent.com/u/3184718?v=3","gravatar_id":"","url":"https://api.github.com/users/markwalkom","html_url":"https://github.com/markwalkom","followers_url":"https://api.github.com/users/markwalkom/followers","following_url":"https://api.github.com/users/markwalkom/following{/other_user}","gists_url":"https://api.github.com/users/markwalkom/gists{/gist_id}","starred_url":"https://api.github.com/users/markwalkom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markwalkom/subscriptions","organizations_url":"https://api.github.com/users/markwalkom/orgs","repos_url":"https://api.github.com/users/markwalkom/repos","events_url":"https://api.github.com/users/markwalkom/events{/privacy}","received_events_url":"https://api.github.com/users/markwalkom/received_events","type":"User","site_admin":false},"created_at":"2015-12-09T23:56:48Z","updated_at":"2015-12-09T23:56:48Z","body":"@hmorgado please use the forums"}}