{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/203","labels_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/203/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/203/comments","events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/203/events","html_url":"https://github.com/Metaswitch/clearwater-etcd/issues/203","id":112359363,"number":203,"title":"When an existing node is broken, adding multiple new nodes fails and never recovers","user":{"login":"AlexHockey","id":4903205,"avatar_url":"https://avatars.githubusercontent.com/u/4903205?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexHockey","html_url":"https://github.com/AlexHockey","followers_url":"https://api.github.com/users/AlexHockey/followers","following_url":"https://api.github.com/users/AlexHockey/following{/other_user}","gists_url":"https://api.github.com/users/AlexHockey/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexHockey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexHockey/subscriptions","organizations_url":"https://api.github.com/users/AlexHockey/orgs","repos_url":"https://api.github.com/users/AlexHockey/repos","events_url":"https://api.github.com/users/AlexHockey/events{/privacy}","received_events_url":"https://api.github.com/users/AlexHockey/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels/high-priority","name":"high-priority","color":"ff0000"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-20T12:41:40Z","updated_at":"2015-10-20T12:49:28Z","closed_at":null,"body":"I performed the following actions:\r\n-  Spun up a deployment using chef. \r\n-  Added a new node to the deployment. This failed, so I deleted the node using chef. This deleted the VM without removing the node from the cluster. \r\n-  Added three new nodes to the deployment. The etcd processes on the new nodes never came up and repeatedly outputed the following logs:\r\n\r\n```\r\n2015/10/20 12:12:37 etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1\r\n2015/10/20 12:12:37 etcdmain: listening for peers on http://10.71.155.71:2380\r\n2015/10/20 12:12:37 etcdmain: listening for client requests on http://10.71.155.71:4000\r\n2015/10/20 12:12:37 etcdmain: stopping listening for client requests on http://10.71.155.71:4000\r\n2015/10/20 12:12:37 etcdmain: stopping listening for peers on http://10.71.155.71:2380\r\n2015/10/20 12:12:37 etcdmain: error validating peerURLs {ClusterID:6c8276528454b43e Members:[&{ID:fb642d0630a4d9c6 RaftAttributes:{PeerURLs:[http://10.81.155.189:2380]} Attributes:{Name:10-81-155-189 ClientURLs:[http://1\r\n0.81.155.189:4000]}} &{ID:36b838dc700e6644 RaftAttributes:{PeerURLs:[http://10.230.163.7:2380]} Attributes:{Name:10-230-163-7 ClientURLs:[http://10.230.163.7:4000]}} &{ID:6482a9ec9ab80aca RaftAttributes:{PeerURLs:[http:/\r\n/10.151.20.136:2380]} Attributes:{Name:10-151-20-136 ClientURLs:[http://10.151.20.136:4000]}} &{ID:adbf1c1978a38ba6 RaftAttributes:{PeerURLs:[http://10.151.142.166:2380]} Attributes:{Name:10-151-142-166 ClientURLs:[http:\r\n//10.151.142.166:4000]}} &{ID:b14f878ebfa44ed1 RaftAttributes:{PeerURLs:[http://10.71.155.71:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:f8cf26bf637a95fc RaftAttributes:{PeerURLs:[http://10.109.154.19:2380]} Attributes\r\n:{Name: ClientURLs:[]}} &{ID:503cb247216a04b9 RaftAttributes:{PeerURLs:[http://10.102.148.132:2380]} Attributes:{Name:10-102-148-132 ClientURLs:[http://10.102.148.132:4000]}} &{ID:cdb458d8aa24cac2 RaftAttributes:{PeerURL\r\ns:[http://10.9.161.233:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:cfae4d18031f8e11 RaftAttributes:{PeerURLs:[http://10.81.146.221:2380]} Attributes:{Name:10-81-146-221 ClientURLs:[http://10.81.146.221:4000]}}] Removed\r\nMemberIDs:[]}: member count is unequal\r\n```"},"comment":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/comments/149555912","html_url":"https://github.com/Metaswitch/clearwater-etcd/issues/203#issuecomment-149555912","issue_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/203","id":149555912,"user":{"login":"AlexHockey","id":4903205,"avatar_url":"https://avatars.githubusercontent.com/u/4903205?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexHockey","html_url":"https://github.com/AlexHockey","followers_url":"https://api.github.com/users/AlexHockey/followers","following_url":"https://api.github.com/users/AlexHockey/following{/other_user}","gists_url":"https://api.github.com/users/AlexHockey/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexHockey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexHockey/subscriptions","organizations_url":"https://api.github.com/users/AlexHockey/orgs","repos_url":"https://api.github.com/users/AlexHockey/repos","events_url":"https://api.github.com/users/AlexHockey/events{/privacy}","received_events_url":"https://api.github.com/users/AlexHockey/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T12:49:27Z","updated_at":"2015-10-20T12:49:27Z","body":"@JimBunch can I get your opinion on the criticality of this issue please? The symptoms are that if you add new nodes when some nodes are failed, the new nodes may fail to join the etcd cluster and the system will not automatically recover."}}