{"action":"created","issue":{"url":"https://api.github.com/repos/openshift/openshift-ansible/issues/1019","labels_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/1019/labels{/name}","comments_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/1019/comments","events_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/1019/events","html_url":"https://github.com/openshift/openshift-ansible/pull/1019","id":120404085,"number":1019,"title":"Install and start etcd servers one at a time.","user":{"login":"dgoodwin","id":51265,"avatar_url":"https://avatars.githubusercontent.com/u/51265?v=3","gravatar_id":"","url":"https://api.github.com/users/dgoodwin","html_url":"https://github.com/dgoodwin","followers_url":"https://api.github.com/users/dgoodwin/followers","following_url":"https://api.github.com/users/dgoodwin/following{/other_user}","gists_url":"https://api.github.com/users/dgoodwin/gists{/gist_id}","starred_url":"https://api.github.com/users/dgoodwin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dgoodwin/subscriptions","organizations_url":"https://api.github.com/users/dgoodwin/orgs","repos_url":"https://api.github.com/users/dgoodwin/repos","events_url":"https://api.github.com/users/dgoodwin/events{/privacy}","received_events_url":"https://api.github.com/users/dgoodwin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-12-04T13:42:40Z","updated_at":"2015-12-11T13:47:05Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/openshift/openshift-ansible/pulls/1019","html_url":"https://github.com/openshift/openshift-ansible/pull/1019","diff_url":"https://github.com/openshift/openshift-ansible/pull/1019.diff","patch_url":"https://github.com/openshift/openshift-ansible/pull/1019.patch"},"body":"This might need some discussion but I believe we have a bug with etcd cluster setup in faster environments, all the cluster members appear to come up at roughly the same time when using all local vms (within a second or two) and one or two of my three will fail to make it to the cluster. \r\n\r\nWhen this bug happens a failed member's etcd logs will look like:\r\n\r\nhttp://fpaste.org/296258/89844131/\r\n\r\nWhereas one of the member's that made it up will look like: \r\n\r\nhttp://fpaste.org/296239/14489825/\r\n\r\nLogging into those members and removing the etcd data dir and restarting the service will get that member to cluster up.\r\n\r\nAdding members one at a time appears to be recommended best practice at least for cluster recovery, which is essentially the same thing: https://github.com/coreos/etcd/blob/master/Documentation/runtime-configuration.md#restart-cluster-from-majority-failure\r\n\r\nSerializing the etcd setup to one server at a time appears to correct this problem, the cluster comes up healthy.\r\n\r\nI suspect this hasn't surfaced in cloud environments just due to the delays in communicating with them and the nodes have time to come up individually.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/openshift/openshift-ansible/issues/comments/163939955","html_url":"https://github.com/openshift/openshift-ansible/pull/1019#issuecomment-163939955","issue_url":"https://api.github.com/repos/openshift/openshift-ansible/issues/1019","id":163939955,"user":{"login":"dgoodwin","id":51265,"avatar_url":"https://avatars.githubusercontent.com/u/51265?v=3","gravatar_id":"","url":"https://api.github.com/users/dgoodwin","html_url":"https://github.com/dgoodwin","followers_url":"https://api.github.com/users/dgoodwin/followers","following_url":"https://api.github.com/users/dgoodwin/following{/other_user}","gists_url":"https://api.github.com/users/dgoodwin/gists{/gist_id}","starred_url":"https://api.github.com/users/dgoodwin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dgoodwin/subscriptions","organizations_url":"https://api.github.com/users/dgoodwin/orgs","repos_url":"https://api.github.com/users/dgoodwin/repos","events_url":"https://api.github.com/users/dgoodwin/events{/privacy}","received_events_url":"https://api.github.com/users/dgoodwin/received_events","type":"User","site_admin":false},"created_at":"2015-12-11T13:47:05Z","updated_at":"2015-12-11T13:47:05Z","body":"That worked, thanks!\r\n\r\nPR updated."}}