{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/936","labels_url":"https://api.github.com/repos/coreos/bugs/issues/936/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/936/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/936/events","html_url":"https://github.com/coreos/bugs/issues/936","id":112397722,"number":936,"title":"Cluster starting unhealthy on CoreOS alpha (835.1.0)","user":{"login":"Zapadlo","id":1801067,"avatar_url":"https://avatars.githubusercontent.com/u/1801067?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapadlo","html_url":"https://github.com/Zapadlo","followers_url":"https://api.github.com/users/Zapadlo/followers","following_url":"https://api.github.com/users/Zapadlo/following{/other_user}","gists_url":"https://api.github.com/users/Zapadlo/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapadlo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapadlo/subscriptions","organizations_url":"https://api.github.com/users/Zapadlo/orgs","repos_url":"https://api.github.com/users/Zapadlo/repos","events_url":"https://api.github.com/users/Zapadlo/events{/privacy}","received_events_url":"https://api.github.com/users/Zapadlo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/bugs/labels/component/distro","name":"component/distro","color":"0052cc"},{"url":"https://api.github.com/repos/coreos/bugs/labels/component/etcd","name":"component/etcd","color":"0052cc"},{"url":"https://api.github.com/repos/coreos/bugs/labels/kind/friction","name":"kind/friction","color":"c7def8"},{"url":"https://api.github.com/repos/coreos/bugs/labels/low%20hanging%20fruit","name":"low hanging fruit","color":"207de5"},{"url":"https://api.github.com/repos/coreos/bugs/labels/priority/P0","name":"priority/P0","color":"DF0101"},{"url":"https://api.github.com/repos/coreos/bugs/labels/team/os","name":"team/os","color":"ffffff"}],"state":"open","locked":false,"assignee":{"login":"crawford","id":368848,"avatar_url":"https://avatars.githubusercontent.com/u/368848?v=3","gravatar_id":"","url":"https://api.github.com/users/crawford","html_url":"https://github.com/crawford","followers_url":"https://api.github.com/users/crawford/followers","following_url":"https://api.github.com/users/crawford/following{/other_user}","gists_url":"https://api.github.com/users/crawford/gists{/gist_id}","starred_url":"https://api.github.com/users/crawford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crawford/subscriptions","organizations_url":"https://api.github.com/users/crawford/orgs","repos_url":"https://api.github.com/users/crawford/repos","events_url":"https://api.github.com/users/crawford/events{/privacy}","received_events_url":"https://api.github.com/users/crawford/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/coreos/bugs/milestones/11","html_url":"https://github.com/coreos/bugs/milestones/CoreOS%20891.0.0","labels_url":"https://api.github.com/repos/coreos/bugs/milestones/11/labels","id":1440914,"number":11,"title":"CoreOS 891.0.0","description":"","creator":{"login":"crawford","id":368848,"avatar_url":"https://avatars.githubusercontent.com/u/368848?v=3","gravatar_id":"","url":"https://api.github.com/users/crawford","html_url":"https://github.com/crawford","followers_url":"https://api.github.com/users/crawford/followers","following_url":"https://api.github.com/users/crawford/following{/other_user}","gists_url":"https://api.github.com/users/crawford/gists{/gist_id}","starred_url":"https://api.github.com/users/crawford/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crawford/subscriptions","organizations_url":"https://api.github.com/users/crawford/orgs","repos_url":"https://api.github.com/users/crawford/repos","events_url":"https://api.github.com/users/crawford/events{/privacy}","received_events_url":"https://api.github.com/users/crawford/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":2,"state":"open","created_at":"2015-12-02T22:26:54Z","updated_at":"2015-12-07T21:39:00Z","due_on":"2015-12-09T08:00:00Z","closed_at":null},"comments":11,"created_at":"2015-10-20T15:40:15Z","updated_at":"2015-12-07T22:42:50Z","closed_at":null,"body":"Tried spinning up a cluster 5 - 6 times with various conf for initial number of members, timeouts for etcd2, most of etcd2 logs look like this:\r\n\r\n```\r\n-- Logs begin at Tue 2015-10-20 15:28:07 UTC, end at Tue 2015-10-20 15:37:43 UTC. --\r\nOct 20 15:28:46 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Starting etcd2...\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.24.67.222:2379,http://172.24.67.222:4001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/27ec448294cfd99ecb2efa7fa236d130\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=3000\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=300\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.24.67.222:2380\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.24.67.222:2380,http://172.24.67.222:7001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_NAME=5d2095e3c4af4ab0a1b319197527799b\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: etcd Version: 2.2.0\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Git SHA: e4561dd\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Go Version: go1.5.1\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Go OS/Arch: linux/amd64\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: setting maximum number of CPUs to 2, total number of available CPUs is 2\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for peers on http://172.24.67.222:2380\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for peers on http://172.24.67.222:7001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for client requests on http://0.0.0.0:2379\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for client requests on http://0.0.0.0:4001\r\nOct 20 15:28:51 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found self e8e8214d7021c972 in the cluster\r\nOct 20 15:28:51 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found 1 peer(s), waiting for 2 more\r\nOct 20 15:29:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found peer bfb257261d8106b9 in the cluster\r\nOct 20 15:29:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found 2 peer(s), waiting for 1 more\r\nOct 20 15:30:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: error #0: client: etcd member https://discovery.etcd.io returns server error [Gateway Timeout]\r\nOct 20 15:30:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: waiting for other nodes: error connecting to https://discovery.etcd.io, retrying in 2s\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Start operation timed out. Terminating.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=2/INVALIDARGUMENT\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Failed to start etcd2.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Unit entered failed state.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Failed with result 'exit-code'.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Service hold-off time over, scheduling restart.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Stopped etcd2.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Starting etcd2...\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.24.67.222:2379,http://172.24.67.222:4001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/27ec448294cfd99ecb2efa7fa236d130\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=3000\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=300\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.24.67.222:2380\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.24.67.222:2380,http://172.24.67.222:7001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_NAME=5d2095e3c4af4ab0a1b319197527799b\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: etcd Version: 2.2.0\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Git SHA: e4561dd\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Go Version: go1.5.1\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Go OS/Arch: linux/amd64\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: setting maximum number of CPUs to 2, total number of available CPUs is 2\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: the server is already initialized as member before, starting as etcd member...\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for peers on http://172.24.67.222:2380\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for peers on http://172.24.67.222:7001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for client requests on http://0.0.0.0:2379\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for client requests on http://0.0.0.0:4001\r\nOct 20 15:30:28 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE\r\nOct 20 15:30:28 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Failed to start etcd2.\r\n```\r\nand keep repeating.\r\n\r\nAnother host:\r\n```\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a is starting a new election at term 109\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a became candidate at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a received vote from 5b34a53a8df3c62a at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a [logterm: 1, index: 3] sent vote request to bfb257261d8106b9 at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a [logterm: 1, index: 3] sent vote request to e8e8214d7021c972 at term 110\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream MsgApp v2 (net/http: transport closed before response was received)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream Message (read tcp 172.24.58.39:47403->172.24.155.123:2380: read: connection reset by peer)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream Message (dial tcp 172.24.155.123:2380: getsockopt: connection refused)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream MsgApp v2 (dial tcp 172.24.155.123:2380: getsockopt: connection refused)\r\n```\r\n\r\n~~I'm spinning up the cluster with ansible, where hosts come up one by one sequentially, does that matter?~~\r\nIt doesn't, I switched back to stable (766.4) with the same settings and cluster came up healthy."},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/162692450","html_url":"https://github.com/coreos/bugs/issues/936#issuecomment-162692450","issue_url":"https://api.github.com/repos/coreos/bugs/issues/936","id":162692450,"user":{"login":"mischief","id":1487996,"avatar_url":"https://avatars.githubusercontent.com/u/1487996?v=3","gravatar_id":"","url":"https://api.github.com/users/mischief","html_url":"https://github.com/mischief","followers_url":"https://api.github.com/users/mischief/followers","following_url":"https://api.github.com/users/mischief/following{/other_user}","gists_url":"https://api.github.com/users/mischief/gists{/gist_id}","starred_url":"https://api.github.com/users/mischief/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mischief/subscriptions","organizations_url":"https://api.github.com/users/mischief/orgs","repos_url":"https://api.github.com/users/mischief/repos","events_url":"https://api.github.com/users/mischief/events{/privacy}","received_events_url":"https://api.github.com/users/mischief/received_events","type":"User","site_admin":false},"created_at":"2015-12-07T22:42:50Z","updated_at":"2015-12-07T22:42:50Z","body":"i think the core issue here is that etcd will not allow you to run a given instance of it twice with the same discovery URL. in the code, if etcd tries to create an entry in the discovery service and finds that the ID already exists, it is treated as a fatal error. it's not clear if this is optimal behavior, but it certainly can cause frustration when the window for cluster convergence is as small as 90 seconds, and you can't restart any etcd member during that time or you will need to intervene and fix it by hand."}}