{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/936","labels_url":"https://api.github.com/repos/coreos/bugs/issues/936/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/936/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/936/events","html_url":"https://github.com/coreos/bugs/issues/936","id":112397722,"number":936,"title":"Cluster starting unhealthy on CoreOS alpha (835.1.0)","user":{"login":"Zapadlo","id":1801067,"avatar_url":"https://avatars.githubusercontent.com/u/1801067?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapadlo","html_url":"https://github.com/Zapadlo","followers_url":"https://api.github.com/users/Zapadlo/followers","following_url":"https://api.github.com/users/Zapadlo/following{/other_user}","gists_url":"https://api.github.com/users/Zapadlo/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapadlo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapadlo/subscriptions","organizations_url":"https://api.github.com/users/Zapadlo/orgs","repos_url":"https://api.github.com/users/Zapadlo/repos","events_url":"https://api.github.com/users/Zapadlo/events{/privacy}","received_events_url":"https://api.github.com/users/Zapadlo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/bugs/labels/component/etcd","name":"component/etcd","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-20T15:40:15Z","updated_at":"2015-10-22T10:05:41Z","closed_at":null,"body":"Tried spinning up a cluster 5 - 6 times with various conf for initial number of members, timeouts for etcd2, most of etcd2 logs look like this:\r\n\r\n```\r\n-- Logs begin at Tue 2015-10-20 15:28:07 UTC, end at Tue 2015-10-20 15:37:43 UTC. --\r\nOct 20 15:28:46 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Starting etcd2...\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.24.67.222:2379,http://172.24.67.222:4001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/27ec448294cfd99ecb2efa7fa236d130\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=3000\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=300\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.24.67.222:2380\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.24.67.222:2380,http://172.24.67.222:7001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: recognized and used environment variable ETCD_NAME=5d2095e3c4af4ab0a1b319197527799b\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: etcd Version: 2.2.0\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Git SHA: e4561dd\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Go Version: go1.5.1\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: Go OS/Arch: linux/amd64\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: setting maximum number of CPUs to 2, total number of available CPUs is 2\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for peers on http://172.24.67.222:2380\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for peers on http://172.24.67.222:7001\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for client requests on http://0.0.0.0:2379\r\nOct 20 15:28:49 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: listening for client requests on http://0.0.0.0:4001\r\nOct 20 15:28:51 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found self e8e8214d7021c972 in the cluster\r\nOct 20 15:28:51 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found 1 peer(s), waiting for 2 more\r\nOct 20 15:29:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found peer bfb257261d8106b9 in the cluster\r\nOct 20 15:29:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: found 2 peer(s), waiting for 1 more\r\nOct 20 15:30:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: error #0: client: etcd member https://discovery.etcd.io returns server error [Gateway Timeout]\r\nOct 20 15:30:16 ip-172-24-67-222.eu-west-1.compute.internal etcd2[833]: waiting for other nodes: error connecting to https://discovery.etcd.io, retrying in 2s\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Start operation timed out. Terminating.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=2/INVALIDARGUMENT\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Failed to start etcd2.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Unit entered failed state.\r\nOct 20 15:30:17 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Failed with result 'exit-code'.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Service hold-off time over, scheduling restart.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Stopped etcd2.\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Starting etcd2...\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.24.67.222:2379,http://172.24.67.222:4001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/27ec448294cfd99ecb2efa7fa236d130\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=3000\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_HEARTBEAT_INTERVAL=300\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.24.67.222:2380\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,http://0.0.0.0:4001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.24.67.222:2380,http://172.24.67.222:7001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: recognized and used environment variable ETCD_NAME=5d2095e3c4af4ab0a1b319197527799b\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: etcd Version: 2.2.0\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Git SHA: e4561dd\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Go Version: go1.5.1\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: Go OS/Arch: linux/amd64\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: setting maximum number of CPUs to 2, total number of available CPUs is 2\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: the server is already initialized as member before, starting as etcd member...\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for peers on http://172.24.67.222:2380\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for peers on http://172.24.67.222:7001\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for client requests on http://0.0.0.0:2379\r\nOct 20 15:30:27 ip-172-24-67-222.eu-west-1.compute.internal etcd2[938]: listening for client requests on http://0.0.0.0:4001\r\nOct 20 15:30:28 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE\r\nOct 20 15:30:28 ip-172-24-67-222.eu-west-1.compute.internal systemd[1]: Failed to start etcd2.\r\n```\r\nand keep repeating.\r\n\r\nAnother host:\r\n```\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a is starting a new election at term 109\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a became candidate at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a received vote from 5b34a53a8df3c62a at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a [logterm: 1, index: 3] sent vote request to bfb257261d8106b9 at term 110\r\nOct 20 15:38:20 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: 5b34a53a8df3c62a [logterm: 1, index: 3] sent vote request to e8e8214d7021c972 at term 110\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream MsgApp v2 (net/http: transport closed before response was received)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream Message (read tcp 172.24.58.39:47403->172.24.155.123:2380: read: connection reset by peer)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream Message (dial tcp 172.24.155.123:2380: getsockopt: connection refused)\r\nOct 20 15:38:22 ip-172-24-58-39.eu-west-1.compute.internal etcd2[835]: failed to dial bfb257261d8106b9 on stream MsgApp v2 (dial tcp 172.24.155.123:2380: getsockopt: connection refused)\r\n```\r\n\r\n~~I'm spinning up the cluster with ansible, where hosts come up one by one sequentially, does that matter?~~\r\nIt doesn't, I switched back to stable (766.4) with the same settings and cluster came up healthy."},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/150168079","html_url":"https://github.com/coreos/bugs/issues/936#issuecomment-150168079","issue_url":"https://api.github.com/repos/coreos/bugs/issues/936","id":150168079,"user":{"login":"Zapadlo","id":1801067,"avatar_url":"https://avatars.githubusercontent.com/u/1801067?v=3","gravatar_id":"","url":"https://api.github.com/users/Zapadlo","html_url":"https://github.com/Zapadlo","followers_url":"https://api.github.com/users/Zapadlo/followers","following_url":"https://api.github.com/users/Zapadlo/following{/other_user}","gists_url":"https://api.github.com/users/Zapadlo/gists{/gist_id}","starred_url":"https://api.github.com/users/Zapadlo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zapadlo/subscriptions","organizations_url":"https://api.github.com/users/Zapadlo/orgs","repos_url":"https://api.github.com/users/Zapadlo/repos","events_url":"https://api.github.com/users/Zapadlo/events{/privacy}","received_events_url":"https://api.github.com/users/Zapadlo/received_events","type":"User","site_admin":false},"created_at":"2015-10-22T10:05:41Z","updated_at":"2015-10-22T10:05:41Z","body":"@mischief Actually most of the clusters I set up today started up successfully, had my first failure, but `journalctl -t etcd2` gives me:\r\n```\r\ncore@ip-172-24-144-171 ~ $ sudo journalctl -t etcd2\r\n-- No entries --\r\n```\r\nTa!"}}