{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2964","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2964/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2964/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2964/events","html_url":"https://github.com/coreos/etcd/issues/2964","id":87606267,"number":2964,"title":"etcd2 cluster in AWS auto-scaling group configuration - not working","user":{"login":"xueshanf","id":4178225,"avatar_url":"https://avatars.githubusercontent.com/u/4178225?v=3","gravatar_id":"","url":"https://api.github.com/users/xueshanf","html_url":"https://github.com/xueshanf","followers_url":"https://api.github.com/users/xueshanf/followers","following_url":"https://api.github.com/users/xueshanf/following{/other_user}","gists_url":"https://api.github.com/users/xueshanf/gists{/gist_id}","starred_url":"https://api.github.com/users/xueshanf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xueshanf/subscriptions","organizations_url":"https://api.github.com/users/xueshanf/orgs","repos_url":"https://api.github.com/users/xueshanf/repos","events_url":"https://api.github.com/users/xueshanf/events{/privacy}","received_events_url":"https://api.github.com/users/xueshanf/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-06-12T04:51:47Z","updated_at":"2015-06-12T15:01:52Z","closed_at":"2015-06-12T15:01:52Z","body":"I am following the [culster-discovery]( https://coreos.com/docs/cluster-management/setup/cluster-discovery) to setup an etcd2 cluster with 3 machines in auto-scaling group. All 3 machines are configured with the same configuration using the same discovery token. Cluster size is 3. The initial new cluster started as expected, no any issues.\r\n\r\nIn order to test auto-scaling,  I terminated one of the 3 machines. AWS auto-scaling policy  started an new instance. \r\n\r\nHowever the new instance (with the cloud-config), cannot join the cluster. I have waited many hours.\r\n\r\nOn  the new instance:\r\n```\r\ncore@ip-10-0-2-116 ~ $ etcdctl member list\r\n329a33449d3bb66e: name=59b0ca34f5794ab487bfed84fcc8b6e9 peerURLs=http://10.0.2.83:2380 clientURLs=http://10.0.2.83:2379\r\n580214ba0e1f72e5: name=b3b39e5158a546f489af32a28613de32 peerURLs=http://10.0.2.55:2380 clientURLs=http://10.0.2.55:2379\r\n5a849f7b5adf207b: name=6ca7c06784414be886efa8c38c354e9d peerURLs=http://10.0.2.154:2380 clientURLs=http://10.0.2.154:2379\r\n\r\ncore@ip-10-0-2-116 ~ $ etcdctl cluster-health\r\ncluster is healthy\r\nmember 329a33449d3bb66e is unhealthy\r\nmember 580214ba0e1f72e5 is healthy\r\nmember 5a849f7b5adf207b is healthy\r\n```\r\nThe 10.0.2.83 instance was the original one that I terminated in order test auto-scaling. The new instance is 10.0.2.116, which is not on the list. \r\n\r\nAnd here is the systemd log:\r\n```\r\ncore@ip-10-0-2-116 ~ $ systemctl status etcd2\r\n● etcd2.service - etcd2\r\n   Loaded: loaded (/usr/lib64/systemd/system/etcd2.service; static; vendor preset: disabled)\r\n  Drop-In: /run/systemd/system/etcd2.service.d\r\n           └─10-oem.conf, 20-cloudinit.conf\r\n   Active: active (running) since Fri 2015-06-12 00:57:07 UTC; 2h 7min ago\r\n Main PID: 610 (etcd2)\r\n   Memory: 13.4M\r\n   CGroup: /system.slice/etcd2.service\r\n           └─610 /usr/bin/etcd2\r\n\r\nJun 12 03:04:26 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:26 proxy: zero endpoints currently available\r\nJun 12 03:04:26 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:26 proxy: marked endpoint http://10.0.2.55:2379 available\r\nJun 12 03:04:26 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:26 proxy: marked endpoint http://10.0.2.154:2379 available\r\nJun 12 03:04:30 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:30 proxy: marked endpoint http://10.0.2.83:2379 available\r\nJun 12 03:04:32 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:32 proxy: failed to direct request to http://10.0.2.83:2379: net/http: request canceled while waiting for connection\r\nJun 12 03:04:32 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:32 proxy: marked endpoint http://10.0.2.83:2379 unavailable\r\nJun 12 03:04:37 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:37 proxy: marked endpoint http://10.0.2.83:2379 available\r\n<bold>Jun 12 03:04:37 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:37 proxy: failed to direct request to http://10.0.2.83:2379: dial tcp 10.0.2.83:2379: no route to host</bold>\r\nJun 12 03:04:37 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:37 proxy: marked endpoint http://10.0.2.83:2379 unavailable\r\nJun 12 03:04:42 ip-10-0-2-116.us-west-2.compute.internal etcd2[610]: 2015/06/12 03:04:42 proxy: marked endpoint http://10.0.2.83:2379 available\r\n```\r\nSo somehow the etcd2 is trying to act in proxy mode and talking to the dead http://10.0.2.83:2379 endpoint. \r\n\r\nTo try to get the new instance in the cluster, I tried:\r\n* Delete the dead node using `etcdctl member <remove>`\r\n* Add new node using `etcdctl member add <name>  http://10.0.2.116:2380`\r\n* Restart etcd2 on all of them\r\n\r\nI still could not get the new node to join the cluster.\r\n\r\nThe last thing I tried is to stop etcd2 on the new instance, remove  /var/lib/etcd2/proxy directory and start the etcd2 again. Still cannot join the cluster. I am getting  __etcd: discovery cluster full, falling back to proxy__. See logs below. \r\n\r\n```\r\nip-10-0-2-116 etcd2 # systemctl start  etcd2\r\nip-10-0-2-116 etcd2 # systemctl status  etcd2\r\n● etcd2.service - etcd2\r\n   Loaded: loaded (/usr/lib64/systemd/system/etcd2.service; static; vendor preset: disabled)\r\n  Drop-In: /run/systemd/system/etcd2.service.d\r\n           └─10-oem.conf, 20-cloudinit.conf\r\n   Active: active (running) since Fri 2015-06-12 04:11:13 UTC; 3s ago\r\n Main PID: 17904 (etcd2)\r\n   Memory: 2.8M\r\n   CGroup: /system.slice/etcd2.service\r\n           └─17904 /usr/bin/etcd2\r\n\r\nJun 12 04:11:13 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:13 etcdserver: datadir is valid for the 2.0.1 format\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 etcd: stopping listening for client requests on http://0.0.0.0:4001\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 etcd: stopping listening for client requests on http://0.0.0.0:2379\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 etcd: stopping listening for peers on http://10.0.2.116:7001\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 etcd: stopping listening for peers on http://10.0.2.116:2380\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 etcd: discovery cluster full, falling back to proxy\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 proxy: using peer urls [http://10.0.2.154:2380 http://10.0.2.55:2380 http:...83:2380]\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 proxy: updated peer urls in cluster file from [http://10.0.2.154:2380 http...55:2380]\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 proxy: listening for client requests on 0.0.0.0:2379\r\nJun 12 04:11:14 ip-10-0-2-116.us-west-2.compute.internal etcd2[17904]: 2015/06/12 04:11:14 proxy: listening for client requests on 0.0.0.0:4001\r\n```\r\nIt seems that, if an instance in an auto-scaling group dies, the replacement instance cannot join the cluster automatically, and I am also having trouble to fix it manually.\r\n\r\nI have been using static IP for etcd cluster for awhile, but would like to use auto-scaling group with discovery token to achieve further automation.\r\n\r\nHere is system information it it  helps:\r\n```\r\nip-10-0-2-116 etcd2 # docker info\r\nContainers: 0\r\nImages: 18\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nKernel Version: 4.0.3\r\nOperating System: CoreOS 695.0.0\r\n```\r\n\r\nCloud-config:\r\n```\r\n#cloud-config\r\ncoreos:\r\n  etcd2:\r\n    # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3\r\n    discovery: https://discovery.etcd.io/84f89c6680532f1c3e97ea520764ab48\r\n    # multi-region and multi-cloud deployments need to use $public_ipv4\r\n    advertise-client-urls: http://$private_ipv4:2379\r\n    initial-advertise-peer-urls: http://$private_ipv4:2380\r\n    # listen on both the official ports and the legacy ports\r\n    # legacy ports can be omitted if your application doesn't depend on them\r\n    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001\r\n    listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001\r\n  fleet:\r\n    metadata: disk=ssd,env=itlab,platform=ec2,provider=aws,region=us-west-2,role=etcd2\r\n    public-ip: $private_ipv4\r\n```\r\nAnother question, when using CoreOS cloud-config to bootstrap a cluster and when the cluster is fully functional, under what condition one needs to use etcdctl to add/remove members manually? Shouldn't the new machine join the cluster automatically if it has the same discovery token?\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/111519465","html_url":"https://github.com/coreos/etcd/issues/2964#issuecomment-111519465","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2964","id":111519465,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T15:01:52Z","updated_at":"2015-06-12T15:01:52Z","body":"@xueshanf Yes. You are right. You need to use new configuration for joining existing cluster. You can build a tine wrapper around etcd to do that too."}}