{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1212","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/events","html_url":"https://github.com/hashicorp/consul/issues/1212","id":104321225,"number":1212,"title":"Node health flapping - EC2","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"milestone":null,"comments":35,"created_at":"2015-09-01T17:55:24Z","updated_at":"2015-11-02T17:18:11Z","closed_at":null,"body":"We have a five node Consul cluster handling roughly 30 nodes across 4 different AWS accounts in a shared VPC across different availability zones. For the most part, everything works great. However, quite frequently, a random node will flap from `healthy` to `critical`. The flapping happens on completely random nodes and no consistency whatsoever.\r\n\r\nEvery time a node \"flaps\" it causes our consul-template, which populates our NGINX reverse-proxy config to reload. This is causes things like our Apache benchmark tests to fail.\r\n\r\nWe are looking to use Consul for production, but this issue has caused a lot of people to worry about consistency.\r\n\r\nWe also have all required TCP/UDP ports open through all the nodes, as well.\r\n\r\nWe believe the issue is just a latency problem with the polling of serf. Is there a way to modify the serf health-check interval to adjust to geographical latency?\r\n\r\nHeres the log from one of the Consul servers:\r\n\r\n```\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:13 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:46:15 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:46:15 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:16 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:26 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:32 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:05 [INFO] serf: EventMemberFailed: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:47:19 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:47:19 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:23 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:47:44 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:47:44 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:49 [INFO] memberlist: Marking ip-10-170-76-170 as failed, suspect timeout reached\r\n    2015/09/01 17:47:49 [INFO] serf: EventMemberFailed: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:48:00 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:48:00 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:48:02 [INFO] serf: EventMemberFailed: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:16 [INFO] serf: EventMemberJoin: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:32 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:48:32 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:33 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:45 [INFO] serf: EventMemberFailed: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:46 [INFO] serf: EventMemberJoin: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:55 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:00 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:00 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:20 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:32 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:32 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:40 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:51 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:51 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:52 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberFailed: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberJoin: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:50:04 [INFO] memberlist: Marking ip-10-190-13-188 as failed, suspect timeout reached\r\n    2015/09/01 17:50:04 [INFO] serf: EventMemberFailed: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:05 [INFO] serf: EventMemberJoin: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:20 [INFO] serf: EventMemberFailed: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:24 [INFO] memberlist: Marking ip-10-185-65-7 as failed, suspect timeout reached\r\n    2015/09/01 17:50:24 [INFO] serf: EventMemberFailed: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:50:31 [INFO] serf: EventMemberJoin: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:47 [INFO] serf: EventMemberJoin: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:51:01 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:02 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:09 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:51:09 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:43 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:45 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:52:22 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:52:22 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:52:30 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n```"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/153088652","html_url":"https://github.com/hashicorp/consul/issues/1212#issuecomment-153088652","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1212","id":153088652,"user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T17:18:11Z","updated_at":"2015-11-02T17:18:11Z","body":"Hi @sstarcher, i'm not sure why I missed your original post, but I did. Sounds like a solid idea. We are currently not have any flapping at the moment because we fragmented our infrastructure such that the maximum number of nodes in a cluster we have right now is 15 nodes.\r\n\r\nWhen we get back to the amount where flapping happens (~20-25+), I'll try this out. \r\n\r\nIf anyone else is experiencing flapping and would like to give this a shot for the sake of resolving this issue, please do so."}}