{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1212","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/events","html_url":"https://github.com/hashicorp/consul/issues/1212","id":104321225,"number":1212,"title":"Node health flapping - EC2","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2015-09-01T17:55:24Z","updated_at":"2015-09-16T21:20:17Z","closed_at":null,"body":"We have a five node Consul cluster handling roughly 30 nodes across 4 different AWS accounts in a shared VPC across different availability zones. For the most part, everything works great. However, quite frequently, a random node will flap from `healthy` to `critical`. The flapping happens on completely random nodes and no consistency whatsoever.\r\n\r\nEvery time a node \"flaps\" it causes our consul-template, which populates our NGINX reverse-proxy config to reload. This is causes things like our Apache benchmark tests to fail.\r\n\r\nWe are looking to use Consul for production, but this issue has caused a lot of people to worry about consistency.\r\n\r\nWe also have all required TCP/UDP ports open through all the nodes, as well.\r\n\r\nWe believe the issue is just a latency problem with the polling of serf. Is there a way to modify the serf health-check interval to adjust to geographical latency?\r\n\r\nHeres the log from one of the Consul servers:\r\n\r\n```\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:13 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:46:15 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:46:15 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:16 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:26 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:32 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:05 [INFO] serf: EventMemberFailed: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:47:19 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:47:19 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:23 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:47:44 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:47:44 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:49 [INFO] memberlist: Marking ip-10-170-76-170 as failed, suspect timeout reached\r\n    2015/09/01 17:47:49 [INFO] serf: EventMemberFailed: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:48:00 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:48:00 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:48:02 [INFO] serf: EventMemberFailed: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:16 [INFO] serf: EventMemberJoin: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:32 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:48:32 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:33 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:45 [INFO] serf: EventMemberFailed: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:46 [INFO] serf: EventMemberJoin: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:55 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:00 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:00 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:20 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:32 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:32 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:40 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:51 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:51 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:52 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberFailed: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberJoin: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:50:04 [INFO] memberlist: Marking ip-10-190-13-188 as failed, suspect timeout reached\r\n    2015/09/01 17:50:04 [INFO] serf: EventMemberFailed: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:05 [INFO] serf: EventMemberJoin: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:20 [INFO] serf: EventMemberFailed: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:24 [INFO] memberlist: Marking ip-10-185-65-7 as failed, suspect timeout reached\r\n    2015/09/01 17:50:24 [INFO] serf: EventMemberFailed: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:50:31 [INFO] serf: EventMemberJoin: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:47 [INFO] serf: EventMemberJoin: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:51:01 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:02 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:09 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:51:09 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:43 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:45 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:52:22 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:52:22 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:52:30 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n```"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/140898240","html_url":"https://github.com/hashicorp/consul/issues/1212#issuecomment-140898240","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1212","id":140898240,"user":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T21:20:16Z","updated_at":"2015-09-16T21:20:16Z","body":"> Since this is such a common issue, it may be worth adding some additional logging for when probes fail or a test command for environment validation.\r\n\r\nThis is definitely a good idea and we've been talking about this internally as well. Given the randomized nature of the pings, and the fact that it will try indirect pings via other nodes Serf/Consul can pave over a lot of different types of failures and configuration issues in ways that can be confusing.  In Consul 0.6 we have added a TCP fallback ping which helps keep the cluster stable while providing some log messages about a possible misconfiguration (\"node X was reachable via TCP but not UDP\").\r\n\r\nI forgot to ask, @djenriquez are you running Consul inside Docker containers as well?"}}