{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1212","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1212/events","html_url":"https://github.com/hashicorp/consul/issues/1212","id":104321225,"number":1212,"title":"Node health flapping - EC2","user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"milestone":null,"comments":17,"created_at":"2015-09-01T17:55:24Z","updated_at":"2015-09-17T21:20:00Z","closed_at":null,"body":"We have a five node Consul cluster handling roughly 30 nodes across 4 different AWS accounts in a shared VPC across different availability zones. For the most part, everything works great. However, quite frequently, a random node will flap from `healthy` to `critical`. The flapping happens on completely random nodes and no consistency whatsoever.\r\n\r\nEvery time a node \"flaps\" it causes our consul-template, which populates our NGINX reverse-proxy config to reload. This is causes things like our Apache benchmark tests to fail.\r\n\r\nWe are looking to use Consul for production, but this issue has caused a lot of people to worry about consistency.\r\n\r\nWe also have all required TCP/UDP ports open through all the nodes, as well.\r\n\r\nWe believe the issue is just a latency problem with the polling of serf. Is there a way to modify the serf health-check interval to adjust to geographical latency?\r\n\r\nHeres the log from one of the Consul servers:\r\n\r\n```\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:13 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:46:13 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:46:15 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:46:15 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:16 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:46:26 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:46:32 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:05 [INFO] serf: EventMemberFailed: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:47:19 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:47:19 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:23 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:47:44 [INFO] memberlist: Marking ip-10-190-71-44 as failed, suspect timeout reached\r\n    2015/09/01 17:47:44 [INFO] serf: EventMemberFailed: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:47:45 [INFO] serf: EventMemberJoin: ip-10-190-71-44 10.190.71.44\r\n    2015/09/01 17:47:49 [INFO] memberlist: Marking ip-10-170-76-170 as failed, suspect timeout reached\r\n    2015/09/01 17:47:49 [INFO] serf: EventMemberFailed: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-76-170 10.170.76.170\r\n    2015/09/01 17:47:50 [INFO] serf: EventMemberJoin: ip-10-170-138-228 10.170.138.228\r\n    2015/09/01 17:48:00 [INFO] memberlist: Marking ip-10-170-155-168 as failed, suspect timeout reached\r\n    2015/09/01 17:48:00 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:48:02 [INFO] serf: EventMemberFailed: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:16 [INFO] serf: EventMemberJoin: ip-10-185-23-211 10.185.23.211\r\n    2015/09/01 17:48:32 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:48:32 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:33 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:48:45 [INFO] serf: EventMemberFailed: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:46 [INFO] serf: EventMemberJoin: ip-10-185-23-210 10.185.23.210\r\n    2015/09/01 17:48:55 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:00 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:00 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:20 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:32 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:32 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberFailed: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:38 [INFO] serf: EventMemberJoin: ip-10-170-155-168 10.170.155.168\r\n    2015/09/01 17:49:40 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:51 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:49:51 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:52 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberFailed: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:49:56 [INFO] serf: EventMemberJoin: ip-10-185-15-217 10.185.15.217\r\n    2015/09/01 17:50:04 [INFO] memberlist: Marking ip-10-190-13-188 as failed, suspect timeout reached\r\n    2015/09/01 17:50:04 [INFO] serf: EventMemberFailed: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:05 [INFO] serf: EventMemberJoin: ip-10-190-13-188 10.190.13.188\r\n    2015/09/01 17:50:20 [INFO] serf: EventMemberFailed: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:24 [INFO] memberlist: Marking ip-10-185-65-7 as failed, suspect timeout reached\r\n    2015/09/01 17:50:24 [INFO] serf: EventMemberFailed: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:50:31 [INFO] serf: EventMemberJoin: ip-10-185-77-94 10.185.77.94\r\n    2015/09/01 17:50:47 [INFO] serf: EventMemberJoin: ip-10-185-65-7 10.185.65.7\r\n    2015/09/01 17:51:01 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:02 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:09 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:51:09 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:43 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:51:45 [INFO] memberlist: Marking ip-10-170-15-71 as failed, suspect timeout reached\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberFailed: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:51:45 [INFO] serf: EventMemberJoin: ip-10-170-15-71 10.170.15.71\r\n    2015/09/01 17:52:22 [INFO] memberlist: Marking ip-10-190-82-4 as failed, suspect timeout reached\r\n    2015/09/01 17:52:22 [INFO] serf: EventMemberFailed: ip-10-190-82-4 10.190.82.4\r\n    2015/09/01 17:52:30 [INFO] serf: EventMemberJoin: ip-10-190-82-4 10.190.82.4\r\n```"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/141231162","html_url":"https://github.com/hashicorp/consul/issues/1212#issuecomment-141231162","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1212","id":141231162,"user":{"login":"djenriquez","id":4636740,"avatar_url":"https://avatars.githubusercontent.com/u/4636740?v=3","gravatar_id":"","url":"https://api.github.com/users/djenriquez","html_url":"https://github.com/djenriquez","followers_url":"https://api.github.com/users/djenriquez/followers","following_url":"https://api.github.com/users/djenriquez/following{/other_user}","gists_url":"https://api.github.com/users/djenriquez/gists{/gist_id}","starred_url":"https://api.github.com/users/djenriquez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djenriquez/subscriptions","organizations_url":"https://api.github.com/users/djenriquez/orgs","repos_url":"https://api.github.com/users/djenriquez/repos","events_url":"https://api.github.com/users/djenriquez/events{/privacy}","received_events_url":"https://api.github.com/users/djenriquez/received_events","type":"User","site_admin":false},"created_at":"2015-09-17T21:20:00Z","updated_at":"2015-09-17T21:20:00Z","body":"Awesome, glad to see its working better for you @jsternberg. Unfortunately in our case, its not a single node but a completely random node that will fail for a short period of time, including nodes in the same VPC as the consul servers. We have all traffic set up to pass through the required consul ports across all servers.\r\n\r\nAt first I had thought there was a VPN issue between our regions, but it wouldn't answer why the nodes in the same region and VPC as the consul servers would also flap periodically.\r\n\r\nI haven't spent much time analyzing the data because the issue is minor, just more of an annoyance. I'll go ahead and start looking deeper at this issue."}}