{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/916","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/916/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/916/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/916/events","html_url":"https://github.com/hashicorp/consul/issues/916","id":74059948,"number":916,"title":"Frequent membership loss for 50-100 node cluster in AWS","user":{"login":"Amit-PivotalLabs","id":3108318,"avatar_url":"https://avatars.githubusercontent.com/u/3108318?v=3","gravatar_id":"","url":"https://api.github.com/users/Amit-PivotalLabs","html_url":"https://github.com/Amit-PivotalLabs","followers_url":"https://api.github.com/users/Amit-PivotalLabs/followers","following_url":"https://api.github.com/users/Amit-PivotalLabs/following{/other_user}","gists_url":"https://api.github.com/users/Amit-PivotalLabs/gists{/gist_id}","starred_url":"https://api.github.com/users/Amit-PivotalLabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Amit-PivotalLabs/subscriptions","organizations_url":"https://api.github.com/users/Amit-PivotalLabs/orgs","repos_url":"https://api.github.com/users/Amit-PivotalLabs/repos","events_url":"https://api.github.com/users/Amit-PivotalLabs/events{/privacy}","received_events_url":"https://api.github.com/users/Amit-PivotalLabs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-05-07T17:17:14Z","updated_at":"2015-11-03T15:28:07Z","closed_at":null,"body":"The main problem we're seeing is we have KV data associated with a lock, and hence a session, and the agents associated with those sessions \"frequently\" lose membership briefly, invalidating the associated KV data.  By \"frequently\", I mean that in a cluster consisting of 3 consul servers and 50-100 other nodes running consul agents, we see individual isolated incidents of agent membership lost every 3-4 hours, roughly (sometimes ~30 mins apart, sometimes 7-8 hours apart, but generally seems to be randomly distributed with an \"average\" of 3-4).\r\n\r\nWe have a heterogeneous cluster configured as a LAN, deployed across 3 AZs in EC2.  It appears that there is no correlation between the type of node and the loss of membership (e.g. if we have 10 instances of service A, and 50 of service B, we will see roughly 2 membership losses of A instances, and 10 instances of B losses in a period of a few days).  It also appears to be independent of stress on the system, i.e. we see roughly the same distribution of membership loss when the system is idle as we do when it's under heavy load.\r\n\r\nWe use consul for service discovery/DNS as well as a KV store.  We use the KV store specifically for the purpose of:\r\n\r\n* maintaining presence -- certain nodes maintain the presence of a key with a TTL on it to indicate they are up and running, and when those disappear another node can react and re-schedule the workload that was being performed by that node, and\r\n* locks -- for multiple instances of the same service to determine which one actually provides the service, in the case where the service needs to be a singleton\r\n\r\nIn addition to the occasional member loss, we also have the issue that when we roll the consul server cluster, which triggers a leader election, the KV store is temporarily unavailable. This potentially prevents a presence maintainer from bumping its lock before the TTL expires.\r\n\r\n**Questions**:\r\n1. Are there ways to configure things so that membership is more robust/more forgiving?  We've seen that WAN has more conservative parameters, e.g. a larger SuspicionMult(iplicationFactor), but switching to WAN feels like it would just be papering over an issue, and would only reduce the frequency of this issue.\r\n2. Is there any advice for a better way to implement \"presence maintenance\"?  Because we use locks, and hence sessions, we're sensitive to the lossiness of UDP and probabilistic nature of Gossip, even though the running application maintaining its presence in the KV stores is running fine.  E.g. is there a way to establish a lock but side-step the coupling to the membership of its agent?\r\n3. What information would help diagnose this problem?  We have plenty of server and agent configuration, consul server logs, consul agent logs, and application logs from services running alongside consul agents, all spanning several days.\r\n\r\nThanks,\r\nAmit + @matt-royal"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/153388433","html_url":"https://github.com/hashicorp/consul/issues/916#issuecomment-153388433","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/916","id":153388433,"user":{"login":"zonzamas","id":1359495,"avatar_url":"https://avatars.githubusercontent.com/u/1359495?v=3","gravatar_id":"","url":"https://api.github.com/users/zonzamas","html_url":"https://github.com/zonzamas","followers_url":"https://api.github.com/users/zonzamas/followers","following_url":"https://api.github.com/users/zonzamas/following{/other_user}","gists_url":"https://api.github.com/users/zonzamas/gists{/gist_id}","starred_url":"https://api.github.com/users/zonzamas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zonzamas/subscriptions","organizations_url":"https://api.github.com/users/zonzamas/orgs","repos_url":"https://api.github.com/users/zonzamas/repos","events_url":"https://api.github.com/users/zonzamas/events{/privacy}","received_events_url":"https://api.github.com/users/zonzamas/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T15:28:07Z","updated_at":"2015-11-03T15:28:07Z","body":"Just wanted to say that we are having the same issue in a VPC network, with around 35 consul members  (5 leaders). From time to time some of the nodes leaves the cluster for a brief time (a few seconds) and that triggers consul-template to reload some services."}}