{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/911","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/911/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/911/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/911/events","html_url":"https://github.com/hashicorp/consul/issues/911","id":73587343,"number":911,"title":"Why the manual steps and shutdown of the whole cluster on outage","user":{"login":"kaelumania","id":4765210,"avatar_url":"https://avatars.githubusercontent.com/u/4765210?v=3","gravatar_id":"","url":"https://api.github.com/users/kaelumania","html_url":"https://github.com/kaelumania","followers_url":"https://api.github.com/users/kaelumania/followers","following_url":"https://api.github.com/users/kaelumania/following{/other_user}","gists_url":"https://api.github.com/users/kaelumania/gists{/gist_id}","starred_url":"https://api.github.com/users/kaelumania/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kaelumania/subscriptions","organizations_url":"https://api.github.com/users/kaelumania/orgs","repos_url":"https://api.github.com/users/kaelumania/repos","events_url":"https://api.github.com/users/kaelumania/events{/privacy}","received_events_url":"https://api.github.com/users/kaelumania/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-05-06T10:45:49Z","updated_at":"2015-05-06T22:10:05Z","closed_at":"2015-05-06T18:19:24Z","body":"The consul guide (https://www.consul.io/docs/guides/outage.html) for outages says that one has to manually remove the failed server from all `raft/peers.json` files. Why is that? Further in order to change that file one has to shutdown all server nodes, which collapses the whole cluster. I think this is not a very cluster like behaviour and would make the whole cluster unavailable. Why isn't there a command to just ignore/remove the failed server from the other servers? Or why does consul not automatically remove the failed server as it also removes failed members after some time?\r\n\r\nIn our case, we have very small clusters (5-10) and every node is a server, because we don't want the overhead of having 3 extra consul servers for a cluster of 5 nodes. Thus when one node fails, our whole cluster would need manual reconfiguration."},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/99625468","html_url":"https://github.com/hashicorp/consul/issues/911#issuecomment-99625468","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/911","id":99625468,"user":{"login":"armon","id":592032,"avatar_url":"https://avatars.githubusercontent.com/u/592032?v=3","gravatar_id":"","url":"https://api.github.com/users/armon","html_url":"https://github.com/armon","followers_url":"https://api.github.com/users/armon/followers","following_url":"https://api.github.com/users/armon/following{/other_user}","gists_url":"https://api.github.com/users/armon/gists{/gist_id}","starred_url":"https://api.github.com/users/armon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/armon/subscriptions","organizations_url":"https://api.github.com/users/armon/orgs","repos_url":"https://api.github.com/users/armon/repos","events_url":"https://api.github.com/users/armon/events{/privacy}","received_events_url":"https://api.github.com/users/armon/received_events","type":"User","site_admin":false},"created_at":"2015-05-06T22:10:05Z","updated_at":"2015-05-06T22:10:05Z","body":"@kaelumania The explanation by @highlyunavailable is almost perfect. Watches actually aren't a special case, and are just like normal reads and even will respect the consistency modes. Raft is implemented like the paper, and the Serf/Gossip data is not used for replication.\r\n\r\nIn the \"default\" read mode, all reads are serviced by the leader. The \"consistent\" read mode forces an additional heartbeat round trip on read. The \"stale\" mode allows any server to service the read, and thus does load shedding from the leader.\r\n\r\nHope that helps!"}}