{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/174","labels_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/174/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/174/comments","events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/174/events","html_url":"https://github.com/Metaswitch/clearwater-etcd/issues/174","id":104666001,"number":174,"title":"mark_node_failed doesn't cope with multiple failures","user":{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels/critical","name":"critical","color":"ff0000"},{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels/high-priority","name":"high-priority","color":"ff0000"}],"state":"closed","locked":false,"assignee":{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false},"milestone":null,"comments":4,"created_at":"2015-09-03T10:42:15Z","updated_at":"2015-09-29T15:59:33Z","closed_at":"2015-09-29T15:59:33Z","body":"If multiple nodes have failed in a cluster then mark_node_failed can't recover it.\r\n\r\nThe scenario was that I had a sprout memcached cluster with 2 initial nodes. For various reasons these nodes failed abruptly, and I span up two new Sprout nodes. This then did bad things to the sprout/memento chronos/memcached/cassandra clusters.\r\n\r\nTaking the sprout memcached cluster as an example then the cluster state was:\r\nOld node 1 - normal\r\nOld node 2 -  normal, acknowledged change\r\nNew node 1 - joining, acknowledged change\r\nNew node 2 - not in cluster at all\r\n\r\nThe cluster state will only allow nodes to be removed if the cluster as a whole is stable (or stable with errors - https://github.com/Metaswitch/clearwater-etcd/blob/master/src/metaswitch/clearwater/cluster_manager/cluster_state.py#L58). This wasn't the case in my setup (and couldn't become true).\r\n\r\nAlso, writing the error state to etcd (https://github.com/Metaswitch/clearwater-etcd/blob/master/src/metaswitch/clearwater/cluster_manager/etcd_synchronizer.py#L134) invariably fails because the index is wrong (so it catches the Key already Exists error), and there's no code to retry this. The actual behaviour on the node's state is quite weird - I'd mark one node that was stuck in a joining state as in the error state, and the effect would be to move it into the normal state"},"comment":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/comments/144103062","html_url":"https://github.com/Metaswitch/clearwater-etcd/issues/174#issuecomment-144103062","issue_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/174","id":144103062,"user":{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false},"created_at":"2015-09-29T15:59:33Z","updated_at":"2015-09-29T15:59:33Z","body":"Fixed in https://github.com/Metaswitch/clearwater-etcd/commit/1ce393334cb6f06d4c4aa5a576fc6b713d1e69a3, https://github.com/Metaswitch/clearwater-etcd/commit/d10bf6ff3ee59e51fc1cd28ef7288aec37fd5db2"}}