{"action":"closed","number":6,"pull_request":{"url":"https://api.github.com/repos/Metaswitch/astaire/pulls/6","id":32850833,"html_url":"https://github.com/Metaswitch/astaire/pull/6","diff_url":"https://github.com/Metaswitch/astaire/pull/6.diff","patch_url":"https://github.com/Metaswitch/astaire/pull/6.patch","issue_url":"https://api.github.com/repos/Metaswitch/astaire/issues/6","number":6,"state":"closed","locked":false,"title":"[Reviewer: Andy] Proactively restore memcached redundancy","user":{"login":"AlexHockey","id":4903205,"avatar_url":"https://avatars.githubusercontent.com/u/4903205?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexHockey","html_url":"https://github.com/AlexHockey","followers_url":"https://api.github.com/users/AlexHockey/followers","following_url":"https://api.github.com/users/AlexHockey/following{/other_user}","gists_url":"https://api.github.com/users/AlexHockey/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexHockey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexHockey/subscriptions","organizations_url":"https://api.github.com/users/AlexHockey/orgs","repos_url":"https://api.github.com/users/AlexHockey/repos","events_url":"https://api.github.com/users/AlexHockey/events{/privacy}","received_events_url":"https://api.github.com/users/AlexHockey/received_events","type":"User","site_admin":false},"body":"Please can you review this PR to enhance Astaire to pro-actively restore redundancy when a memcached server restarts. This also fixes some edge cases where data could be lost on a scaling operation if one of the current servers had recently restarted. \r\n\r\nThe code should match the design we discussed. Specifically I have:\r\n\r\n* Refactored the code to use a new \"control thread\" that periodically polls the local memcached server to see if it has restarted, and can also be kicked by the signal handlers.\r\n* Changed the work-list processing so that Astaire streams vbuckets from all replicas, not just one of them. The resync is only considered as failed if a vbucket could not be streamed from any replica. \r\n* Added code to tag (and un-tag) the memcached server as being up-to-date. \r\n* Modified the ENT log test to reflect that resync is not always caused by scaling. The alarm text was already general enough that no change was needed to https://github.com/Metaswitch/cpp-common/blob/master/src/alarmdefinition.cpp.\r\n\r\nI have not tested this at all yet. Astaire has no UTs, and I wanted to get it reviewed before I start doing non-regressible live testing. ","created_at":"2015-04-08T11:28:19Z","updated_at":"2015-04-10T15:28:37Z","closed_at":"2015-04-10T15:28:37Z","merged_at":"2015-04-10T15:28:37Z","merge_commit_sha":"58f4708d834af5c96e2e16e72278dd5a04e50794","assignee":{"login":"bossmc","id":1073983,"avatar_url":"https://avatars.githubusercontent.com/u/1073983?v=3","gravatar_id":"","url":"https://api.github.com/users/bossmc","html_url":"https://github.com/bossmc","followers_url":"https://api.github.com/users/bossmc/followers","following_url":"https://api.github.com/users/bossmc/following{/other_user}","gists_url":"https://api.github.com/users/bossmc/gists{/gist_id}","starred_url":"https://api.github.com/users/bossmc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bossmc/subscriptions","organizations_url":"https://api.github.com/users/bossmc/orgs","repos_url":"https://api.github.com/users/bossmc/repos","events_url":"https://api.github.com/users/bossmc/events{/privacy}","received_events_url":"https://api.github.com/users/bossmc/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/Metaswitch/astaire/pulls/6/commits","review_comments_url":"https://api.github.com/repos/Metaswitch/astaire/pulls/6/comments","review_comment_url":"https://api.github.com/repos/Metaswitch/astaire/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Metaswitch/astaire/issues/6/comments","statuses_url":"https://api.github.com/repos/Metaswitch/astaire/statuses/272c978de119f14d7b9db8c863d0373f9fc1cbee","head":{"label":"Metaswitch:sto1162","ref":"sto1162","sha":"272c978de119f14d7b9db8c863d0373f9fc1cbee","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":32105381,"name":"astaire","full_name":"Metaswitch/astaire","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/astaire","description":"Astaire, an active resynchronization tool for Memcached clusters","fork":false,"url":"https://api.github.com/repos/Metaswitch/astaire","forks_url":"https://api.github.com/repos/Metaswitch/astaire/forks","keys_url":"https://api.github.com/repos/Metaswitch/astaire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/astaire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/astaire/teams","hooks_url":"https://api.github.com/repos/Metaswitch/astaire/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/astaire/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/astaire/events","assignees_url":"https://api.github.com/repos/Metaswitch/astaire/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/astaire/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/astaire/tags","blobs_url":"https://api.github.com/repos/Metaswitch/astaire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/astaire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/astaire/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/astaire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/astaire/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/astaire/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/astaire/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/astaire/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/astaire/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/astaire/subscription","commits_url":"https://api.github.com/repos/Metaswitch/astaire/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/astaire/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/astaire/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/astaire/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/astaire/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/astaire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/astaire/merges","archive_url":"https://api.github.com/repos/Metaswitch/astaire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/astaire/downloads","issues_url":"https://api.github.com/repos/Metaswitch/astaire/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/astaire/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/astaire/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/astaire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/astaire/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/astaire/releases{/id}","created_at":"2015-03-12T22:05:49Z","updated_at":"2015-04-07T11:06:22Z","pushed_at":"2015-04-10T15:28:37Z","git_url":"git://github.com/Metaswitch/astaire.git","ssh_url":"git@github.com:Metaswitch/astaire.git","clone_url":"https://github.com/Metaswitch/astaire.git","svn_url":"https://github.com/Metaswitch/astaire","homepage":null,"size":434,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"dev"}},"base":{"label":"Metaswitch:dev","ref":"dev","sha":"7334075002744096289b47f5053c60db867c07af","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":32105381,"name":"astaire","full_name":"Metaswitch/astaire","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/astaire","description":"Astaire, an active resynchronization tool for Memcached clusters","fork":false,"url":"https://api.github.com/repos/Metaswitch/astaire","forks_url":"https://api.github.com/repos/Metaswitch/astaire/forks","keys_url":"https://api.github.com/repos/Metaswitch/astaire/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/astaire/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/astaire/teams","hooks_url":"https://api.github.com/repos/Metaswitch/astaire/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/astaire/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/astaire/events","assignees_url":"https://api.github.com/repos/Metaswitch/astaire/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/astaire/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/astaire/tags","blobs_url":"https://api.github.com/repos/Metaswitch/astaire/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/astaire/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/astaire/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/astaire/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/astaire/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/astaire/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/astaire/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/astaire/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/astaire/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/astaire/subscription","commits_url":"https://api.github.com/repos/Metaswitch/astaire/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/astaire/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/astaire/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/astaire/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/astaire/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/astaire/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/astaire/merges","archive_url":"https://api.github.com/repos/Metaswitch/astaire/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/astaire/downloads","issues_url":"https://api.github.com/repos/Metaswitch/astaire/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/astaire/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/astaire/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/astaire/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/astaire/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/astaire/releases{/id}","created_at":"2015-03-12T22:05:49Z","updated_at":"2015-04-07T11:06:22Z","pushed_at":"2015-04-10T15:28:37Z","git_url":"git://github.com/Metaswitch/astaire.git","ssh_url":"git@github.com:Metaswitch/astaire.git","clone_url":"https://github.com/Metaswitch/astaire.git","svn_url":"https://github.com/Metaswitch/astaire","homepage":null,"size":434,"stargazers_count":0,"watchers_count":0,"language":"C++","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":1,"forks":1,"open_issues":1,"watchers":0,"default_branch":"dev"}},"_links":{"self":{"href":"https://api.github.com/repos/Metaswitch/astaire/pulls/6"},"html":{"href":"https://github.com/Metaswitch/astaire/pull/6"},"issue":{"href":"https://api.github.com/repos/Metaswitch/astaire/issues/6"},"comments":{"href":"https://api.github.com/repos/Metaswitch/astaire/issues/6/comments"},"review_comments":{"href":"https://api.github.com/repos/Metaswitch/astaire/pulls/6/comments"},"review_comment":{"href":"https://api.github.com/repos/Metaswitch/astaire/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Metaswitch/astaire/pulls/6/commits"},"statuses":{"href":"https://api.github.com/repos/Metaswitch/astaire/statuses/272c978de119f14d7b9db8c863d0373f9fc1cbee"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"AlexHockey","id":4903205,"avatar_url":"https://avatars.githubusercontent.com/u/4903205?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexHockey","html_url":"https://github.com/AlexHockey","followers_url":"https://api.github.com/users/AlexHockey/followers","following_url":"https://api.github.com/users/AlexHockey/following{/other_user}","gists_url":"https://api.github.com/users/AlexHockey/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexHockey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexHockey/subscriptions","organizations_url":"https://api.github.com/users/AlexHockey/orgs","repos_url":"https://api.github.com/users/AlexHockey/repos","events_url":"https://api.github.com/users/AlexHockey/events{/privacy}","received_events_url":"https://api.github.com/users/AlexHockey/received_events","type":"User","site_admin":false},"comments":3,"review_comments":3,"commits":25,"additions":572,"deletions":107,"changed_files":6}}