{"action":"closed","number":1187,"pull_request":{"url":"https://api.github.com/repos/rackerlabs/otter/pulls/1187","id":30917426,"html_url":"https://github.com/rackerlabs/otter/pull/1187","diff_url":"https://github.com/rackerlabs/otter/pull/1187.diff","patch_url":"https://github.com/rackerlabs/otter/pull/1187.patch","issue_url":"https://api.github.com/repos/rackerlabs/otter/issues/1187","number":1187,"state":"closed","locked":false,"title":"Keep track of divergent groups in ZooKeeper","user":{"login":"radix","id":227068,"avatar_url":"https://avatars.githubusercontent.com/u/227068?v=3","gravatar_id":"","url":"https://api.github.com/users/radix","html_url":"https://github.com/radix","followers_url":"https://api.github.com/users/radix/followers","following_url":"https://api.github.com/users/radix/following{/other_user}","gists_url":"https://api.github.com/users/radix/gists{/gist_id}","starred_url":"https://api.github.com/users/radix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/radix/subscriptions","organizations_url":"https://api.github.com/users/radix/orgs","repos_url":"https://api.github.com/users/radix/repos","events_url":"https://api.github.com/users/radix/events{/privacy}","received_events_url":"https://api.github.com/users/radix/received_events","type":"User","site_admin":false},"body":"This fixes #971.\r\n\r\nThe main benefit of this PR is that it finally allows us to survive a restart while we're in the middle of performing actions on behalf of the user. The main changes are:\r\n\r\n- the `Converger` service has been split up into two parts: `ConvergenceStarter`, which is what the controller calls `start_convergence` on when it executes a policy or obeys a config change, and `Converger` which actually does the work of convergence (potentially on a different node than the API node where `start_convergence` was invoked).\r\n- The only thing `ConvergenceStarter` does is create (or update) a ZNode in ZooKeeper, at a path like `/groups/divergent/{tenant_id}_{group_id}`.\r\n-  `Converger` uses a ZooKeeper set partitioner to divvy up the work of converging the set of divergent groups between all servers running a Converger.\r\n- We converge all groups every 10 seconds (as long as we're not already converging them).\r\n- When the converger is done converging a group, it deletes the ZNode which indicates it's divergent *only if its version has not changed*. This is an important behavior that allows us to avoid a race condition where a second policy execution happens *just* as an existing convergence finishes up. This behavior ensures that the second policy execution will cause another convergence. Thanks to @cyli for coming up with this elegant design :)\r\n\r\nNote to reviewers: I recommend jumping straight to ConvergenceStarter and Converger, reading them (they're very small) and then following the chain of dependent functions down from there, in a top-down fashion. (For example, `Converger.buckets_acquired` calls `converge_all` calls `converge_one` calls `execute_convergence`). I think it'll help to review it in that order so you understand the context that each function is used in.\r\n","created_at":"2015-03-11T05:58:23Z","updated_at":"2015-03-19T17:54:36Z","closed_at":"2015-03-19T17:54:36Z","merged_at":"2015-03-19T17:54:36Z","merge_commit_sha":"c299c7228ef902461659d3e3646ffe51cbb90a4d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/rackerlabs/otter/pulls/1187/commits","review_comments_url":"https://api.github.com/repos/rackerlabs/otter/pulls/1187/comments","review_comment_url":"https://api.github.com/repos/rackerlabs/otter/pulls/comments{/number}","comments_url":"https://api.github.com/repos/rackerlabs/otter/issues/1187/comments","statuses_url":"https://api.github.com/repos/rackerlabs/otter/statuses/d078591b3a30aa17cc168831a3bdc2540d1aebbe","head":{"label":"rackerlabs:dirty-groups-3","ref":"dirty-groups-3","sha":"d078591b3a30aa17cc168831a3bdc2540d1aebbe","user":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"repo":{"id":6097139,"name":"otter","full_name":"rackerlabs/otter","owner":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/rackerlabs/otter","description":"Rackspace Auto Scale","fork":false,"url":"https://api.github.com/repos/rackerlabs/otter","forks_url":"https://api.github.com/repos/rackerlabs/otter/forks","keys_url":"https://api.github.com/repos/rackerlabs/otter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rackerlabs/otter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rackerlabs/otter/teams","hooks_url":"https://api.github.com/repos/rackerlabs/otter/hooks","issue_events_url":"https://api.github.com/repos/rackerlabs/otter/issues/events{/number}","events_url":"https://api.github.com/repos/rackerlabs/otter/events","assignees_url":"https://api.github.com/repos/rackerlabs/otter/assignees{/user}","branches_url":"https://api.github.com/repos/rackerlabs/otter/branches{/branch}","tags_url":"https://api.github.com/repos/rackerlabs/otter/tags","blobs_url":"https://api.github.com/repos/rackerlabs/otter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rackerlabs/otter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rackerlabs/otter/git/refs{/sha}","trees_url":"https://api.github.com/repos/rackerlabs/otter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rackerlabs/otter/statuses/{sha}","languages_url":"https://api.github.com/repos/rackerlabs/otter/languages","stargazers_url":"https://api.github.com/repos/rackerlabs/otter/stargazers","contributors_url":"https://api.github.com/repos/rackerlabs/otter/contributors","subscribers_url":"https://api.github.com/repos/rackerlabs/otter/subscribers","subscription_url":"https://api.github.com/repos/rackerlabs/otter/subscription","commits_url":"https://api.github.com/repos/rackerlabs/otter/commits{/sha}","git_commits_url":"https://api.github.com/repos/rackerlabs/otter/git/commits{/sha}","comments_url":"https://api.github.com/repos/rackerlabs/otter/comments{/number}","issue_comment_url":"https://api.github.com/repos/rackerlabs/otter/issues/comments{/number}","contents_url":"https://api.github.com/repos/rackerlabs/otter/contents/{+path}","compare_url":"https://api.github.com/repos/rackerlabs/otter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rackerlabs/otter/merges","archive_url":"https://api.github.com/repos/rackerlabs/otter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rackerlabs/otter/downloads","issues_url":"https://api.github.com/repos/rackerlabs/otter/issues{/number}","pulls_url":"https://api.github.com/repos/rackerlabs/otter/pulls{/number}","milestones_url":"https://api.github.com/repos/rackerlabs/otter/milestones{/number}","notifications_url":"https://api.github.com/repos/rackerlabs/otter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rackerlabs/otter/labels{/name}","releases_url":"https://api.github.com/repos/rackerlabs/otter/releases{/id}","created_at":"2012-10-05T21:35:59Z","updated_at":"2015-03-18T21:34:21Z","pushed_at":"2015-03-19T17:54:36Z","git_url":"git://github.com/rackerlabs/otter.git","ssh_url":"git@github.com:rackerlabs/otter.git","clone_url":"https://github.com/rackerlabs/otter.git","svn_url":"https://github.com/rackerlabs/otter","homepage":"http://www.rackspace.com/cloud/auto-scale/","size":43082,"stargazers_count":43,"watchers_count":43,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":17,"mirror_url":null,"open_issues_count":80,"forks":17,"open_issues":80,"watchers":43,"default_branch":"master"}},"base":{"label":"rackerlabs:master","ref":"master","sha":"312be5bf9f3ed5b3572e018e03ac6bc450b5f294","user":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"repo":{"id":6097139,"name":"otter","full_name":"rackerlabs/otter","owner":{"login":"rackerlabs","id":84293,"avatar_url":"https://avatars.githubusercontent.com/u/84293?v=3","gravatar_id":"","url":"https://api.github.com/users/rackerlabs","html_url":"https://github.com/rackerlabs","followers_url":"https://api.github.com/users/rackerlabs/followers","following_url":"https://api.github.com/users/rackerlabs/following{/other_user}","gists_url":"https://api.github.com/users/rackerlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rackerlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rackerlabs/subscriptions","organizations_url":"https://api.github.com/users/rackerlabs/orgs","repos_url":"https://api.github.com/users/rackerlabs/repos","events_url":"https://api.github.com/users/rackerlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rackerlabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/rackerlabs/otter","description":"Rackspace Auto Scale","fork":false,"url":"https://api.github.com/repos/rackerlabs/otter","forks_url":"https://api.github.com/repos/rackerlabs/otter/forks","keys_url":"https://api.github.com/repos/rackerlabs/otter/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rackerlabs/otter/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rackerlabs/otter/teams","hooks_url":"https://api.github.com/repos/rackerlabs/otter/hooks","issue_events_url":"https://api.github.com/repos/rackerlabs/otter/issues/events{/number}","events_url":"https://api.github.com/repos/rackerlabs/otter/events","assignees_url":"https://api.github.com/repos/rackerlabs/otter/assignees{/user}","branches_url":"https://api.github.com/repos/rackerlabs/otter/branches{/branch}","tags_url":"https://api.github.com/repos/rackerlabs/otter/tags","blobs_url":"https://api.github.com/repos/rackerlabs/otter/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rackerlabs/otter/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rackerlabs/otter/git/refs{/sha}","trees_url":"https://api.github.com/repos/rackerlabs/otter/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rackerlabs/otter/statuses/{sha}","languages_url":"https://api.github.com/repos/rackerlabs/otter/languages","stargazers_url":"https://api.github.com/repos/rackerlabs/otter/stargazers","contributors_url":"https://api.github.com/repos/rackerlabs/otter/contributors","subscribers_url":"https://api.github.com/repos/rackerlabs/otter/subscribers","subscription_url":"https://api.github.com/repos/rackerlabs/otter/subscription","commits_url":"https://api.github.com/repos/rackerlabs/otter/commits{/sha}","git_commits_url":"https://api.github.com/repos/rackerlabs/otter/git/commits{/sha}","comments_url":"https://api.github.com/repos/rackerlabs/otter/comments{/number}","issue_comment_url":"https://api.github.com/repos/rackerlabs/otter/issues/comments{/number}","contents_url":"https://api.github.com/repos/rackerlabs/otter/contents/{+path}","compare_url":"https://api.github.com/repos/rackerlabs/otter/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rackerlabs/otter/merges","archive_url":"https://api.github.com/repos/rackerlabs/otter/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rackerlabs/otter/downloads","issues_url":"https://api.github.com/repos/rackerlabs/otter/issues{/number}","pulls_url":"https://api.github.com/repos/rackerlabs/otter/pulls{/number}","milestones_url":"https://api.github.com/repos/rackerlabs/otter/milestones{/number}","notifications_url":"https://api.github.com/repos/rackerlabs/otter/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rackerlabs/otter/labels{/name}","releases_url":"https://api.github.com/repos/rackerlabs/otter/releases{/id}","created_at":"2012-10-05T21:35:59Z","updated_at":"2015-03-18T21:34:21Z","pushed_at":"2015-03-19T17:54:36Z","git_url":"git://github.com/rackerlabs/otter.git","ssh_url":"git@github.com:rackerlabs/otter.git","clone_url":"https://github.com/rackerlabs/otter.git","svn_url":"https://github.com/rackerlabs/otter","homepage":"http://www.rackspace.com/cloud/auto-scale/","size":43082,"stargazers_count":43,"watchers_count":43,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":17,"mirror_url":null,"open_issues_count":80,"forks":17,"open_issues":80,"watchers":43,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/rackerlabs/otter/pulls/1187"},"html":{"href":"https://github.com/rackerlabs/otter/pull/1187"},"issue":{"href":"https://api.github.com/repos/rackerlabs/otter/issues/1187"},"comments":{"href":"https://api.github.com/repos/rackerlabs/otter/issues/1187/comments"},"review_comments":{"href":"https://api.github.com/repos/rackerlabs/otter/pulls/1187/comments"},"review_comment":{"href":"https://api.github.com/repos/rackerlabs/otter/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/rackerlabs/otter/pulls/1187/commits"},"statuses":{"href":"https://api.github.com/repos/rackerlabs/otter/statuses/d078591b3a30aa17cc168831a3bdc2540d1aebbe"}},"merged":true,"mergeable":null,"mergeable_state":"unknown","merged_by":{"login":"radix","id":227068,"avatar_url":"https://avatars.githubusercontent.com/u/227068?v=3","gravatar_id":"","url":"https://api.github.com/users/radix","html_url":"https://github.com/radix","followers_url":"https://api.github.com/users/radix/followers","following_url":"https://api.github.com/users/radix/following{/other_user}","gists_url":"https://api.github.com/users/radix/gists{/gist_id}","starred_url":"https://api.github.com/users/radix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/radix/subscriptions","organizations_url":"https://api.github.com/users/radix/orgs","repos_url":"https://api.github.com/users/radix/repos","events_url":"https://api.github.com/users/radix/events{/privacy}","received_events_url":"https://api.github.com/users/radix/received_events","type":"User","site_admin":false},"comments":6,"review_comments":34,"commits":87,"additions":922,"deletions":209,"changed_files":8}}