{"action":"opened","number":164,"pull_request":{"url":"https://api.github.com/repos/Metaswitch/chef/pulls/164","id":32979225,"html_url":"https://github.com/Metaswitch/chef/pull/164","diff_url":"https://github.com/Metaswitch/chef/pull/164.diff","patch_url":"https://github.com/Metaswitch/chef/pull/164.patch","issue_url":"https://api.github.com/repos/Metaswitch/chef/issues/164","number":164,"state":"open","locked":false,"title":"[Reviewer: Ellie] Orchestrate fast Chronos scaling","user":{"login":"rkd-msw","id":4363492,"avatar_url":"https://avatars.githubusercontent.com/u/4363492?v=3","gravatar_id":"","url":"https://api.github.com/users/rkd-msw","html_url":"https://github.com/rkd-msw","followers_url":"https://api.github.com/users/rkd-msw/followers","following_url":"https://api.github.com/users/rkd-msw/following{/other_user}","gists_url":"https://api.github.com/users/rkd-msw/gists{/gist_id}","starred_url":"https://api.github.com/users/rkd-msw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rkd-msw/subscriptions","organizations_url":"https://api.github.com/users/rkd-msw/orgs","repos_url":"https://api.github.com/users/rkd-msw/repos","events_url":"https://api.github.com/users/rkd-msw/events{/privacy}","received_events_url":"https://api.github.com/users/rkd-msw/received_events","type":"User","site_admin":false},"body":"I've tested by:\r\n\r\n* creating a deployment with two Sprouts and two Ralfs\r\n* scaling up to four of each\r\n* scaling back down to two of each\r\n\r\nAfter scaling up, I checked for logs showing scale-up:\r\n\r\n```\r\n09-04-2015 16:44:40.897 UTC Debug httpconnection.cpp:607: Sending HTTP request : http://10.141.201.79:7253/timers?requesting-node=10.4.93.100;sync-mode=SCALE;cluster-view-id=2747321667500416022 (trying 10.141.201.79) on new connection\r\n09-04-2015 16:44:40.900 UTC Debug httpconnection.cpp:622: Received HTTP response: status=200, doc={\"Timers\":[]}\r\n09-04-2015 16:44:40.900 UTC Debug chronos_internal_connection.cpp:123: Finished scaling operation\r\n09-04-2015 16:44:40.900 UTC Debug statistic.cpp:103: Send new value for statistic chronos_scale_nodes_to_query, size 1\r\n```\r\n\r\nWhile scaling down, I checked in the Chef output that 'leaving' nodes were being added:\r\n\r\n```\r\nec2-54-196-209-80.compute-1.amazonaws.com          --- /etc/chronos/chronos.conf        2015-04-09 16:53:21.917403430 +0000\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          +++ /tmp/chef-rendered-template20150409-18009-472n7y 2015-04-09 16:53:22.893384995 +0000\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          @@ -4,7 +4,10 @@\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com           [cluster]\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com           localhost = 10.236.202.26\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          -node = 10.236.202.26\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          +node = 10.143.165.54\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          +leaving = 10.236.202.26\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          +leaving = 10.236.151.79\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com          +leaving = 10.73.129.16\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\nec2-54-196-209-80.compute-1.amazonaws.com\r\n```\r\n\r\nAfter scaling down, I checked that the 'leaving' lines had been removed from /etc/chronos/chronos.conf.\r\n\r\nI did see one oddity - ralf-4 wouldn't complete wait-sync, because `/usr/share/clearwater/bin/cw_stat chronos chronos_scale_nodes_to_query` just hung forever (though Chronos was running). Restarting the Chronos process fixed it, but there were no useful diagnostics. I don't think that's a bug in the Chef side of things, though, so shouldn't hold up this review. (I'm going to re-run scale-down to see if it repros.)","created_at":"2015-04-09T17:48:56Z","updated_at":"2015-04-09T17:48:57Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Metaswitch/chef/pulls/164/commits","review_comments_url":"https://api.github.com/repos/Metaswitch/chef/pulls/164/comments","review_comment_url":"https://api.github.com/repos/Metaswitch/chef/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Metaswitch/chef/issues/164/comments","statuses_url":"https://api.github.com/repos/Metaswitch/chef/statuses/a93fd7425fc691ca51a45bc4252115ac0936f244","head":{"label":"Metaswitch:chronos_scaling","ref":"chronos_scaling","sha":"a93fd7425fc691ca51a45bc4252115ac0936f244","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":9885464,"name":"chef","full_name":"Metaswitch/chef","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/chef","description":"Repo to hold chef recipes for Clearwater deployment","fork":false,"url":"https://api.github.com/repos/Metaswitch/chef","forks_url":"https://api.github.com/repos/Metaswitch/chef/forks","keys_url":"https://api.github.com/repos/Metaswitch/chef/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/chef/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/chef/teams","hooks_url":"https://api.github.com/repos/Metaswitch/chef/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/chef/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/chef/events","assignees_url":"https://api.github.com/repos/Metaswitch/chef/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/chef/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/chef/tags","blobs_url":"https://api.github.com/repos/Metaswitch/chef/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/chef/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/chef/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/chef/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/chef/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/chef/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/chef/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/chef/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/chef/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/chef/subscription","commits_url":"https://api.github.com/repos/Metaswitch/chef/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/chef/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/chef/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/chef/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/chef/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/chef/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/chef/merges","archive_url":"https://api.github.com/repos/Metaswitch/chef/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/chef/downloads","issues_url":"https://api.github.com/repos/Metaswitch/chef/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/chef/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/chef/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/chef/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/chef/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/chef/releases{/id}","created_at":"2013-05-06T11:34:36Z","updated_at":"2015-03-31T14:44:04Z","pushed_at":"2015-04-09T17:41:12Z","git_url":"git://github.com/Metaswitch/chef.git","ssh_url":"git@github.com:Metaswitch/chef.git","clone_url":"https://github.com/Metaswitch/chef.git","svn_url":"https://github.com/Metaswitch/chef","homepage":null,"size":2369,"stargazers_count":2,"watchers_count":2,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":11,"forks":3,"open_issues":11,"watchers":2,"default_branch":"master"}},"base":{"label":"Metaswitch:master","ref":"master","sha":"9e0cb1c0dc2f5d8a68fb53ed4c83add4734e52ec","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":9885464,"name":"chef","full_name":"Metaswitch/chef","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/chef","description":"Repo to hold chef recipes for Clearwater deployment","fork":false,"url":"https://api.github.com/repos/Metaswitch/chef","forks_url":"https://api.github.com/repos/Metaswitch/chef/forks","keys_url":"https://api.github.com/repos/Metaswitch/chef/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/chef/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/chef/teams","hooks_url":"https://api.github.com/repos/Metaswitch/chef/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/chef/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/chef/events","assignees_url":"https://api.github.com/repos/Metaswitch/chef/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/chef/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/chef/tags","blobs_url":"https://api.github.com/repos/Metaswitch/chef/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/chef/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/chef/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/chef/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/chef/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/chef/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/chef/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/chef/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/chef/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/chef/subscription","commits_url":"https://api.github.com/repos/Metaswitch/chef/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/chef/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/chef/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/chef/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/chef/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/chef/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/chef/merges","archive_url":"https://api.github.com/repos/Metaswitch/chef/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/chef/downloads","issues_url":"https://api.github.com/repos/Metaswitch/chef/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/chef/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/chef/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/chef/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/chef/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/chef/releases{/id}","created_at":"2013-05-06T11:34:36Z","updated_at":"2015-03-31T14:44:04Z","pushed_at":"2015-04-09T17:41:12Z","git_url":"git://github.com/Metaswitch/chef.git","ssh_url":"git@github.com:Metaswitch/chef.git","clone_url":"https://github.com/Metaswitch/chef.git","svn_url":"https://github.com/Metaswitch/chef","homepage":null,"size":2369,"stargazers_count":2,"watchers_count":2,"language":"Ruby","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":11,"forks":3,"open_issues":11,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Metaswitch/chef/pulls/164"},"html":{"href":"https://github.com/Metaswitch/chef/pull/164"},"issue":{"href":"https://api.github.com/repos/Metaswitch/chef/issues/164"},"comments":{"href":"https://api.github.com/repos/Metaswitch/chef/issues/164/comments"},"review_comments":{"href":"https://api.github.com/repos/Metaswitch/chef/pulls/164/comments"},"review_comment":{"href":"https://api.github.com/repos/Metaswitch/chef/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Metaswitch/chef/pulls/164/commits"},"statuses":{"href":"https://api.github.com/repos/Metaswitch/chef/statuses/a93fd7425fc691ca51a45bc4252115ac0936f244"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":1,"additions":34,"deletions":4,"changed_files":6}}