{"action":"created","comment":{"url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/comments/45820278","id":45820278,"diff_hunk":"@@ -1,4 +1,64 @@\n module ClouderaManager\n   class Service < BaseResource\n+    class RestartFailedException < BaseException; end\n+\n+    belongs_to :cluster\n+    has_many :roles\n+    primary_key :name\n+    collection_path \"clusters/:cluster_id/services\"\n+    after_find { |s| s.cluster_id = s.clusterRef[:clusterName] }\n+\n+    BULK_RESTART_SAFE_ROLE_TYPES = [\n+      \"IMPALAD\",\n+      \"DATANODE\",\n+      \"NODEMANAGER\"\n+    ]\n+\n+    RESTART_ORDER = {\n+      \"YARN\" => [\"JOBHISTORY\", \"NODEMANAGER\", \"RESOURCEMANAGER\"],\n+      \"HDFS\" => [\"SECONDARYNAMENODE\", \"HTTPFS\", \"FAILOVERCONTROLLER\", \"NFSGATEWAY\", \"DATANODE\", \"NAMENODE\", \"JOURNALNODE\"],","path":"lib/cloudera-manager/service.rb","position":19,"original_position":19,"commit_id":"c731b3007183e730836ab18d9e19e2a3eba7ba60","original_commit_id":"c731b3007183e730836ab18d9e19e2a3eba7ba60","user":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"body":"I was writing for the general case (figure we may as well open source this sucker) and figured that the secondary namenode when not in HA is a good one to restart first, but on second consideration that doesn't really make much sense, I'll do the other stuff first. It is robust to there being no roles of that type for a given service. ","created_at":"2015-11-25T01:34:18Z","updated_at":"2015-11-25T01:34:18Z","html_url":"https://github.com/airhorns/cloudera-manager-client/pull/2#discussion_r45820278","pull_request_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2","_links":{"self":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/comments/45820278"},"html":{"href":"https://github.com/airhorns/cloudera-manager-client/pull/2#discussion_r45820278"},"pull_request":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2"}}},"pull_request":{"url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2","id":51616889,"html_url":"https://github.com/airhorns/cloudera-manager-client/pull/2","diff_url":"https://github.com/airhorns/cloudera-manager-client/pull/2.diff","patch_url":"https://github.com/airhorns/cloudera-manager-client/pull/2.patch","issue_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/2","number":2,"state":"open","locked":false,"title":"WIP Rolling restarts support","user":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"body":"This works on top of #1 to add rolling restart functionality to the client. It:\r\n\r\n - Adds a Service resource\r\n - Adds a BulkCommand abstraction to represent the list of commands returned by the bulk command endpoints in the CM API\r\n - Teaches BulkCommand to report on success ratios and block until the commands are complete by polling with a timeout\r\n - Adds a basic algorithm for executing a rolling restart on a given service\r\n\r\nThe algorithm is this: iterate the list of role types for a service, and then restart the instances of each type in batches, in order. Using the `/roleCommands/restart` endpoint which restarts a batch of role instances and reports on the success, we are able to issue batches of restarts. Each batch must be at least 90% successful for the next one to take place.\r\n\r\nFor services we care deeply about, we hardcode an order to restart the different kinds of service members in from least-scary to most-scary. My reasoning here is that we want to detect any kind of failure early before restarting the masters if possible. If the config is broken or the processes are systematically not going to come back up for any reason, I’d rather find that out on the doinky worker nodes instead of the coordinators for the services. I can’t think of any situations where we actually want to restart the coordinators first, but we should reason it out more for sure.\r\n\r\nI also added a delay of 20s after each batch to wait for that batch of processes to actually finish connecting and coming online. This is arbitrary but I figure it’s good enough, and since a computer is doing this work in the background I am not really worried about it taking too long.\r\n\r\nWhat this lacks: any kind of rack awareness, any kind of parallelism other than submitting batch commands to CDM, any kind of speed, or much production validation.","created_at":"2015-11-24T04:47:56Z","updated_at":"2015-11-25T01:34:18Z","closed_at":null,"merged_at":null,"merge_commit_sha":"62fbbcd41f01da4b891dfa1a430ab52a9319cd44","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2/commits","review_comments_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2/comments","review_comment_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/comments{/number}","comments_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/2/comments","statuses_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/statuses/c731b3007183e730836ab18d9e19e2a3eba7ba60","head":{"label":"airhorns:rolling-restarts","ref":"rolling-restarts","sha":"c731b3007183e730836ab18d9e19e2a3eba7ba60","user":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"repo":{"id":46534695,"name":"cloudera-manager-client","full_name":"airhorns/cloudera-manager-client","owner":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/airhorns/cloudera-manager-client","description":"Ruby client gem for the Cloudera Manager REST API","fork":false,"url":"https://api.github.com/repos/airhorns/cloudera-manager-client","forks_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/forks","keys_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/teams","hooks_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/hooks","issue_events_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/events{/number}","events_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/events","assignees_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/assignees{/user}","branches_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/branches{/branch}","tags_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/tags","blobs_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/refs{/sha}","trees_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/statuses/{sha}","languages_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/languages","stargazers_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/stargazers","contributors_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/contributors","subscribers_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/subscribers","subscription_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/subscription","commits_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/commits{/sha}","git_commits_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/commits{/sha}","comments_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/comments{/number}","issue_comment_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/comments{/number}","contents_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/contents/{+path}","compare_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/merges","archive_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/downloads","issues_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues{/number}","pulls_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls{/number}","milestones_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/milestones{/number}","notifications_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/labels{/name}","releases_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/releases{/id}","created_at":"2015-11-20T02:44:57Z","updated_at":"2015-11-20T03:30:42Z","pushed_at":"2015-11-24T04:47:56Z","git_url":"git://github.com/airhorns/cloudera-manager-client.git","ssh_url":"git@github.com:airhorns/cloudera-manager-client.git","clone_url":"https://github.com/airhorns/cloudera-manager-client.git","svn_url":"https://github.com/airhorns/cloudera-manager-client","homepage":"","size":85,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"base":{"label":"airhorns:client","ref":"client","sha":"bafd4bada9c2398f6c6fea8c3e5089fa7b257c9a","user":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"repo":{"id":46534695,"name":"cloudera-manager-client","full_name":"airhorns/cloudera-manager-client","owner":{"login":"airhorns","id":158950,"avatar_url":"https://avatars.githubusercontent.com/u/158950?v=3","gravatar_id":"","url":"https://api.github.com/users/airhorns","html_url":"https://github.com/airhorns","followers_url":"https://api.github.com/users/airhorns/followers","following_url":"https://api.github.com/users/airhorns/following{/other_user}","gists_url":"https://api.github.com/users/airhorns/gists{/gist_id}","starred_url":"https://api.github.com/users/airhorns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/airhorns/subscriptions","organizations_url":"https://api.github.com/users/airhorns/orgs","repos_url":"https://api.github.com/users/airhorns/repos","events_url":"https://api.github.com/users/airhorns/events{/privacy}","received_events_url":"https://api.github.com/users/airhorns/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/airhorns/cloudera-manager-client","description":"Ruby client gem for the Cloudera Manager REST API","fork":false,"url":"https://api.github.com/repos/airhorns/cloudera-manager-client","forks_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/forks","keys_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/keys{/key_id}","collaborators_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/teams","hooks_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/hooks","issue_events_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/events{/number}","events_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/events","assignees_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/assignees{/user}","branches_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/branches{/branch}","tags_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/tags","blobs_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/refs{/sha}","trees_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/trees{/sha}","statuses_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/statuses/{sha}","languages_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/languages","stargazers_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/stargazers","contributors_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/contributors","subscribers_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/subscribers","subscription_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/subscription","commits_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/commits{/sha}","git_commits_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/git/commits{/sha}","comments_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/comments{/number}","issue_comment_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/comments{/number}","contents_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/contents/{+path}","compare_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/compare/{base}...{head}","merges_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/merges","archive_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/downloads","issues_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues{/number}","pulls_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls{/number}","milestones_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/milestones{/number}","notifications_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/labels{/name}","releases_url":"https://api.github.com/repos/airhorns/cloudera-manager-client/releases{/id}","created_at":"2015-11-20T02:44:57Z","updated_at":"2015-11-20T03:30:42Z","pushed_at":"2015-11-24T04:47:56Z","git_url":"git://github.com/airhorns/cloudera-manager-client.git","ssh_url":"git@github.com:airhorns/cloudera-manager-client.git","clone_url":"https://github.com/airhorns/cloudera-manager-client.git","svn_url":"https://github.com/airhorns/cloudera-manager-client","homepage":"","size":85,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":2,"forks":0,"open_issues":2,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2"},"html":{"href":"https://github.com/airhorns/cloudera-manager-client/pull/2"},"issue":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/2"},"comments":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/issues/2/comments"},"review_comments":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2/comments"},"review_comment":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/pulls/2/commits"},"statuses":{"href":"https://api.github.com/repos/airhorns/cloudera-manager-client/statuses/c731b3007183e730836ab18d9e19e2a3eba7ba60"}}}}