{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/46625450","id":46625450,"diff_hunk":"@@ -0,0 +1,216 @@\n+<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->\n+\n+<!-- BEGIN STRIP_FOR_RELEASE -->\n+\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+\n+<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>\n+\n+If you are using a released version of Kubernetes, you should\n+refer to the docs that go with that version.\n+\n+<strong>\n+The latest release of this document can be found\n+[here](http://releases.k8s.io/release-1.1/docs/proposals/maintenance.md).\n+\n+Documentation for other releases can be found at\n+[releases.k8s.io](http://releases.k8s.io).\n+</strong>\n+--\n+\n+<!-- END STRIP_FOR_RELEASE -->\n+\n+<!-- END MUNGE: UNVERSIONED_WARNING -->\n+\n+# Node Maintenance\n+\n+## Abstract\n+\n+A proposal for implementing safe node maintenance primitives, in stages.\n+\n+## Overview\n+\n+Nodes need periodic maintenance, most of which require partial or total\n+unavailability to pods.  The cluster admin would like for this to happen (1)\n+automatically, and (2) respecting any service quality constraints.\n+\n+Examples of maintenance:\n+\n+- unmount/remount filesystems to change mkfs params\n+- remove from service but leave up for debugging\n+- reboot for kernel or image upgrade\n+- remove from service for hardware repair\n+- gracefully reducing the size of the cluster\n+- take down machines all machines in a rack for maintenance\n+\n+## Service Quality\n+\n+This may be thought of as the cluster admin offering a guarantee or SLO, or as the\n+user specifying their requirements.  Around evictions, there are typically three\n+parameters:\n+\n+- time between evictions (of pods in the same replica set)\n+- minimum shard strength (minimum fraction of pods healthy)\n+- eviction notice\n+\n+## Implementations\n+\n+In order of increasing complexity (and benefit).\n+\n+### Drain one node\n+\n+This is implemented in https://github.com/kubernetes/kubernetes/pull/16698\n+\n+### One-shot drain of multiple nodes\n+\n+In this case, we try to meet some specific service goals specified by the admin\n+on the command line.  The specification of these parameters only when we run the\n+tool lets us provide the possibility of safety while avoiding questions of where\n+to configure and store such specification (treated further down).\n+\n+```\n+kubectl drain \\\n+  --min-shard-strength=0.75 \\\n+  --min-seconds-between-evictions=900 \\\n+  --grace=900 $nodes\n+```\n+\n+This would still be pretty simple.  We offer no guarantees about safety in the\n+face of crashes or restarts, but we do meet the needs of admins who want\n+something more sophisticated than `sleep`.  If kubectl was interrupted and then\n+restarted with the same arguments, the end state would be exactly the same and\n+we'd never break the shard strength rule, but the evictions might be more\n+closely spaced.\n+\n+\n+### Implementation of the above as a controller\n+\n+The parameters of a specific lame, drain, or unlame request could be pushed to\n+the API server as a drain object.  The drain controller would have to manage\n+multiple drains in flight at once.  If the drains have differing parameters,\n+then more aggressive drains would generally starve safer ones.  While this could\n+be a useful feature in emergencies, it will be a lot easier to reason about\n+multiple drains in progress if they all share the same parameters, which would\n+be configured via kube-env and would represent the cluster admin's SLO.\n+\n+The drain object would thus look something like this","path":"docs/proposals/maintenance.md","position":105,"original_position":105,"commit_id":"c39e4c2352da3e0aac262daee504cc09b83f6a8e","original_commit_id":"7bdf740379f7e7ee59fc2421f85103cd29e1f319","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"body":"Another reason for multiple drains is just that the need for them arises at different times and for different reasons.  For example, 10 machines are on faulty electrical equipment and we have to evacuate them in 30 minutes so the electrician can fix that.  In the background, we've been draining 30 other machines for non-urgent kernel upgrades.  I'll add an example like this.\r\n\r\nThis example conveniently avoids overlapping machines, which I will have to say something about.","created_at":"2015-12-03T22:51:30Z","updated_at":"2015-12-03T22:51:30Z","html_url":"https://github.com/kubernetes/kubernetes/pull/17393#discussion_r46625450","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/46625450"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17393#discussion_r46625450"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393","id":51012463,"html_url":"https://github.com/kubernetes/kubernetes/pull/17393","diff_url":"https://github.com/kubernetes/kubernetes/pull/17393.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/17393.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393","number":17393,"state":"open","locked":false,"title":"Add proposal for node maintenance.","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"body":"AKA draining","created_at":"2015-11-17T22:10:36Z","updated_at":"2015-12-03T22:51:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"530832a87168e72532f48906f5de11e5fab28440","assignee":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/c39e4c2352da3e0aac262daee504cc09b83f6a8e","head":{"label":"mml:drain-doc","ref":"drain-doc","sha":"c39e4c2352da3e0aac262daee504cc09b83f6a8e","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"repo":{"id":45429185,"name":"kubernetes","full_name":"mml/kubernetes","owner":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mml/kubernetes","description":"Container Cluster Manager from Google","fork":true,"url":"https://api.github.com/repos/mml/kubernetes","forks_url":"https://api.github.com/repos/mml/kubernetes/forks","keys_url":"https://api.github.com/repos/mml/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mml/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mml/kubernetes/teams","hooks_url":"https://api.github.com/repos/mml/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/mml/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/mml/kubernetes/events","assignees_url":"https://api.github.com/repos/mml/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/mml/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/mml/kubernetes/tags","blobs_url":"https://api.github.com/repos/mml/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mml/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mml/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/mml/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mml/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/mml/kubernetes/languages","stargazers_url":"https://api.github.com/repos/mml/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/mml/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/mml/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/mml/kubernetes/subscription","commits_url":"https://api.github.com/repos/mml/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/mml/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/mml/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/mml/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/mml/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/mml/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mml/kubernetes/merges","archive_url":"https://api.github.com/repos/mml/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mml/kubernetes/downloads","issues_url":"https://api.github.com/repos/mml/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/mml/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/mml/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/mml/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mml/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/mml/kubernetes/releases{/id}","created_at":"2015-11-02T23:37:23Z","updated_at":"2015-11-02T23:37:41Z","pushed_at":"2015-12-03T21:27:45Z","git_url":"git://github.com/mml/kubernetes.git","ssh_url":"git@github.com:mml/kubernetes.git","clone_url":"https://github.com/mml/kubernetes.git","svn_url":"https://github.com/mml/kubernetes","homepage":"http://kubernetes.io","size":144865,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"611770778ff7c25b1c49fb3c075479960caf9f53","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-12-03T22:11:54Z","pushed_at":"2015-12-03T22:50:06Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":294948,"stargazers_count":11603,"watchers_count":11603,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3127,"mirror_url":null,"open_issues_count":2468,"forks":3127,"open_issues":2468,"watchers":11603,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17393"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/c39e4c2352da3e0aac262daee504cc09b83f6a8e"}}}}