{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/45560774","id":45560774,"diff_hunk":"@@ -0,0 +1,216 @@\n+<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->\n+\n+<!-- BEGIN STRIP_FOR_RELEASE -->\n+\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+\n+<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>\n+\n+If you are using a released version of Kubernetes, you should\n+refer to the docs that go with that version.\n+\n+<strong>\n+The latest release of this document can be found\n+[here](http://releases.k8s.io/release-1.1/docs/proposals/maintenance.md).\n+\n+Documentation for other releases can be found at\n+[releases.k8s.io](http://releases.k8s.io).\n+</strong>\n+--\n+\n+<!-- END STRIP_FOR_RELEASE -->\n+\n+<!-- END MUNGE: UNVERSIONED_WARNING -->\n+\n+# Node Maintenance\n+\n+## Abstract\n+\n+A proposal for implementing safe node maintenance primitives, in stages.\n+\n+## Overview\n+\n+Nodes need periodic maintenance, most of which require partial or total\n+unavailability to pods.  The cluster admin would like for this to happen (1)\n+automatically, and (2) respecting any service quality constraints.\n+\n+Examples of maintenance:\n+\n+- unmount/remount filesystems to change mkfs params\n+- remove from service but leave up for debugging\n+- reboot for kernel or image upgrade\n+- remove from service for hardware repair\n+- gracefully reducing the size of the cluster\n+- take down machines all machines in a rack for maintenance\n+\n+## Service Quality\n+\n+This may be thought of as the cluster admin offering a guarantee or SLO, or as the\n+user specifying their requirements.  Around evictions, there are typically three\n+parameters:\n+\n+- time between evictions (of pods in the same replica set)\n+- minimum shard strength (minimum fraction of pods healthy)\n+- eviction notice\n+\n+## Implementations\n+\n+In order of increasing complexity (and benefit).\n+\n+### Drain one node\n+\n+This is implemented in https://github.com/kubernetes/kubernetes/pull/16698\n+\n+### One-shot drain of multiple nodes\n+\n+In this case, we try to meet some specific service goals specified by the admin\n+on the command line.  The specification of these parameters only when we run the\n+tool lets us provide the possibility of safety while avoiding questions of where\n+to configure and store such specification (treated further down).\n+\n+```\n+kubectl drain \\\n+  --min-shard-strength=0.75 \\\n+  --min-seconds-between-evictions=900 \\\n+  --grace=900 $nodes\n+```\n+\n+This would still be pretty simple.  We offer no guarantees about safety in the\n+face of crashes or restarts, but we do meet the needs of admins who want\n+something more sophisticated than `sleep`.  If kubectl was interrupted and then\n+restarted with the same arguments, the end state would be exactly the same and\n+we'd never break the shard strength rule, but the evictions might be more\n+closely spaced.\n+\n+\n+### Implementation of the above as a controller\n+\n+The parameters of a specific lame, drain, or unlame request could be pushed to\n+the API server as a drain object.  The drain controller would have to manage\n+multiple drains in flight at once.  If the drains have differing parameters,\n+then more aggressive drains would generally starve safer ones.  While this could\n+be a useful feature in emergencies, it will be a lot easier to reason about\n+multiple drains in progress if they all share the same parameters, which would\n+be configured via kube-env and would represent the cluster admin's SLO.\n+\n+The drain object would thus look something like this","path":"docs/proposals/maintenance.md","position":105,"original_position":105,"commit_id":"7bdf740379f7e7ee59fc2421f85103cd29e1f319","original_commit_id":"7bdf740379f7e7ee59fc2421f85103cd29e1f319","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"body":"I wasn't invested in the alternative, but want to have the design call out why it is a resource so we can extract that into API conventions and use it for future designs.  E.g this is a Job on the Cluster, it runs to completion but must record arguments state, has a termination state, is garbage collected by another controller eventually, etc.","created_at":"2015-11-22T16:26:27Z","updated_at":"2015-11-22T16:26:27Z","html_url":"https://github.com/kubernetes/kubernetes/pull/17393#discussion_r45560774","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/45560774"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17393#discussion_r45560774"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393","id":51012463,"html_url":"https://github.com/kubernetes/kubernetes/pull/17393","diff_url":"https://github.com/kubernetes/kubernetes/pull/17393.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/17393.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393","number":17393,"state":"open","locked":false,"title":"Add proposal for node maintenance.","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"body":"AKA draining","created_at":"2015-11-17T22:10:36Z","updated_at":"2015-11-22T16:26:27Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c752efff66f0e06141abfc9607330cea1d61717c","assignee":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/7bdf740379f7e7ee59fc2421f85103cd29e1f319","head":{"label":"mml:drain-doc","ref":"drain-doc","sha":"7bdf740379f7e7ee59fc2421f85103cd29e1f319","user":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"repo":{"id":45429185,"name":"kubernetes","full_name":"mml/kubernetes","owner":{"login":"mml","id":3115,"avatar_url":"https://avatars.githubusercontent.com/u/3115?v=3","gravatar_id":"","url":"https://api.github.com/users/mml","html_url":"https://github.com/mml","followers_url":"https://api.github.com/users/mml/followers","following_url":"https://api.github.com/users/mml/following{/other_user}","gists_url":"https://api.github.com/users/mml/gists{/gist_id}","starred_url":"https://api.github.com/users/mml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mml/subscriptions","organizations_url":"https://api.github.com/users/mml/orgs","repos_url":"https://api.github.com/users/mml/repos","events_url":"https://api.github.com/users/mml/events{/privacy}","received_events_url":"https://api.github.com/users/mml/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mml/kubernetes","description":"Container Cluster Manager from Google","fork":true,"url":"https://api.github.com/repos/mml/kubernetes","forks_url":"https://api.github.com/repos/mml/kubernetes/forks","keys_url":"https://api.github.com/repos/mml/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mml/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mml/kubernetes/teams","hooks_url":"https://api.github.com/repos/mml/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/mml/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/mml/kubernetes/events","assignees_url":"https://api.github.com/repos/mml/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/mml/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/mml/kubernetes/tags","blobs_url":"https://api.github.com/repos/mml/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mml/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mml/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/mml/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mml/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/mml/kubernetes/languages","stargazers_url":"https://api.github.com/repos/mml/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/mml/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/mml/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/mml/kubernetes/subscription","commits_url":"https://api.github.com/repos/mml/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/mml/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/mml/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/mml/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/mml/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/mml/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mml/kubernetes/merges","archive_url":"https://api.github.com/repos/mml/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mml/kubernetes/downloads","issues_url":"https://api.github.com/repos/mml/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/mml/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/mml/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/mml/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mml/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/mml/kubernetes/releases{/id}","created_at":"2015-11-02T23:37:23Z","updated_at":"2015-11-02T23:37:41Z","pushed_at":"2015-11-21T00:55:08Z","git_url":"git://github.com/mml/kubernetes.git","ssh_url":"git@github.com:mml/kubernetes.git","clone_url":"https://github.com/mml/kubernetes.git","svn_url":"https://github.com/mml/kubernetes","homepage":"http://kubernetes.io","size":139517,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"e25aa2e1e3e215cfa1119cd7b4e62c6d1718d566","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-11-22T15:26:35Z","pushed_at":"2015-11-22T13:16:47Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":289448,"stargazers_count":11374,"watchers_count":11374,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3019,"mirror_url":null,"open_issues_count":2385,"forks":3019,"open_issues":2385,"watchers":11374,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17393"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17393/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17393/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/7bdf740379f7e7ee59fc2421f85103cd29e1f319"}}}}