{"action":"created","comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/34839413","id":34839413,"diff_hunk":"@@ -45,7 +45,76 @@ of the relevant log files.  (note that on systemd based systems, you may need to\n \n ### Worker Nodes\n    * /var/log/kubelet.log - Kubelet, responsible for running containers on the node\n-   * /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing\n+   * /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing## A general overview of cluster failure modes\n+\n+This is an incomplete list of things that could go wrong, and how to deal with them.\n+\n+Root causes:\n+  - VM(s) shutdown\n+  - network partition within cluster, or between cluster and users.\n+  - crashes in Kubernetes software \n+  - data loss or unavailability of persistent storage (e.g. GCE PD or AWS EBS volume).\n+  - operator error misconfigures kubernetes software or application software.\n+\n+Specific scenarios:\n+  - Apiserver VM shutdown or apiserver crashing\n+    - Results\n+      - unable to stop, update, or start new pods, services, replication controller\n+      - existing pods and services should continue to work normally, unless they depend on the Kubernetes API\n+  - Apiserver backing storage lost\n+    - Results\n+      - apiserver should fail to come up.\n+      - kubelets will not be able to reach it but will continue to run the same pods and provide the same service proxying.\n+      - manual recovery or recreation of apiserver state necessary before apiserver is restarted.\n+  - Supporting services (node controller, replication controller manager, scheduler, etc) VM shutdown or crashes\n+    - currently those are colocated with the apiserver, and their unavailability has similar consequences as apiserver\n+    - in future, these will be replicated as well and may not be co-located\n+    - they do not have own persistent state\n+  - Node (thing that runs kubelet and kube-proxy and pods) shutdown\n+    - Results\n+      - pods on that Node stop running","path":"docs/admin/cluster-troubleshooting.md","position":32,"original_position":32,"commit_id":"d825f6789874a6f17b5fd0775986331619c36dab","original_commit_id":"d825f6789874a6f17b5fd0775986331619c36dab","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"body":"done","created_at":"2015-07-16T21:22:31Z","updated_at":"2015-07-16T21:22:31Z","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367#discussion_r34839413","pull_request_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367","_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/34839413"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367#discussion_r34839413"},"pull_request":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367"}}},"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367","id":40109619,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367.patch","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11367","number":11367,"state":"open","locked":false,"title":"Take availability.md doc and","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"body":"- extract the portion related to multi-cluster operation into a new multi-cluster.md doc\r\n- merge the remainder (that was basically high-level troubleshooting advice) into cluster-troubleshooting.md\r\n\r\n@erictune @lavalamp \r\n","created_at":"2015-07-16T09:22:26Z","updated_at":"2015-07-16T21:22:31Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1c60f1a3b1d5faa79843997b77ea4c39b2f2a11c","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367/commits","review_comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367/comments","review_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11367/comments","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/d825f6789874a6f17b5fd0775986331619c36dab","head":{"label":"davidopp:doc2","ref":"doc2","sha":"d825f6789874a6f17b5fd0775986331619c36dab","user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"repo":{"id":25228292,"name":"kubernetes","full_name":"davidopp/kubernetes","owner":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/davidopp/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/davidopp/kubernetes","forks_url":"https://api.github.com/repos/davidopp/kubernetes/forks","keys_url":"https://api.github.com/repos/davidopp/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/davidopp/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/davidopp/kubernetes/teams","hooks_url":"https://api.github.com/repos/davidopp/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/davidopp/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/davidopp/kubernetes/events","assignees_url":"https://api.github.com/repos/davidopp/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/davidopp/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/davidopp/kubernetes/tags","blobs_url":"https://api.github.com/repos/davidopp/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/davidopp/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/davidopp/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/davidopp/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/davidopp/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/davidopp/kubernetes/languages","stargazers_url":"https://api.github.com/repos/davidopp/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/davidopp/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/davidopp/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/davidopp/kubernetes/subscription","commits_url":"https://api.github.com/repos/davidopp/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/davidopp/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/davidopp/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/davidopp/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/davidopp/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/davidopp/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/davidopp/kubernetes/merges","archive_url":"https://api.github.com/repos/davidopp/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/davidopp/kubernetes/downloads","issues_url":"https://api.github.com/repos/davidopp/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/davidopp/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/davidopp/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/davidopp/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/davidopp/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/davidopp/kubernetes/releases{/id}","created_at":"2014-10-14T21:53:51Z","updated_at":"2015-04-19T21:08:14Z","pushed_at":"2015-07-16T20:21:41Z","git_url":"git://github.com/davidopp/kubernetes.git","ssh_url":"git@github.com:davidopp/kubernetes.git","clone_url":"https://github.com/davidopp/kubernetes.git","svn_url":"https://github.com/davidopp/kubernetes","homepage":null,"size":89679,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"GoogleCloudPlatform:master","ref":"master","sha":"596a8a40d12498b5335140f50753980bfaea4f6b","user":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"GoogleCloudPlatform/kubernetes","owner":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes","forks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/forks","keys_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/teams","hooks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/events","assignees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/tags","blobs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/languages","stargazers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscription","commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/merges","archive_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/downloads","issues_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-07-16T20:27:00Z","pushed_at":"2015-07-16T21:13:44Z","git_url":"git://github.com/GoogleCloudPlatform/kubernetes.git","ssh_url":"git@github.com:GoogleCloudPlatform/kubernetes.git","clone_url":"https://github.com/GoogleCloudPlatform/kubernetes.git","svn_url":"https://github.com/GoogleCloudPlatform/kubernetes","homepage":"http://kubernetes.io","size":317941,"stargazers_count":8951,"watchers_count":8951,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2032,"mirror_url":null,"open_issues_count":1628,"forks":2032,"open_issues":1628,"watchers":8951,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11367"},"issue":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11367"},"comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11367/comments"},"review_comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367/comments"},"review_comment":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11367/commits"},"statuses":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/d825f6789874a6f17b5fd0775986331619c36dab"}}}}