{"action":"created","comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/35287802","id":35287802,"diff_hunk":"@@ -0,0 +1,125 @@\n+# Quality of Service in Kubelet\r\n+\r\n+**Author**: Ananya Kumar (@AnanyaKumar)\r\n+\r\n+**Status**: Draft proposal; prototype in progress.\r\n+\r\n+This document presents the design of quality of service tiers for jobs scheduled on Kubelet, describes use cases, and lays out milestones for its development.\r\n+\r\n+## Motivation\r\n+\r\n+The Kubelet allocates resources to pods in a simple way. Right now there aren’t robust mechanisms in place to ensure that pods run reliably on an overloaded system, and nodes are not very efficiently utilized.\r\n+\r\n+In Kubernetes, some pods have limits specified, and some pods do not. **Pods with limits specified are not very well protected from pods without limits specified**. If a pod without an explicit memory limit goes overboard and uses lots of memory, other pods (with resource specifications) might be killed. This is bad, because users often want a way to launch containers that have resources guarantees, and that stay up reliably.\r\n+\r\n+On the other hand in the current system, **if all pods have specified limits, cluster utilization can be poor**. In the current implementation, Kubelet does not support over-committing pods. Very often, pods don’t use all the resources that they request, which leads to a lot of wasted resources. For example, we might have 4 pods, each reserving 1GB of memory in a system with 4GB memory, but only using 500MB of memory. Theoretically, we could slot in another 4 pods of this type, but Kubernetes will reject new pods.\r\n+\r\n+In this design document, we propose **implementing two tiers of pods** - one in which we guarantee resources (top-tier pods) and the second wherein we provide resources if they’re available and at a slightly lower quality (best-effort pods). Adding a best-effort tier of pods will increase node’s utilization in a Kubernetes cluster (in Borg, supporting best-effort pods increased utilization by  about 20%). We also implement **a policy of automatically classifying pods that don’t provide limits as best-effort**. This provides much better resource guarantees to pods with limits specified.\r\n+\r\n+## QoS Functionality\r\n+\r\n+We will introduce two tiers of pods: top-tier pods and best-effort pods. We will not support multiple levels or priorities.\r\n+\r\n+Best-effort pods will be invisible to top-tier pods:\r\n+- **Existence of best-effort pods should not prevent top tier pods from launching**. When deciding whether to launch a top-tier pod, we only look at resource limits of other top-tier pod. Suppose we have a machine with 4GB of memory. Top-tier pod P1 has a 2GB limit, and 4 best-effort pods have a 500MB limit each. Suppose the user launches a new top-tier pod P2 with a 1GB limit. The Kubelet will evict best-effort pods, in unspecified order, to make room for P2. P2 will then be admitted and allocated the requested resources.\r\n+- **Allocation of excess compressible resources (e.g. CPU) to top-tier pods will be the same as if best-effort pods did not exist**. For example if top-tier pod P1 asks for minimum 20% CPU, and we also have best-effort pods B1 and B2 running. Suppose P1 needs more CPU (because of a huge number of requests coming in), then P1 will starve B1, B2 of CPU and use the whole 100% of the CPU (assuming there are no other top tier pods running).\r\n+- **Allocation of non-compressible resources (e.g. memory) to top-tier pods will be the same as if best-effort pods did not exist**. For example consider a system with 4GB of memory, where top-tier pod P1 is using 500MB of memory (with a 1GB limit) and best-effort pods are using the other 3.5GB of memory. The best-effort pods will not prevent P1 from accessing more memory (assuming that P1 remains within its memory limits). So if P1 asks for 100MB more memory, P1 will get the memory it needs, and one of the other pods will be OOM killed. \r\n+\r\n+Resource guarantees for best-effort pods:\r\n+- Compressible resources\r\n+  - For now, we are only implementing CPU limits.\r\n+  - CPU shares not used by top-tier jobs will be distributed proportionally amongst best-effort pods (based on the amount of CPU the pods requested).\r\n+  - Best effort pods will not be evicted because of CPU constraints, they will simply be starved if CPU resources are not available.\r\n+- Non-compressible resources\r\n+  - For now, we are only implementing memory limits.\r\n+  - We will always launch a best-effort pod, regardless of its memory requirements.\r\n+  - Best effort pods will be evicted if a top tier pod needing the memory requirements comes in, or if some other top-tier pod asks for memory.\r\n+  - Best-effort pods will not be allowed to exceed their memory limit.\r\n+  - Extension: Consider distinguishing between best-effort pods that specify memory limits and give them even lower priority. Otherwise a best-effort pod without a memory limit could use up system resources and kill off other best-effort pods.\r\n+- Extension: support other resources like network, disk.\r\n+\r\n+Resource guarantees for top tier pods\r\n+- Will not change.\r\n+- Extension: Consider giving top-tier pods that do not specify memory limits a lower memory priority than top tier pods that specify memory limits (but a higher priority than best-effort pods)\r\n+\r\n+## Policy\r\n+\r\n+The QoS design does not make assumptions about the “policy” used to label pods as best-effort and top-tier.\r\n+\r\n+In the first iteration, we will stick to a simple policy. Pods that specify CPU and memory limits for every container will be classified as top-tier, all other pods will be classified as best-effort. This policy offers protection to pods that specify resource requirements (from pods that have no limits and can go overboard in resource consumption), which gives them better resource guarantees. \r\n+\r\n+Extension: In the future, we might add a field to PodSpec in the API, to allow users to specify whether a pod is best-effort or top-tier.\r\n+\r\n+## Alternatives Considered\r\n+\r\n+Designing a simple but flexible quality of service framework is very challenging because there are many possible applications that can use tiered pods in different ways. Here are a couple of alternatives considered, and reasons for why we are not implementing them.\r\n+\r\n+One alternative is that we put all best-effort pods into a parent cgroup, and restrict the memory of the parent cgroup based on resource limits specified by the top-tier pods. This alternative would reduce the chance of a system OOM kill because it prevents jobs from using too much memory, but it reduces utilization. Also, the alternative would require require constant adjustment of the memory limit of cgroups (using a user-space control loop), which is tricky, racy, and error-prone. In the proposal outlined in this design document, best-effort pods will be allowed to \"eat\" into top-tier pods' memory, but will be killed if top-tier pods needs the memory they reserved. This leads to better utilization. It is a difficult tradeoff, but the better utilization of the proposal is why we rejected this alternative.\r","path":"docs/proposals/kubelet-qos.md","position":57,"original_position":57,"commit_id":"ba9f2dc29e33fe0003081483d3cf05d574591860","original_commit_id":"ba9f2dc29e33fe0003081483d3cf05d574591860","user":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"body":"Memory expands like a gas. Do we need a user-space control loop regardless to apply memory pressure?","created_at":"2015-07-23T03:39:11Z","updated_at":"2015-07-23T03:39:11Z","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713#discussion_r35287802","pull_request_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713","_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/35287802"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713#discussion_r35287802"},"pull_request":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713"}}},"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713","id":40616905,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713.patch","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11713","number":11713,"state":"open","locked":false,"title":"QoS proposal","user":{"login":"AnanyaKumar","id":1862850,"avatar_url":"https://avatars.githubusercontent.com/u/1862850?v=3","gravatar_id":"","url":"https://api.github.com/users/AnanyaKumar","html_url":"https://github.com/AnanyaKumar","followers_url":"https://api.github.com/users/AnanyaKumar/followers","following_url":"https://api.github.com/users/AnanyaKumar/following{/other_user}","gists_url":"https://api.github.com/users/AnanyaKumar/gists{/gist_id}","starred_url":"https://api.github.com/users/AnanyaKumar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AnanyaKumar/subscriptions","organizations_url":"https://api.github.com/users/AnanyaKumar/orgs","repos_url":"https://api.github.com/users/AnanyaKumar/repos","events_url":"https://api.github.com/users/AnanyaKumar/events{/privacy}","received_events_url":"https://api.github.com/users/AnanyaKumar/received_events","type":"User","site_admin":false},"body":"Ref. #147","created_at":"2015-07-22T22:01:03Z","updated_at":"2015-07-23T03:39:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a01916392ddf85615c6c287103db077a5a8200b6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713/commits","review_comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713/comments","review_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11713/comments","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/ba9f2dc29e33fe0003081483d3cf05d574591860","head":{"label":"AnanyaKumar:qos-proposal","ref":"qos-proposal","sha":"ba9f2dc29e33fe0003081483d3cf05d574591860","user":{"login":"AnanyaKumar","id":1862850,"avatar_url":"https://avatars.githubusercontent.com/u/1862850?v=3","gravatar_id":"","url":"https://api.github.com/users/AnanyaKumar","html_url":"https://github.com/AnanyaKumar","followers_url":"https://api.github.com/users/AnanyaKumar/followers","following_url":"https://api.github.com/users/AnanyaKumar/following{/other_user}","gists_url":"https://api.github.com/users/AnanyaKumar/gists{/gist_id}","starred_url":"https://api.github.com/users/AnanyaKumar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AnanyaKumar/subscriptions","organizations_url":"https://api.github.com/users/AnanyaKumar/orgs","repos_url":"https://api.github.com/users/AnanyaKumar/repos","events_url":"https://api.github.com/users/AnanyaKumar/events{/privacy}","received_events_url":"https://api.github.com/users/AnanyaKumar/received_events","type":"User","site_admin":false},"repo":{"id":37498217,"name":"kubernetes","full_name":"AnanyaKumar/kubernetes","owner":{"login":"AnanyaKumar","id":1862850,"avatar_url":"https://avatars.githubusercontent.com/u/1862850?v=3","gravatar_id":"","url":"https://api.github.com/users/AnanyaKumar","html_url":"https://github.com/AnanyaKumar","followers_url":"https://api.github.com/users/AnanyaKumar/followers","following_url":"https://api.github.com/users/AnanyaKumar/following{/other_user}","gists_url":"https://api.github.com/users/AnanyaKumar/gists{/gist_id}","starred_url":"https://api.github.com/users/AnanyaKumar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AnanyaKumar/subscriptions","organizations_url":"https://api.github.com/users/AnanyaKumar/orgs","repos_url":"https://api.github.com/users/AnanyaKumar/repos","events_url":"https://api.github.com/users/AnanyaKumar/events{/privacy}","received_events_url":"https://api.github.com/users/AnanyaKumar/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/AnanyaKumar/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/AnanyaKumar/kubernetes","forks_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/forks","keys_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/teams","hooks_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/events","assignees_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/tags","blobs_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/languages","stargazers_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/subscription","commits_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/merges","archive_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/downloads","issues_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/AnanyaKumar/kubernetes/releases{/id}","created_at":"2015-06-16T00:14:24Z","updated_at":"2015-06-16T00:14:30Z","pushed_at":"2015-07-23T00:06:28Z","git_url":"git://github.com/AnanyaKumar/kubernetes.git","ssh_url":"git@github.com:AnanyaKumar/kubernetes.git","clone_url":"https://github.com/AnanyaKumar/kubernetes.git","svn_url":"https://github.com/AnanyaKumar/kubernetes","homepage":null,"size":91515,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"GoogleCloudPlatform:master","ref":"master","sha":"20293058ec8944e74ad23979c9494e9d7090147e","user":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"GoogleCloudPlatform/kubernetes","owner":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes","forks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/forks","keys_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/teams","hooks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/events","assignees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/tags","blobs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/languages","stargazers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscription","commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/merges","archive_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/downloads","issues_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-07-23T03:31:56Z","pushed_at":"2015-07-23T03:38:40Z","git_url":"git://github.com/GoogleCloudPlatform/kubernetes.git","ssh_url":"git@github.com:GoogleCloudPlatform/kubernetes.git","clone_url":"https://github.com/GoogleCloudPlatform/kubernetes.git","svn_url":"https://github.com/GoogleCloudPlatform/kubernetes","homepage":"http://kubernetes.io","size":325132,"stargazers_count":9244,"watchers_count":9244,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2100,"mirror_url":null,"open_issues_count":1715,"forks":2100,"open_issues":1715,"watchers":9244,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/11713"},"issue":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11713"},"comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11713/comments"},"review_comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713/comments"},"review_comment":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/11713/commits"},"statuses":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/ba9f2dc29e33fe0003081483d3cf05d574591860"}}}}