{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/44742534","id":44742534,"diff_hunk":"@@ -0,0 +1,641 @@\n+<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->\n+\n+<!-- BEGIN STRIP_FOR_RELEASE -->\n+\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+\n+<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>\n+\n+If you are using a released version of Kubernetes, you should\n+refer to the docs that go with that version.\n+\n+<strong>\n+The latest 1.0.x release of this document can be found\n+[here](http://releases.k8s.io/release-1.0/docs/proposals/disk-accounting.md).\n+\n+Documentation for other releases can be found at\n+[releases.k8s.io](http://releases.k8s.io).\n+</strong>\n+--\n+\n+<!-- END STRIP_FOR_RELEASE -->\n+\n+<!-- END MUNGE: UNVERSIONED_WARNING -->\n+\n+**Author**: Vishnu Kannan\n+\n+**Last** **Updated**: 11/5/2015\n+\n+**Status**: Pending Review\n+\n+This proposal is an attempt to come up with a means for accounting disk usage in Kubernetes clusters that are running docker as the container runtime. Some of the principles here might apply for other runtimes too.\n+\n+### Why is disk accounting necessary?\n+\n+Present kubernetes clusters become unusable over time due to the local disk becoming full. The kubelets on the node attempt to perform garbage collection of old containers and images, but that doesn’t prevent running pods from using up all the available disk space.\n+\n+Kubernetes users have no insight into how the disk is being consumed.\n+\n+Large images and rapid logging, can lead to temporary downtime on the nodes. The node has to free up disk space by deleting images and containers. During this cleanup, existing pods can fail and new pods cannot be started. The node will also transition into an `OutOfDisk` condition, preventing more pods from being scheduled to the node.\n+\n+Automated eviction of pods that are hogging the local disk is not possible since proper accounting isn’t available.\n+\n+Since local disk is a non-compressible resource, users need means to restrict usage of local disk by pods and containers. Proper disk accounting is a prerequisite. As of today, a misconfigured low QoS class pod, can end up bringing down the entire cluster by taking up all the available disk space (misconfigured logging for example)\n+\n+### Goals\n+\n+1. Account for disk usage on the nodes.\n+\n+2. Compatibility with the most important docker storage backends - devicemapper, aufs and overlayfs\n+\n+3. Provide a roadmap for enabling disk as a schedulable resource in the future.\n+\n+4. Provide a plugin interface for extending support to non-default filesystems and storage drivers.\n+\n+### Non Goals\n+\n+1. Compatibility with all storage backends. The matrix is pretty large already and the priority is to get disk accounting to on most widely deployed platforms.\n+\n+2. Support for filesystems other than ext4.\n+\n+### Introduction\n+\n+Disk accounting in Kubernetes cluster running with docker is complex because of the plethora of ways in which disk gets utilized by a container.\n+\n+Disk can be consumed for,\n+\n+1. Container images\n+\n+2. Container’s writable layer\n+\n+3. Container’s logs - when written to stdout/stderr and default logging backend in docker is used.\n+\n+4. Local volumes - hostPath and emptyDir (TODO: Do git volumes consume local disk?)\n+\n+Kubelet exposes disk usage for the entire node and the container’s writable layer for aufs docker storage driver. This information is made available to end users via the heapster monitoring pipeline.\n+\n+#### Image layers\n+\n+Image layers are shared between containers (COW) and so accounting for images is complicated.\n+\n+Image layers will have to be accounted as system overhead.\n+\n+As of today, it is not possible to perform feasibility checks on images before being pulled on the node.\n+\n+#### Writable Layer\n+\n+Docker creates a writable layer for every container on the host. Depending on the storage driver, the location and the underlying filesystem of this layer will change.\n+\n+Any files that the container creates or updates (assuming there are no volumes) will be considered as writable layer usage.\n+\n+The underlying filesystem is whatever the docker storage directory resides on. It is ext4 by default on most distributions.\n+\n+#### Container’s logs\n+\n+Docker engine provides a pluggable logging interface. Kubernetes is currently using the default logging mode which is `local file`. In this mode, the docker daemon stores bytes written by containers to their stdout or stderr, to local disk. These log files are contained in a special directory that is managed by the docker daemon. These logs are exposed via `docker logs` interface which is then exposed via kubelet and apiserver APIs. Currently, there is a hard-requirement for persisting these log files on the disk.\n+\n+#### Local Volumes\n+\n+Volumes are slightly different from the other use cases local disk. They are pod scoped. Their lifetime is tied to that of a pod. Due to this property accounting of volumes will also be at the pod level.\n+\n+As of now, the volume types that use local disk directly are ‘HostPath’, ‘EmptyDir’, and ‘GitRepo’. Everything else is a network based volume.\n+\n+‘HostPath’ volumes map in existing directories in the host filesystem into a pod. Kubernetes manages only the mapping. It does not manage the source on the host filesystem.\n+\n+In addition to this, the changes introduced by a pod on the source of a hostPath volume is not cleaned by kubernetes once the pod exits. Due to these limitations, we will have to account hostPath volumes to system overhead. We should explicitly discourage use of HostPath in read-write mode.\n+\n+`EmptyDir` and `GitRepo` volumes map to a directory on the host root filesystem, that is managed by Kubernetes (kubelet). Their contents are erased as soon as the pod exits. Tracking and potentially restricting usage for volumes is possible.\n+\n+### Docker storage model\n+\n+Before we start exploring solutions, let’s get familiar with how docker handles storage for images, writable layer and logs.\n+\n+On all storage drivers, logs are stored under `<docker root dir>/containers/<container-id>/`\n+\n+The default location of the docker root directory is `/var/lib/docker`.\n+\n+Volumes are handled by kubernetes.\n+\n+Container images and writable layers are managed by docker and their location will change depending on the storage driver. Each image layer and writable layer is referred to by an ID. The image layers are read-only. Once saved, existing writable layers can be frozen. Saving feature is not of importance to kubernetes since it works only on immutable images.\n+\n+*Note: Image layer IDs can be obtained by running `docker history -q --no-trunc <imagename>`*\n+\n+##### Aufs\n+\n+Image layers and writable layers are stored under `/var/lib/docker/aufs/diff/<id>`.\n+\n+The writable layers ID is equivalent to that of the container ID.\n+\n+##### Devicemapper\n+\n+Each container and each image gets own block device. Since this driver works at the block level, it is not possible to access the layers directly without mounting them. Each container gets its own block device while running.\n+\n+##### Overlayfs\n+\n+Image layers and writable layers are stored under `/var/lib/docker/overlay/<id>`.\n+\n+Identical files are hardlinked between images.\n+\n+The image layers contain all their data under a `root` subdirectory.\n+\n+Everything under  `/var/lib/docker/overlay/<container-id>` are files required for running the container, including its writable layer.\n+\n+### Improve disk accounting\n+\n+Disk accounting is dependent on the storage driver in docker. A common solution that works across all storage drivers isn't available.\n+\n+I’m listing a few possible solutions for disk accounting below along with their limitations.\n+\n+We need a plugin model for disk accounting. Some storage drivers in docker will require special plugins.\n+\n+#### Container Images\n+\n+As of today, the partition that is holding docker images is flagged by cadvisor, and it uses filesystem stats to identify the overall disk usage of that partition.\n+\n+Isolated usage of just image layers isn’t available today.\n+\n+However, continuing to use the entire partition availability for garbage collection purposes in kubelet, should not affect reliability.\n+We might garbage collect more often.\n+As long as we do not expose features that require persisting old containers, computing image layer usage wouldn’t be necessary.\n+\n+In case we choose to compute the size of image layers alone, the following are some of the ways to achieve that.\n+\n+*Note that the strategies mentioned below are applicable in general to other kinds of storage like volumes, etc.*\n+\n+##### Overlayfs and Aufs\n+\n+1. `du`\n+\n+\tWe can list all the image layer specific directories, excluding container directories, and run `du` on each of those directories.\n+\n+**Pros**:\n+\n+* This is the least-intrusive approach.\n+\n+* It will work off the box without requiring any additional configuration.\n+\n+**Cons**:\n+\n+* `du` can consume a lot of cpu and memory. There have been several issues reported against the kubelet in the past that were related to `du`.\n+\n+* It is time consuming. Cannot be run frequently. Requires special handling to constrain resource usage - setting lower nice value or running in a sub-container.\n+\n+* Can block container deletion by keeping file descriptors open.\n+\n+\n+2. Linux gid based Disk Quota\n+\n+\t[Disk quota](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-disk-quotas.html) feature provided by the linux kernel can be used to track the usage of image layers. Ideally, we need `project` support for disk quota, which let’s us track usage of directory hierarchies using `project ids`. Unfortunately, that feature is only available for zfs filesystems. Since most of our distributions use `ext4` by default, we will have to use either `uid` or `gid` based quota tracking.\n+\n+\tBoth `uids` and `gids` are meant for security. Overloading that concept for disk tracking is painful and ugly. But, that is what we have today.\n+\n+\tKubelet needs to define a gid for tracking image layers and make that gid or group the owner of `/var/lib/docker/[aufs | overlayfs]` recursively. Once this is done, the quota sub-system in the kernel will report the blocks being consumed by the storage driver on the underlying partition.\n+\n+\tSince this number also includes the container’s writable layer, we will have to somehow subtract that usage from the overall usage of the storage driver directory. Luckily, we can use the same mechanism for tracking container’s writable layer. Once we apply a different `gid` to the container’s writable layer, which is located under `/var/lib/docker/<storage_driver>/diff/<container_id>`, the quota subsystem will not include the container’s writable layer usage.\n+\n+**Pros**:\n+\n+* Low overhead tracking provided by the kernel.\n+\n+\n+**Cons**\n+\n+* Requires updates to default ownership on docker’s internal storage driver directories. We will have to deal with storage driver implementation implementation details in any approach that is not docker native.\n+\n+* Requires additional node configuration - quota subsystem needs to be setup on the node. This can either be automated or made a requirement for the node.\n+\n+* Kubelet needs to perform gid management. A range of gids have to allocated to the kubelet for the purposes of quota management. This range must not be used for any other purposes out of band.\n+\n+* Breaks `docker save` semantics. Since kubernetes assumes immutable images, this is not a blocker. To support quota in docker, we will need user-namespaces along with custom gid mapping for each container. This feature does not exist today.","path":"docs/proposals/disk-accounting.md","position":218,"original_position":218,"commit_id":"07d246add8b62e7b991fbb216afbaeb55d54d5aa","original_commit_id":"07d246add8b62e7b991fbb216afbaeb55d54d5aa","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"body":"I will take some time to think through side-car approach and report back.\nIt's not a bad suggestion.\n\nOn Thursday, November 12, 2015, Vish Kannan <notifications@github.com>\nwrote:\n\n> In docs/proposals/disk-accounting.md\n> <https://github.com/kubernetes/kubernetes/pull/16889#discussion_r44740600>\n> :\n>\n> > +\tSince this number also includes the container’s writable layer, we will have to somehow subtract that usage from the overall usage of the storage driver directory. Luckily, we can use the same mechanism for tracking container’s writable layer. Once we apply a different `gid` to the container’s writable layer, which is located under `/var/lib/docker/<storage_driver>/diff/<container_id>`, the quota subsystem will not include the container’s writable layer usage.\n> > +\n> > +**Pros**:\n> > +\n> > +* Low overhead tracking provided by the kernel.\n> > +\n> > +\n> > +**Cons**\n> > +\n> > +* Requires updates to default ownership on docker’s internal storage driver directories. We will have to deal with storage driver implementation implementation details in any approach that is not docker native.\n> > +\n> > +* Requires additional node configuration - quota subsystem needs to be setup on the node. This can either be automated or made a requirement for the node.\n> > +\n> > +* Kubelet needs to perform gid management. A range of gids have to allocated to the kubelet for the purposes of quota management. This range must not be used for any other purposes out of band.\n> > +\n> > +* Breaks `docker save` semantics. Since kubernetes assumes immutable images, this is not a blocker. To support quota in docker, we will need user-namespaces along with custom gid mapping for each container. This feature does not exist today.\n>\n> Allowing such a functionality will essentially eliminate use of quota for\n> writable layer tracking. I can see why it is useful though. It could be a\n> cluster configuration option - Need better accounting vs I care only about\n> CI.\n>\n> @derekwaynecarr <https://github.com/derekwaynecarr>: Is it possible to\n> run a side-car that waits for the main container to exit, identifies the\n> main container's ID through Kube APIs (its there in container Status), and\n> then perform a commit and a push?\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kubernetes/kubernetes/pull/16889/files#r44740600>.\n>\n","created_at":"2015-11-13T01:57:50Z","updated_at":"2015-11-13T01:57:50Z","html_url":"https://github.com/kubernetes/kubernetes/pull/16889#discussion_r44742534","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/44742534"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/16889#discussion_r44742534"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889","id":49898133,"html_url":"https://github.com/kubernetes/kubernetes/pull/16889","diff_url":"https://github.com/kubernetes/kubernetes/pull/16889.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/16889.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16889","number":16889,"state":"open","locked":false,"title":"Proposal for disk accounting","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"body":"@kubernetes/goog-node @bgrant0607 @thockin @smarterclayton ","created_at":"2015-11-05T23:48:31Z","updated_at":"2015-11-13T01:57:50Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c844a2684d9ed023eb64d863f9529730393ce5a1","assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16889/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/07d246add8b62e7b991fbb216afbaeb55d54d5aa","head":{"label":"vishh:disk-accounting","ref":"disk-accounting","sha":"07d246add8b62e7b991fbb216afbaeb55d54d5aa","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"repo":{"id":26461931,"name":"kubernetes","full_name":"vishh/kubernetes","owner":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vishh/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/vishh/kubernetes","forks_url":"https://api.github.com/repos/vishh/kubernetes/forks","keys_url":"https://api.github.com/repos/vishh/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vishh/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vishh/kubernetes/teams","hooks_url":"https://api.github.com/repos/vishh/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/vishh/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/vishh/kubernetes/events","assignees_url":"https://api.github.com/repos/vishh/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/vishh/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/vishh/kubernetes/tags","blobs_url":"https://api.github.com/repos/vishh/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vishh/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vishh/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/vishh/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vishh/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/vishh/kubernetes/languages","stargazers_url":"https://api.github.com/repos/vishh/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/vishh/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/vishh/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/vishh/kubernetes/subscription","commits_url":"https://api.github.com/repos/vishh/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/vishh/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/vishh/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/vishh/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/vishh/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/vishh/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vishh/kubernetes/merges","archive_url":"https://api.github.com/repos/vishh/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vishh/kubernetes/downloads","issues_url":"https://api.github.com/repos/vishh/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/vishh/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/vishh/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/vishh/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vishh/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/vishh/kubernetes/releases{/id}","created_at":"2014-11-11T00:05:03Z","updated_at":"2014-12-18T20:48:07Z","pushed_at":"2015-11-11T23:20:22Z","git_url":"git://github.com/vishh/kubernetes.git","ssh_url":"git@github.com:vishh/kubernetes.git","clone_url":"https://github.com/vishh/kubernetes.git","svn_url":"https://github.com/vishh/kubernetes","homepage":null,"size":121549,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"3a5768a0ad04842a3b0e183218d9711f6bcfc4d6","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-11-13T01:56:05Z","pushed_at":"2015-11-13T01:57:05Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":524627,"stargazers_count":11228,"watchers_count":11228,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2947,"mirror_url":null,"open_issues_count":2270,"forks":2947,"open_issues":2270,"watchers":11228,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/16889"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/16889"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/16889/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/16889/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/07d246add8b62e7b991fbb216afbaeb55d54d5aa"}}}}