{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/40709317","id":40709317,"diff_hunk":"@@ -0,0 +1,136 @@\n+<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->\n+\n+<!-- BEGIN STRIP_FOR_RELEASE -->\n+\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+<img src=\"http://kubernetes.io/img/warning.png\" alt=\"WARNING\"\n+     width=\"25\" height=\"25\">\n+\n+<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>\n+\n+If you are using a released version of Kubernetes, you should\n+refer to the docs that go with that version.\n+\n+<strong>\n+The latest 1.0.x release of this document can be found\n+[here](http://releases.k8s.io/release-1.0/docs/proposals/metrics-plumbing.md).\n+\n+Documentation for other releases can be found at\n+[releases.k8s.io](http://releases.k8s.io).\n+</strong>\n+--\n+\n+<!-- END STRIP_FOR_RELEASE -->\n+\n+<!-- END MUNGE: UNVERSIONED_WARNING -->\n+\n+# Resource Usage Metrics plumbing in Kubernetes\n+\n+**Author**: Vishnu Kannan (@vishh)\n+\n+**Status**: Draft proposal; some parts are already implemented\n+\n+* This document presents a design for handling container metrics in Kubernetes clusters*\n+\n+## Motivation\n+\n+Resource usage metrics are critical for various reasons:\n+* Monitor and maintain the health of the cluster and user applications.\n+* Improve the efficiency of the cluster by making more optimal scheduling decisions and enabling components like auto-scalers.\n+\n+There are multiple types of metrics that describe the state of a container.\n+Numerous strategies exist to aggregate these metrics from containers.\n+There are a variety of storage backends that can handle metrics.\n+\n+This document presents a design to abstract out collection and storage backends, and provide stable Kubernetes APIs that can be consumed by users and other cluster components.\n+\n+## Introduction\n+\n+Container metrics can be of two types.\n+\n+1. `Compute resource metrics` refers to compute resources being used by a container. Ex.: CPU, Memory, Network, File-system\n+2. `Service metrics` refers to container app specific metrics. Ex: QPS, query latency, etc.\n+\n+Metrics can be collected either for cluster components or for user containers.\n+\n+[cAdvisor](https://github.com/google/cadvisor) is a node level container metrics aggregator that is built into the kubelet. cAdvisor can collect both types of metrics, although the support for service metrics is limited at this point. cAdvisor collects metrics for both system components and user containers.\n+\n+[heapster](https://github.com/kubernetes/heapster) is a cluster level metrics aggregator that is run by default on most Kubernetes cluster. Heapster aggregates all the metrics exposed by cAdvisor from the nodes. Heapster has a pluggable storage backend. It supports the following timeseries storage backends - InfluxDB, Google Cloud Monitoring and Hawkular.\n+Heapster builds a model of the cluster and can aggregate metrics across pods, nodes, namespaces and the entire cluster. It exposes this data via [REST endpoints](https://github.com/kubernetes/heapster/blob/master/docs/model.md#api-documentation).\n+\n+Metrics data will be consumed by many different clients - scheduler, horizontal and vertical pod auto scalers, initial pod limits controller, kubectl, web consoles, cluster management software, etc.\n+\n+Storage backends can be shared for both monitoring of clusters and powering advanced cluster features.\n+\n+## Goals\n+\n+* Abstract out timeseries storage backends from Kubernetes components.\n+* Provide stable Kubernetes Metrics APIs that other components can consume.\n+\n+#### Non Goals\n+\n+* Requiring users to run a specific storage backend.\n+* Compatibility with other node level metrics aggregator. cAdvisor should be able to provide all the metrics.\n+\n+## Design\n+\n+The basic idea is to evolve heapster to serve Metrics APIs which can then be consumed by other cluster components.\n+Heapster will be run in all clusters by default. Heapster's memory usage is proportional to the number of containers in the cluster and so it should be possible to run heapster by default even on small development or test clusters.\n+A cluster administrator will have to either run one of the supported storage backends or write a new storage plugin in heapster to support custom storage backends.\n+Heapster will manage versioning and storage schema for the various storage backends it supports.\n+Heapster APIs will be exposed as Kubernetes APIs once the apiserver supports [dynamic API plugins](https://github.com/kubernetes/kubernetes/issues/991).\n+\n+Heapster stores a days worth of historical metrics. Heapster will fetch data from storage backends on-demand to serve metrics that are older than a day. Setting [initial pod resources](initial-resources.md) requires access to metrics from the past 30 days.\n+\n+To make heapster APIs compatible with Kubernetes API requirements, heapster will have to incorporate the API server library. Until that is possible, we will run a secondary API server binary that supports the metrics APIs being consumed by other components. The initial plan is to use etcd to store the most recent metrics. Eventually, we would like to get rid of etcd for metrics and make heapster act as a backend to the api-server.\n+\n+This is the current plan for supporting node and pod metrics API as described in this [proposal](compute-resource-metrics-api.md).\n+\n+There will be proposals in the future for adding more heapster metrics APIs in Kubernetes.\n+\n+## Implementation plan\n+\n+Heapster has an in-build model of a cluster and can expose the average, 95%ile and max of compute resource metrics for containers, pods, nodes, namespaces and entire cluster.\n+However the existing APIs are not suitable for Kubernetes components.\n+The metrics are stored in a rolling window. Adding support for other percentiles should be straightforward.\n+Heapster is currently stateless and so it will loose its history upon restart.\n+Some of the specific work items include,\n+\n+1. Improve the existing API schema to be Kubernetes compatible ([Related issue](https://github.com/kubernetes/heapster/issues/476))\n+2. Add support for fetching historical data from storage backends.\n+3. Fetch historical metrics from storage backends upon restarts to pre-populate the internal model.\n+4. Add support for image based aggregation.\n+5. Add support for label queries.\n+6. Expose heapster APIs via a Kubernetes service until the primary API server can handle plugins.\n+\n+### Non Goals","path":"docs/proposals/metrics-plumbing.md","position":113,"original_position":113,"commit_id":"63a44952efe92de9274b2f7788301a563e7be267","original_commit_id":"63a44952efe92de9274b2f7788301a563e7be267","user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"body":"You already have a non goal at line 77 above, can we combine them together?","created_at":"2015-09-29T18:28:30Z","updated_at":"2015-09-29T18:28:30Z","html_url":"https://github.com/kubernetes/kubernetes/pull/13540#discussion_r40709317","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/40709317"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/13540#discussion_r40709317"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540","id":44061323,"html_url":"https://github.com/kubernetes/kubernetes/pull/13540","diff_url":"https://github.com/kubernetes/kubernetes/pull/13540.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/13540.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13540","number":13540,"state":"open","locked":false,"title":"Adding a proposal for handling metrics.","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"body":"I am posting a WIP to gain some initial feedback on the overall design. \r\nThis is critical for designing auto-scalers and improving the current scheduler.\r\n\r\nKindly do not nit-pick just yet. Some of the links might be broken.\r\n\r\ncc @piosz @fgrzadkowski @mwielgus @smarterclayton @bgrant0607 @lavalamp ","created_at":"2015-09-03T01:29:00Z","updated_at":"2015-09-29T18:28:30Z","closed_at":null,"merged_at":null,"merge_commit_sha":"dde53c8e18e800e337064d6692cbc2df0c4cd947","assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/13540/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/63a44952efe92de9274b2f7788301a563e7be267","head":{"label":"vishh:metrics-plumbing","ref":"metrics-plumbing","sha":"63a44952efe92de9274b2f7788301a563e7be267","user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"repo":{"id":26461931,"name":"kubernetes","full_name":"vishh/kubernetes","owner":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/vishh/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/vishh/kubernetes","forks_url":"https://api.github.com/repos/vishh/kubernetes/forks","keys_url":"https://api.github.com/repos/vishh/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/vishh/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/vishh/kubernetes/teams","hooks_url":"https://api.github.com/repos/vishh/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/vishh/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/vishh/kubernetes/events","assignees_url":"https://api.github.com/repos/vishh/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/vishh/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/vishh/kubernetes/tags","blobs_url":"https://api.github.com/repos/vishh/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/vishh/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/vishh/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/vishh/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/vishh/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/vishh/kubernetes/languages","stargazers_url":"https://api.github.com/repos/vishh/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/vishh/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/vishh/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/vishh/kubernetes/subscription","commits_url":"https://api.github.com/repos/vishh/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/vishh/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/vishh/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/vishh/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/vishh/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/vishh/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/vishh/kubernetes/merges","archive_url":"https://api.github.com/repos/vishh/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/vishh/kubernetes/downloads","issues_url":"https://api.github.com/repos/vishh/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/vishh/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/vishh/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/vishh/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/vishh/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/vishh/kubernetes/releases{/id}","created_at":"2014-11-11T00:05:03Z","updated_at":"2014-12-18T20:48:07Z","pushed_at":"2015-09-29T17:54:36Z","git_url":"git://github.com/vishh/kubernetes.git","ssh_url":"git@github.com:vishh/kubernetes.git","clone_url":"https://github.com/vishh/kubernetes.git","svn_url":"https://github.com/vishh/kubernetes","homepage":null,"size":110981,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"ca3ae476fc54f9455fb64958541553003759276d","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-09-29T17:57:26Z","pushed_at":"2015-09-29T18:25:04Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":437790,"stargazers_count":10490,"watchers_count":10490,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":2618,"mirror_url":null,"open_issues_count":1972,"forks":2618,"open_issues":1972,"watchers":10490,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/13540"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/13540"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/13540/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/13540/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/63a44952efe92de9274b2f7788301a563e7be267"}}}}