{"action":"opened","issue":{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/60","labels_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/60/comments","events_url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/issues/60/events","html_url":"https://github.com/cloudfoundry-incubator/cf-abacus/issues/60","id":108846745,"number":60,"title":"CPU Usage Reporting","user":{"login":"georgethebeatle","id":9860469,"avatar_url":"https://avatars.githubusercontent.com/u/9860469?v=3","gravatar_id":"","url":"https://api.github.com/users/georgethebeatle","html_url":"https://github.com/georgethebeatle","followers_url":"https://api.github.com/users/georgethebeatle/followers","following_url":"https://api.github.com/users/georgethebeatle/following{/other_user}","gists_url":"https://api.github.com/users/georgethebeatle/gists{/gist_id}","starred_url":"https://api.github.com/users/georgethebeatle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/georgethebeatle/subscriptions","organizations_url":"https://api.github.com/users/georgethebeatle/orgs","repos_url":"https://api.github.com/users/georgethebeatle/repos","events_url":"https://api.github.com/users/georgethebeatle/events{/privacy}","received_events_url":"https://api.github.com/users/georgethebeatle/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/labels/Discussion","name":"Discussion","color":"d4c5f9"},{"url":"https://api.github.com/repos/cloudfoundry-incubator/cf-abacus/labels/open%20for%20contribution","name":"open for contribution","color":"009800"}],"state":"open","locked":false,"assignee":{"login":"georgethebeatle","id":9860469,"avatar_url":"https://avatars.githubusercontent.com/u/9860469?v=3","gravatar_id":"","url":"https://api.github.com/users/georgethebeatle","html_url":"https://github.com/georgethebeatle","followers_url":"https://api.github.com/users/georgethebeatle/followers","following_url":"https://api.github.com/users/georgethebeatle/following{/other_user}","gists_url":"https://api.github.com/users/georgethebeatle/gists{/gist_id}","starred_url":"https://api.github.com/users/georgethebeatle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/georgethebeatle/subscriptions","organizations_url":"https://api.github.com/users/georgethebeatle/orgs","repos_url":"https://api.github.com/users/georgethebeatle/repos","events_url":"https://api.github.com/users/georgethebeatle/events{/privacy}","received_events_url":"https://api.github.com/users/georgethebeatle/received_events","type":"User","site_admin":false},"milestone":null,"comments":0,"created_at":"2015-09-29T11:02:02Z","updated_at":"2015-09-29T11:02:02Z","closed_at":null,"body":"# CPU Usage Reporting\r\n\r\n## Background\r\n\r\nCurrently we report two CF apps measurements - memory usage and instance number. CPU usage is usually tightly connected with memory usage. However the current CF bridge in Abacus does not provide CPU usage measurements.\r\n\r\nThis document reviews the existing solutions and proposes several approaches to add CPU usage measurements.\r\n\r\n## Existing solutions overview\r\n\r\nExisting PaaS offerings do not generally charge by CPU. IaaS Platforms like Amazon charge based on the instance size, which can include number of CPUs, but in the IaaS case these CPUs are committed to the customer.\r\n\r\n[Google App Engine](https://cloud.google.com/appengine/pricing) charge based on number of instances, network traffic, API calls, and storage in GB per month.\r\n\r\n[Heroku](https://www.heroku.com/pricing) base their pricing on container sizes but do not mention CPU.\r\n\r\n[Pivotal Web Services](https://run.pivotal.io/pricing/) charge per instance per time based on memory quota.\r\n\r\n[Engine Yard](https://www.engineyard.com/pricing) define several plans differing on number of instances allowed, support level and connectivity to your backend - no memory or CPU quotas are mentioned.\r\n\r\n## Possible Approaches:\r\n\r\n### Extend app metadata to include CPU limit that garden set on the container\r\nPros:\r\n - The CF Bridge will be able to get all apps with one request and report the CPU limit for each app\r\n\r\nCons:\r\n- All apps now have the same CPU limit. If we want CPU to be meaningful CF will have to provide support for scaling the CPU limit\r\n\r\n### Each app runner to regularly report current stats in an event queue/database\r\nThe runner (DEA or Diego) can report the stats to a dedicated queue or database.\r\n\r\nPros:\r\n- Ease of use from the POV of CF Bridge\r\n- Cheapness in terms of requests - the Bridge will be able to poll all apps usage with one request\r\n\r\nCons:\r\n- An event queue with regular stats for each app will not scale well in terms of storage and connections\r\n- The CF architecture may be misfit for this approach\r\n\r\n### CF Bridge to regularly poll the stats for each application\r\nPros:\r\n- Easy to implement\r\n\r\nCons:\r\n- Does not scale in terms of number of requests\r\n\r\n### Log CPU and memory usage\r\nLog CPU and memory usage in special format. The usage stats can then be processed/parsed in loggregator and send the data to either CF-bridge or to Abacus directly.\r\n\r\nPros:\r\n- Re-use existing log stream infrastructure\r\n\r\nCons:\r\n- Connections intensive approach\r\n\r\n## Implementation considerations\r\n\r\n### The stats endpoint\r\n\r\nCurrently CPU, memory and disk usage of an app at a given point in time can be obtained fro the `/v2/apps/<app_guid>/stats` endpoint. Here's an example response:\r\n\r\n```\r\n{\r\n   \"0\": {\r\n      \"state\": \"RUNNING\",\r\n      \"stats\": {\r\n         \"name\": \"a7\",\r\n         \"uris\": [\r\n            \"a7.10.244.0.34.xip.io\"\r\n         ],\r\n         \"host\": \"10.244.0.26\",\r\n         \"port\": 61032,\r\n         \"uptime\": 408282,\r\n         \"mem_quota\": 268435456,\r\n         \"disk_quota\": 1073741824,\r\n         \"fds_quota\": 16384,\r\n         \"usage\": {\r\n            \"time\": \"2015-09-28 08:57:50 +0000\",\r\n            \"cpu\": 0.00023182570557697148,\r\n            \"mem\": 2600960,\r\n            \"disk\": 0\r\n         }\r\n      }\r\n   }\r\n}\r\n```\r\n\r\nThis endpoint is implemented in CC's [StatsController](https://github.com/cloudfoundry/cloud_controller_ng/blob/master/app/controllers/runtime/stats_controller.rb#L10). The implementation basically retrieves the app's stats from the node(s) it is running on. This retrieval is delegated to one of two implementations - the [DEA](https://github.com/cloudfoundry/cloud_controller_ng/blob/master/lib/cloud_controller/dea/instances_reporter.rb#L33) and [Diego cell](https://github.com/cloudfoundry/cloud_controller_ng/blob/master/lib/cloud_controller/diego/instances_reporter.rb#L97) implementations.\r\n\r\n### CC Database\r\n\r\nThe `events` or `app_usage_events` tables in CCDB could accommodate periodic events. However two things are missing in current implementation:\r\n- an API in CC that writes stats events\r\n- a component in the DEA/cell should be responsible for pushing these stats periodically to this API\r\n\r\n### External stats database\r\n\r\nThe CPU stats will use a significant amount of storage. They will most probably consume a lot of connections as well. Therefore we might want to use a dedicated database such as Hadoop to handle big-scale CF deployments.\r\n\r\n# Conclusion\r\n\r\nAll of the above approaches require a lot of connections or storage resources in order to provide CPU usage reports.\r\n\r\nThe CPU stats are vital part of the monitoring infrastructure since it helps the developers and operators to triage problems. It only makes sense to support CPU metrics if they are one of the most important components for billing. However since the applications are run in containers that effectively cap the CPU, this is rarely so, as also seen from the existing solutions.\r\n"}}