{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9581","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9581/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9581/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9581/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/9581","id":87015639,"number":9581,"title":"Prometheus stats holding onto a fair amount of memory (potential unbounded rest stats)","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-06-10T15:51:04Z","updated_at":"2015-06-10T17:06:09Z","closed_at":null,"body":"Right now the faceted search stats from cadvisor, the rest client, and the apiserver are holding on to a fair amount of memory:\r\n\r\nhttps://github.com/openshift/origin/issues/3053#issue-87005876\r\n\r\nPrometheus is holding about 225mb, 35 from the rest client, probably 40-50 from the apiserver (we log more summary data there), and then the bulk is probably from cadvisor.\r\n\r\nThe growth is proportional to the number of containers running (cadvisor) but should be capped out for apiserver (because we only expose a limited number of endpoints). \r\n\r\nHowever REST is capturing the URL, which means that every URL endpoint that the API server uses to talk to itself (including every unique named resource fetched by a controller) creates one new summary.  I think we either need to:\r\n\r\na. limit the cardinality of the rest call\r\nb. age out the stat points (does prometheus support this?)\r\nc. garbage collect the stats on REST\r\n\r\nThis applies to both the kubelet and controller manager, since both hold a long running rest client that is used to fetch individual resources.\r\n\r\n@fgrzadkowski "},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/110836913","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/9581#issuecomment-110836913","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9581","id":110836913,"user":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"created_at":"2015-06-10T17:06:09Z","updated_at":"2015-06-10T17:06:09Z","body":"Isn't this already behind a debug flag? https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/server.go#L124/ListerAnd\r\n"}}