{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16168","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16168/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16168/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16168/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16168","id":113069986,"number":16168,"title":"heapster polling stats for long-dead pods","user":{"login":"erictune","id":8105590,"avatar_url":"https://avatars.githubusercontent.com/u/8105590?v=3","gravatar_id":"","url":"https://api.github.com/users/erictune","html_url":"https://github.com/erictune","followers_url":"https://api.github.com/users/erictune/followers","following_url":"https://api.github.com/users/erictune/following{/other_user}","gists_url":"https://api.github.com/users/erictune/gists{/gist_id}","starred_url":"https://api.github.com/users/erictune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erictune/subscriptions","organizations_url":"https://api.github.com/users/erictune/orgs","repos_url":"https://api.github.com/users/erictune/repos","events_url":"https://api.github.com/users/erictune/events{/privacy}","received_events_url":"https://api.github.com/users/erictune/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/monitoring","name":"area/monitoring","color":"fef2c0"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/16","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.2-candidate","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/16/labels","id":1278500,"number":16,"title":"v1.2-candidate","description":"Things to consider for next minor release.","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":154,"closed_issues":10,"state":"open","created_at":"2015-08-29T03:54:34Z","updated_at":"2015-10-23T19:02:18Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2015-10-23T18:01:01Z","updated_at":"2015-10-23T23:39:02Z","closed_at":null,"body":"EDITED: the thing getting the stats is heapster, not fluentd (duh).\r\n\r\nheapster is asking kubelet for stats for long-dead pods, and getting 404.\r\n\r\nOn my kubelet, 64% of all kubelet log lines are 404s of stats requests by heapster.\r\n\r\nExample log lines:\r\n```\r\nI1022 00:17:30.008559    3663 server.go:1060] GET /stats/e2e-tests-job-46it6/rand-local-1snay/719df9be-7850-11e5-ad9f-42010af00002/c: (7.085453ms) 404 [[Go 1.1 package http] 10.244.2.3:46783]\r\n... 17 hours passes ....\r\nI1023 00:17:00.011542   28808 server.go:1060] GET /stats/e2e-tests-job-46it6/rand-local-1snay/719df9be-7850-11e5-ad9f-42010af00002/c: (34.632µs) 404 [[Go 1.1 package http] 10.244.2.3:47205]\r\n```\r\n\r\n10.244.2.3 is the heapster pod:\r\n```console\r\n$ kubectl get pods --namespace kube-system heapster-v10-ygqcu -o yaml | grep podIP\r\n  podIP: 10.244.2.3\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/150717364","html_url":"https://github.com/kubernetes/kubernetes/issues/16168#issuecomment-150717364","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16168","id":150717364,"user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T23:39:02Z","updated_at":"2015-10-23T23:39:02Z","body":"Fair point. If the pods are not garbage collected from the API server\nautomatically, this will be an issue. Heapster fix underway!\n\nOn Fri, Oct 23, 2015 at 4:09 PM, Eric Tune <notifications@github.com> wrote:\n\n> IMO, you should still update heapster.\n>\n> It is fine if it occasionally queries a pod after it has terminated, but\n> it should eventually notice that a pod is gone, such as by periodically\n> relisting.\n>\n> In the example I gave, Heapster was still querying pods 17 hours after\n> they had terminated.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kubernetes/kubernetes/issues/16168#issuecomment-150713001>\n> .\n>\n"}}