{"action":"closed","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8291","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8291/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8291/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8291/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/8291","id":76516035,"number":8291,"title":"Kubelet stats query fails for running containers","user":{"login":"mbforbes","id":1170062,"avatar_url":"https://avatars.githubusercontent.com/u/1170062?v=3","gravatar_id":"","url":"https://api.github.com/users/mbforbes","html_url":"https://github.com/mbforbes","followers_url":"https://api.github.com/users/mbforbes/followers","following_url":"https://api.github.com/users/mbforbes/following{/other_user}","gists_url":"https://api.github.com/users/mbforbes/gists{/gist_id}","starred_url":"https://api.github.com/users/mbforbes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbforbes/subscriptions","organizations_url":"https://api.github.com/users/mbforbes/orgs","repos_url":"https://api.github.com/users/mbforbes/repos","events_url":"https://api.github.com/users/mbforbes/events{/privacy}","received_events_url":"https://api.github.com/users/mbforbes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0-candidate","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12/labels","id":1086011,"number":12,"title":"v1.0-candidate","description":"These are candidates for inclusion to v1.0 but will be pushed to v1.0-bubble if they're triaged as not meeting the v1.0 bar as a required launch blocker.","creator":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":15,"state":"open","created_at":"2015-04-27T19:03:36Z","updated_at":"2015-05-15T21:36:10Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2015-05-14T22:48:59Z","updated_at":"2015-05-15T21:36:10Z","closed_at":"2015-05-15T21:36:10Z","body":"This is blocking #7580 (reboot test).\r\n\r\nIn running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test:\r\n\r\n```\r\nMonitoring\r\n  verify monitoring pods and all cluster nodes are available on influxdb using heapster.\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51\r\n[BeforeEach] Monitoring\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:42\r\n>>> testContext.KubeConfig: /Users/maxforbes/.kube/config\r\n[It] verify monitoring pods and all cluster nodes are available on influxdb using heapster.\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51\r\nINFO: failed to query list of pods from influxdb. Query: \"select distinct(pod_id) from /cpu.*/\", Err: expected only one series from Influxdb for query \"select distinct(pod_id) from /cpu.*/\". Got []\r\nINFO: failed to query list of pods from influxdb. Query: \"select distinct(pod_id) from /cpu.*/\", Err: expected only one series from Influxdb for query \"select distinct(pod_id) from /cpu.*/\". Got []\r\nINFO: failed to query list of pods from influxdb. Query: \"select distinct(pod_id) from /cpu.*/\", Err: expected only one series from Influxdb for query \"select distinct(pod_id) from /cpu.*/\". Got []\r\n...\r\n<lots more of these>\r\n...\r\nINFO: failed to query list of pods from influxdb. Query: \"select distinct(pod_id) from /cpu.*/\", Err: Server returned (400): Field pod_id doesn't exist in series cpu/usage_ns_cumulative\r\n\r\n• Failure [303.125 seconds]\r\nMonitoring\r\n/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:52\r\n  verify monitoring pods and all cluster nodes are available on influxdb using heapster. [It]\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51\r\n\r\n  monitoring using heapster and influxdb test failed\r\n\r\n  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:236\r\n```\r\n\r\nThe heapster logs, from running `kubectl log monitoring-heapster-controller-xxxxx heapster` ([output](https://gist.github.com/mbforbes/214b59e41d12c1e377cb)), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns):\r\n\r\n```\r\n...\r\n2015-05-14T21:43:07.429325471Z E0514 21:43:07.427620      10 kubelet.go:85] failed to get stats from kubelet url: http://10.240.117.245:10255/stats/default/kube-dns-i7v2g/14002c59-fa6e-11e4-a2fd-42010af0174f/skydns - request failed - \"500 Internal Server Error\", response: \"Internal Error: unable to find Docker container \\\"8f489c214090c89d5c2773507b2505ab47cabe73eb82e5ac5c9c875aba4d6941\\\"\\n\"\r\n2015-05-14T21:43:07.429353483Z E0514 21:43:07.427643      10 kube_pods.go:106] failed to get stats for container \"skydns\" in pod \"default\"/\"kube-dns-i7v2g\"\r\n...\r\n```\r\n\r\nThe [kubelet log on the node (massive)](https://gist.githubusercontent.com/mbforbes/64624f694de522f79047/raw/gistfile1.txt) shows a bunch of 500 errors, such as:\r\n\r\n```\r\nI0514 22:05:07.427063    2503 server.go:158] HTTP InternalServerError: Internal Error: unable to find Docker container \"8f489c214090c89d5c2773507b2505ab47cabe73eb82e5ac5c9c875aba4d6941\"\r\nI0514 22:05:07.427148    2503 server.go:636] GET /stats/default/kube-dns-i7v2g/14002c59-fa6e-11e4-a2fd-42010af0174f/skydns: (23.441891ms) 500\r\ngoroutine 4144 [running]:\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc208c9a9c0, 0x1f4)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/httplog/log.go:188 +0x9a\r\nnet/http.Error(0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a71d400, 0x72, 0x1f4)\r\n\t/usr/src/go/src/net/http/server.go:1274 +0x9a\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.(*Server).error(0xc20827ca20, 0x7eff7cbcf080, 0xc208c9a9c0, 0x7eff7cd57d80, 0xc208f92fa0)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/server.go:159 +0x213\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.(*Server).serveStats(0xc20827ca20, 0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a3709c0)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/server.go:726 +0x78f\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.(*Server).handleStats(0xc20827ca20, 0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a3709c0)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/server.go:329 +0x46\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.*Server.(github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.handleStats)Â·fm(0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a3709c0)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/server.go:85 +0x45\r\nnet/http.HandlerFunc.ServeHTTP(0xc20833cd40, 0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a3709c0)\r\n\t/usr/src/go/src/net/http/server.go:1265 +0x41\r\nnet/http.(*ServeMux).ServeHTTP(0xc2084094d0, 0x7eff7cbcf080, 0xc208c9a9c0, 0xc20a3709c0)\r\n\t/usr/src/go/src/net/http/server.go:1541 +0x17d\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/kubelet.(*Server).ServeHTTP(0xc20827ca20, 0x7eff7cd876f8, 0xc20ad308c0, 0xc20a3709c0)\r\n\t/go/src/github.com/GoogleCloudPlatform/kubernetes/_out [[Go 1.1 package http] 10.245.0.5:35262]\r\n```\r\n\r\nHowever,\r\n\r\n- the apiserver thinks that all the pods are running (`kubectl get pods` gives this [output](https://gist.github.com/mbforbes/0aded988fa727fa3cab8))\r\n- Docker thinks that the pods are running (`sudo docker ps` on that node gives this [output](https://gist.github.com/mbforbes/e860bf28a399e4a2dbdf))\r\n\r\nIn summary:\r\n- apiserver and docker think containers are running\r\n- but kubelet can't find them\r\n- this manifested in the influxdb / heapster monitoring e2e test after node reboot\r\n\r\nI've still got the cluster around if anyone would like to debug.\r\n\r\n+cc @vishh (helped me debug this (thanks!))\r\n+cc @dchen1107 (node)\r\n+cc @roberthbailey (interested in getting #7580 in, which this is blocking)"}}