{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/4188","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/4188/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/4188/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/4188/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/4188","id":56737854,"number":4188,"title":"API status for pods is flaky","user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/bug","name":"kind/bug","color":"ff0000"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/7","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/7/labels","id":969319,"number":7,"title":"P1 Issue Fix-it","description":"Issues we're targeting for a dedicated fix-it milestone.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":4,"state":"open","created_at":"2015-02-06T20:29:31Z","updated_at":"2015-02-11T19:20:13Z","due_on":"2015-02-12T08:00:00Z","closed_at":null},"comments":29,"created_at":"2015-02-05T22:11:14Z","updated_at":"2015-02-11T19:36:10Z","closed_at":null,"body":"I have a cluster with ~130 pods.  Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything.\r\n\r\nthockin@freakshow kubernetes master /$ ./cluster/kubectl.sh get pods -l app=hostnames | awk '{print $(NF)}' | sort | uniq -c\r\nRunning: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames\r\n    127 Running\r\n      1 STATUS\r\n      6 Unknown\r\n\r\nthockin@freakshow kubernetes master /$ ./cluster/kubectl.sh get pods -l app=hostnames | awk '{print $(NF)}' | sort | uniq -c\r\nRunning: ./cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames\r\n     82 Running\r\n      1 STATUS\r\n     51 Unknown\r\n\r\nNothing changed in between these calls."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/73948550","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/4188#issuecomment-73948550","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/4188","id":73948550,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-02-11T19:36:10Z","updated_at":"2015-02-11T19:36:10Z","body":"Assign the issue to me ...\r\n\r\nI didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. \r\n\r\n```\r\nI0211 18:44:31.732281   27704 kubelet.go:1559] Query docker container info failed with error: dial unix /var/run/docker.sock: resource temporarily unavailable\r\nI0211 18:44:31.732359   27704 server.go:351] GET /api/v1beta1/podInfo?podID=hostnames-bxfcy&podNamespace=default: (3.832648315s) 500\r\n```\r\n\r\nThe proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this. \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n"}}