{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15671","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15671/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15671/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15671/events","html_url":"https://github.com/kubernetes/kubernetes/issues/15671","id":111510408,"number":15671,"title":"Incorrect state for pods with crashlooping containers","user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-14T22:49:23Z","updated_at":"2015-10-15T17:26:32Z","closed_at":null,"body":"With the enhancement of container restart backoff, pod phase may not progress monotonically.\r\n\r\nA pod is considered running if at least one container in the pod has been started. If the container happens to be crashlooping, the restart backoff would eventually kick in. From that point on, the container status becomes waiting, and [kubelet determines the phase](https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go#L2497) as **pending**.\r\n\r\nOnce the pod transitions from `Running` to `Pending`, it's impossible to retrieve logs from the crashlooping container using kubectl. This prevents the users from debugging the cause of container failures.\r\n```console\r\n$ kubectl logs -p heapster-v10-3ayyj --namespace=kube-system\r\nError from server: Internal error occurred: Pod \"heapster-v10-3ayyj\" in namespace \"kube-system\" : pod is not in 'Running', 'Succeeded' or 'Failed' state - State: \"Pending\"\r\n```\r\n\r\nAn easy fix is to check `LastTerminationState` of a container before determining the pod phase.\r\n\r\n/cc @samsabed "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/148465596","html_url":"https://github.com/kubernetes/kubernetes/issues/15671#issuecomment-148465596","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15671","id":148465596,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2015-10-15T17:26:32Z","updated_at":"2015-10-15T17:26:32Z","body":"> @yujuhong so change it to stay in running when one of the containers is not running ?\r\n\r\nHm...once the pod has transitioned to \"running\", it should stay as \"running\" or transition to a final phase based on the restart policy. It should never go back to \"pending\" though.\r\nThis wasn't a problem before we had restart backoff, but now there could be a time when all containers in a pod are in the \"waiting\" status, and none of them are running.\r\n\r\nI think we should treat the pod as running as long as every container in the pod matches one of the following conditions (@bgrant0607, correct me if I am wrong):\r\n 1. container is running.\r\n 2. container is terminated.\r\n 3. container is waiting (due to backoff) but container has been run before (non-nil `LastTerminationState`)\r\n\r\nIf *all* containers are terminated, we may transition the pod into phase Succeeded or Failed based on the termination status and the restart policy.\r\n\r\nOne exception I can think of is the \"failed to start\" containers, which appear as terminated containers. @bgrant0607, if kubelet has tried starting all containers in the pod, but all of the failed to run. Can we consider the pod running?"}}