{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15835","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15835/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15835/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15835/events","html_url":"https://github.com/kubernetes/kubernetes/issues/15835","id":112109232,"number":15835,"title":"ReplicationController failed to notice Pod Eviction","user":{"login":"gmarek","id":1703842,"avatar_url":"https://avatars.githubusercontent.com/u/1703842?v=3","gravatar_id":"","url":"https://api.github.com/users/gmarek","html_url":"https://github.com/gmarek","followers_url":"https://api.github.com/users/gmarek/followers","following_url":"https://api.github.com/users/gmarek/following{/other_user}","gists_url":"https://api.github.com/users/gmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/gmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmarek/subscriptions","organizations_url":"https://api.github.com/users/gmarek/orgs","repos_url":"https://api.github.com/users/gmarek/repos","events_url":"https://api.github.com/users/gmarek/events{/privacy}","received_events_url":"https://api.github.com/users/gmarek/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI","name":"team/CSI","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-19T09:39:54Z","updated_at":"2015-10-19T18:39:12Z","closed_at":null,"body":"TL;DR: I don't have a reliable repro yet, it might be a one-off flake. If it's not it can be serious, as we can permanently kill a Pod.\r\n\r\nI was running 250 Node tests (denisty 30') looking at heapster resource consumption. The first run was close to the limit (3G), so the second one was likely to end in OOM. To nobody's surprise it did. Because it was eating by far most memory it was (most likely) killed by the sys oom killer, which, for some reason ended up in:\r\n```\r\nMon, 19 Oct 2015 11:11:11 +0200   Mon, 19 Oct 2015 11:11:11 +0200   1         heapster-v10-j8a4y                                  Pod                                                NodeControllerEviction   {controllermanager }                    Marking for deletion Pod heapster-v10-j8a4y from Node e2e-test-gmarek-minion-9ei\r\n```\r\n(Node itself was healthy at the moment)\r\n\r\nSmall problem is that the Pod is not running anywhere now:\r\n```\r\nkubectl --namespace=kube-system get pod | grep heapster\r\n```\r\nreturns an empty result.\r\n\r\nThe real question is why this is the case:\r\n```\r\n$:~/go/src/k8s.io/kubernetes$ kubectl --namespace=kube-system describe rc heapster-v10\r\nName:\t\theapster-v10\r\nNamespace:\tkube-system\r\nImage(s):\tgcr.io/google_containers/heapster:v0.18.2\r\nSelector:\tk8s-app=heapster,version=v10\r\nLabels:\t\tk8s-app=heapster,kubernetes.io/cluster-service=true,version=v10\r\nReplicas:\t1 current / 1 desired\r\nPods Status:\t0 Running / 0 Waiting / 0 Succeeded / 0 Failed\r\nNo events.\r\n```\r\n\r\ncc @wojtek-t @fgrzadkowski @davidopp @lavalamp @dchen1107 "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/149308128","html_url":"https://github.com/kubernetes/kubernetes/issues/15835#issuecomment-149308128","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/15835","id":149308128,"user":{"login":"gmarek","id":1703842,"avatar_url":"https://avatars.githubusercontent.com/u/1703842?v=3","gravatar_id":"","url":"https://api.github.com/users/gmarek","html_url":"https://github.com/gmarek","followers_url":"https://api.github.com/users/gmarek/followers","following_url":"https://api.github.com/users/gmarek/following{/other_user}","gists_url":"https://api.github.com/users/gmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/gmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmarek/subscriptions","organizations_url":"https://api.github.com/users/gmarek/orgs","repos_url":"https://api.github.com/users/gmarek/repos","events_url":"https://api.github.com/users/gmarek/events{/privacy}","received_events_url":"https://api.github.com/users/gmarek/received_events","type":"User","site_admin":false},"created_at":"2015-10-19T18:39:12Z","updated_at":"2015-10-19T18:39:12Z","body":"It never reappeared. I kept the cluster running for at least half an hour, but nothing happened. I have logs from controller manager if you're interested."}}