{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14652","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14652/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14652/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14652/events","html_url":"https://github.com/kubernetes/kubernetes/issues/14652","id":108647877,"number":14652,"title":"GCE - after autoscaler-initiated cluster resize the removed node is still present in kubectl get","user":{"login":"mwielgus","id":11994812,"avatar_url":"https://avatars.githubusercontent.com/u/11994812?v=3","gravatar_id":"","url":"https://api.github.com/users/mwielgus","html_url":"https://github.com/mwielgus","followers_url":"https://api.github.com/users/mwielgus/followers","following_url":"https://api.github.com/users/mwielgus/following{/other_user}","gists_url":"https://api.github.com/users/mwielgus/gists{/gist_id}","starred_url":"https://api.github.com/users/mwielgus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwielgus/subscriptions","organizations_url":"https://api.github.com/users/mwielgus/orgs","repos_url":"https://api.github.com/users/mwielgus/repos","events_url":"https://api.github.com/users/mwielgus/events{/privacy}","received_events_url":"https://api.github.com/users/mwielgus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-09-28T12:28:59Z","updated_at":"2015-09-28T14:03:44Z","closed_at":null,"body":"```\r\n$ kubectl get nodes\r\n\r\nNAME                     LABELS                                          STATUS     AGE\r\nkubernetes-minion-7h7k   kubernetes.io/hostname=kubernetes-minion-7h7k   NotReady   31m\r\nkubernetes-minion-a6a6   kubernetes.io/hostname=kubernetes-minion-a6a6   NotReady   31m\r\nkubernetes-minion-a7v8   kubernetes.io/hostname=kubernetes-minion-a7v8   Ready      31m\r\nkubernetes-minion-mfi1   kubernetes.io/hostname=kubernetes-minion-mfi1   Ready      31m\r\nkubernetes-minion-q62b   kubernetes.io/hostname=kubernetes-minion-q62b   Ready      31m\r\nkubernetes-minion-v5ai   kubernetes.io/hostname=kubernetes-minion-v5ai   Ready      31m\r\n```\r\n\r\n```\r\n$ kubectl describe node kubernetes-minion-7h7k\r\n\r\nName:\t\t\tkubernetes-minion-7h7k\r\nLabels:\t\t\tkubernetes.io/hostname=kubernetes-minion-7h7k\r\nCreationTimestamp:\tMon, 28 Sep 2015 13:56:14 +0200\r\nConditions:\r\n  Type\t\tStatus\t\tLastHeartbeatTime\t\t\tLastTransitionTime\t\t\tReason\t\t\tMessage\r\n  Ready \tUnknown \tMon, 28 Sep 2015 14:02:03 +0200 \tMon, 28 Sep 2015 14:02:46 +0200 \tNodeStatusUnknown \tKubelet stopped posting node status.\r\nAddresses:\t10.240.200.165,104.197.12.16\r\nCapacity:\r\n cpu:\t\t1\r\n memory:\t3800808Ki\r\n pods:\t\t40\r\nAllocated resources (total requests):\r\n cpu:\t\t300m\r\n memory:\t300Mi\r\n pods:\t\t3\r\nVersion:\r\n Kernel Version:\t\t3.16.0-0.bpo.4-amd64\r\n OS Image:\t\t\tDebian GNU/Linux 7 (wheezy)\r\n Container Runtime Version:\tdocker://1.8.2\r\n Kubelet Version:\t\tv1.1.0-alpha.1.1241+c53df247c8ad65-dirty\r\n Kube-Proxy Version:\t\tv1.1.0-alpha.1.1241+c53df247c8ad65-dirty\r\nPodCIDR:\t\t\t10.244.1.0/24\r\nExternalID:\t\t\t6098904938256922746\r\nPods:\t\t\t\t(3 in total)\r\n  Namespace\t\t\tName\t\t\t\t\t\t\tCPU(milliCPU)\t\t\tMemory(bytes)\r\n  kube-system\t\t\tfluentd-cloud-logging-kubernetes-minion-7h7k\t\t100 (10% of total)\t\t209715200 (5% of total)\r\n  kube-system\t\t\tkube-registry-proxy-kubernetes-minion-7h7k\t\t100 (10% of total)\t\t52428800 (1% of total)\r\n  kube-system\t\t\tkube-ui-v2-yxyna\t\t\t\t\t100 (10% of total)\t\t52428800 (1% of total)\r\nTotalResourceLimits:\r\n  CPU(milliCPU):\t\t300 (30% of total)\r\n  Memory(bytes):\t\t314572800 (8% of total)\r\nEvents:\r\n  FirstSeen\tLastSeen\tCount\tFrom\t\t\t\t\tSubobjectPath\tReason\t\tMessage\r\n  29m\t\t29m\t\t1\t{kube-proxy kubernetes-minion-7h7k}\t\t\tStarting\tStarting kube-proxy.\r\n  29m\t\t29m\t\t1\t{kubelet kubernetes-minion-7h7k}\t\t\tNodeNotReady\tNode kubernetes-minion-7h7k status is now: NodeNotReady\r\n  28m\t\t28m\t\t1\t{kube-proxy kubernetes-minion-7h7k}\t\t\tStarting\tStarting kube-proxy.\r\n  28m\t\t28m\t\t1\t{kubelet kubernetes-minion-7h7k}\t\t\tNodeNotReady\tNode kubernetes-minion-7h7k status is now: NodeNotReady\r\n  28m\t\t28m\t\t1\t{controllermanager }\t\t\t\t\tRegisteredNode\tNode kubernetes-minion-7h7k event: Registered Node kubernetes-minion-7h7k in NodeController\r\n  28m\t\t28m\t\t1\t{kubelet kubernetes-minion-7h7k}\t\t\tStarting\tStarting kubelet.\r\n  28m\t\t28m\t\t1\t{kubelet kubernetes-minion-7h7k}\t\t\tNodeReady\tNode kubernetes-minion-7h7k status is now: NodeReady\r\n  23m\t\t23m\t\t1\t{kube-proxy kubernetes-minion-7h7k}\t\t\tStarting\tStarting kube-proxy.\r\n  23m\t\t23m\t\t1\t{kubelet kubernetes-minion-7h7k}\t\t\tStarting\tStarting kubelet.\r\n  23m\t\t23m\t\t2\t{kubelet kubernetes-minion-7h7k}\t\t\tNodeNotReady\tNode kubernetes-minion-7h7k status is now: NodeNotReady\r\n  22m\t\t15s\t\t103\t{controllermanager }\t\t\t\t\tDeletingAllPods\tNode kubernetes-minion-7h7k event: Deleting all Pods from Node kubernetes-minion-7h7k.\r\n  22m\t\t1s\t\t257\t{controllermanager }\t\t\t\t\tDeletingNode\tNode kubernetes-minion-7h7k event: Deleting Node kubernetes-minion-7h7k because it's not present according to cloud provider\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/143751858","html_url":"https://github.com/kubernetes/kubernetes/issues/14652#issuecomment-143751858","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14652","id":143751858,"user":{"login":"mwielgus","id":11994812,"avatar_url":"https://avatars.githubusercontent.com/u/11994812?v=3","gravatar_id":"","url":"https://api.github.com/users/mwielgus","html_url":"https://github.com/mwielgus","followers_url":"https://api.github.com/users/mwielgus/followers","following_url":"https://api.github.com/users/mwielgus/following{/other_user}","gists_url":"https://api.github.com/users/mwielgus/gists{/gist_id}","starred_url":"https://api.github.com/users/mwielgus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwielgus/subscriptions","organizations_url":"https://api.github.com/users/mwielgus/orgs","repos_url":"https://api.github.com/users/mwielgus/repos","events_url":"https://api.github.com/users/mwielgus/events{/privacy}","received_events_url":"https://api.github.com/users/mwielgus/received_events","type":"User","site_admin":false},"created_at":"2015-09-28T14:03:44Z","updated_at":"2015-09-28T14:03:44Z","body":"And pods have the terminating status set despite expiration long gone (2h).\r\n\r\n```\r\nkubectl describe pod kube-registry-proxy-kubernetes-minion-7h7k --namespace=kube-system\r\n\r\nName:\t\t\t\tkube-registry-proxy-kubernetes-minion-7h7k\r\nNamespace:\t\t\tkube-system\r\nImage(s):\t\t\tgcr.io/google_containers/kube-registry-proxy:0.3\r\nNode:\t\t\t\tkubernetes-minion-7h7k/10.240.200.165\r\nLabels:\t\t\t\t<none>\r\nStatus:\t\t\t\tTerminating (expires Mon, 28 Sep 2015 14:02:55 +0200)\r\nTermination Grace Period:\t5231405558240s\r\nReason:\t\t\t\t\r\nMessage:\t\t\t\r\nIP:\t\t\t\t10.244.1.2\r\nReplication Controllers:\t<none>\r\nContainers:\r\n  kube-registry-proxy:\r\n    Image:\tgcr.io/google_containers/kube-registry-proxy:0.3\r\n    Limits:\r\n      memory:\t\t50Mi\r\n      cpu:\t\t100m\r\n    State:\t\tRunning\r\n      Started:\t\tMon, 28 Sep 2015 13:57:08 +0200\r\n    Ready:\t\tTrue\r\n    Restart Count:\t0\r\n    Variables:\r\n      REGISTRY_HOST:\tkube-registry.kube-system.svc.cluster.local\r\n      REGISTRY_PORT:\t5000\r\n      FORWARD_PORT:\t5000\r\nConditions:\r\n  Type\t\tStatus\r\n  Ready \tTrue \r\nNo volumes.\r\nNo events.\r\n"}}