{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3950","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3950/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3950/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3950/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3950","id":55962890,"number":3950,"title":"apiserver develops a dangling pod w/o name","user":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/hosting","name":"area/hosting","color":"0052cc"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/reliability","name":"area/reliability","color":"a10d01"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/component/apiserver","name":"component/apiserver","color":"0052cc"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/bug","name":"kind/bug","color":"ff0000"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/7","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/7/labels","id":969319,"number":7,"title":"P1 Issue Fix-it","description":"Issues we're targeting for a dedicated fix-it milestone.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":3,"state":"open","created_at":"2015-02-06T20:29:31Z","updated_at":"2015-02-10T01:24:07Z","due_on":"2015-02-12T08:00:00Z","closed_at":null},"comments":14,"created_at":"2015-01-29T21:54:26Z","updated_at":"2015-02-10T01:55:10Z","closed_at":null,"body":"@davidopp After creating and deleting a bunch of unique-named pause pods, the master ended up with a zombie? pod entry:\r\n\r\n> % gcloud preview container pods --cluster=${PROJECT} list\r\n> POD                 IP                  CONTAINER(S)        IMAGE(S)            HOST                                                               LABELS              STATUS\r\n>                                                                                 k8s-prod5-node-1.c.mohr-instance-manager.internal/130.211.164.97   <none>              Unknown\r\n\r\nHere's the 1 node's view:\r\n> node-1% docker ps\r\n> CONTAINER ID        IMAGE                   COMMAND               CREATED             STATUS              PORTS                      NAMES\r\n> 9558807f5bd4        kubernetes/pause:go     \"/pause\"              About an hour ago   Up About an hour                               k8s_pause-pod-a3c58ab1.fd6cd67a_pause-pod-a3c58ab1.default.etcd_2896301e-a7f7-11e4-a2ec-42010af0de1f_70214b24   \r\n> f4a4f659dc37        kubernetes/pause:go     \"/pause\"              About an hour ago   Up About an hour    0.0.0.0:15670->15670/tcp   k8s_net.e468acc9_pause-pod-a3c58ab1.default.etcd_2896301e-a7f7-11e4-a2ec-42010af0de1f_286dfd30                  \r\n> 3eb6b397cbfe        google/cadvisor:0.8.0   \"/usr/bin/cadvisor\"   About an hour ago   Up About an hour                               k8s_cadvisor.54c805f9_cadvisor-agent.default.file_cadvisormanifes12uqn2ohido76855gdecd9roadm7l0_a7d38033        \r\n> 4e76f64b66ff        kubernetes/pause:go     \"/pause\"              About an hour ago   Up About an hour    0.0.0.0:4194->8080/tcp     k8s_net.1509ae25_cadvisor-agent.default.file_cadvisormanifes12uqn2ohido76855gdecd9roadm7l0_706122ee             \r\n\r\nAnd etcd's db dump on the master VM:\r\n% curl -s -L http://127.0.0.1:4001/v2/keys/registry/pods/default | python -m json.tool\r\n{\r\n    \"action\": \"get\", \r\n    \"node\": {\r\n        \"createdIndex\": 26, \r\n        \"dir\": true, \r\n        \"key\": \"/registry/pods/default\", \r\n        \"modifiedIndex\": 26, \r\n        \"nodes\": [\r\n            {\r\n                \"createdIndex\": 2266, \r\n                \"key\": \"/registry/pods/default/pause-pod-3da11ec1\", \r\n                \"modifiedIndex\": 2266, \r\n                \"value\": \"{\\\"kind\\\":\\\"Pod\\\",\\\"creationTimestamp\\\":null,\\\"apiVersion\\\":\\\"v1beta1\\\",\\\"desiredState\\\":{\\\"manifest\\\":{\\\"version\\\":\\\"v1beta2\\\",\\\"id\\\":\\\"\\\",\\\"volumes\\\":null,\\\"containers\\\":null,\\\"restartPolicy\\\":{}}},\\\"currentState\\\":{\\\"manifest\\\":{\\\"version\\\":\\\"\\\",\\\"id\\\":\\\"\\\",\\\"volumes\\\":null,\\\"containers\\\":null,\\\"restartPolicy\\\":{}},\\\"host\\\":\\\"k8s-prod5-node-1.c.mohr-instance-manager.internal\\\"}}\"\r\n            }\r\n        ]\r\n    }\r\n}\r\n\r\nPart of long-running masters: #3936"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/73630268","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3950#issuecomment-73630268","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3950","id":73630268,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2015-02-10T01:55:10Z","updated_at":"2015-02-10T01:55:10Z","body":"I suspect this is an etcd issue. As mentioned in the previous comment, pod was still being scheduled after the pod has supposedly been deleted. Here is a snippet of such a pod:\r\n\r\n```\r\nkube-scheduler.log:I0210 01:16:41.316617    5233 factory.go:136] glog.v2 --> About to try and schedule pod pause-pod-1616b21\r\nkube-scheduler.log:I0210 01:16:41.316668    5233 scheduler.go:71] Attempting to schedule: &{{ } {pause-pod-1616b21  default /api/v1beta1/pods/pause-pod-1616b21 78111d6c-b0c2-11e4-a4b9-42010af0b14c 4043 2015-02-10 01:16:41 +0000 UTC map[name:pause-pod-1616b21] map[]} {[] [{pause-container kubernetes/pause []  [{ 12490 12490 TCP }] [] {map[]} [] <nil> <nil> /dev/termination-log false IfNotPresent {[] []}}] {0xaec2a0 <nil> <nil>} ClusterFirst map[] } {Pending     map[]}}\r\nkube-scheduler.log:I0210 01:16:41.317215    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-1.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:16:41.317244    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-2.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:16:41.317622    5233 factory.go:274] Attempting to bind pause-pod-1616b21 to kubernetes-minion-1.c.cloud-containers-yjhong.internal\r\nkube-scheduler.log:I0210 01:18:01.785769    5233 factory.go:136] glog.v2 --> About to try and schedule pod pause-pod-1616b21\r\nkube-scheduler.log:I0210 01:18:01.785796    5233 scheduler.go:71] Attempting to schedule: &{{ } {pause-pod-1616b21  default /api/v1beta1/pods/pause-pod-1616b21?namespace=default 78111d6c-b0c2-11e4-a4b9-42010af0b14c 4022 2015-02-10 01:16:41 +0000 UTC map[name:pause-pod-1616b21] map[]} {[] [{pause-container kubernetes/pause []  [{ 12490 12490 TCP }] [] {map[]} [] <nil> <nil> /dev/termination-log false IfNotPresent {[] []}}] {0xaec2a0 <nil> <nil>} ClusterFirst map[] } {Running   130.211.162.42 10.244.0.180 map[POD:{{<nil> 0xc2082c3ae0 <nil>} 0 10.244.0.180 kubernetes/pause:latest docker://6c4579af347b649857e915521132f15a06186d73faa62145e3eeeb6be0e97c27 docker://67034a7b2350dd1dd56605691dfd8667431a5affb3dde4ce40ba4bd33529a58b} pause-container:{{<nil> 0xc2082c3ca0 <nil>} 0  kubernetes/pause docker://6c4579af347b649857e915521132f15a06186d73faa62145e3eeeb6be0e97c27 docker://ef2ccf0757a397928b09feff5f0dbe4de865ac5c4a450eb1552b28f356a405a9}]}}\r\nkube-scheduler.log:I0210 01:18:01.786140    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-2.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:18:01.786187    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-1.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:18:01.786267    5233 factory.go:274] Attempting to bind pause-pod-1616b21 to kubernetes-minion-2.c.cloud-containers-yjhong.internal\r\nkube-apiserver.log:I0210 01:20:56.586529    5045 handlers.go:108] GET /api/v1beta1/pods/pause-pod-1616b21?namespace=default: (2.512143ms) 200\r\nkube-apiserver.log:I0210 01:20:56.798084    5045 handlers.go:108] DELETE /api/v1beta1/pods/pause-pod-1616b21?namespace=default: (168.413613ms) 200\r\nkube-scheduler.log:I0210 01:32:01.628871    5233 factory.go:136] glog.v2 --> About to try and schedule pod pause-pod-1616b21\r\nkube-scheduler.log:I0210 01:32:01.628893    5233 scheduler.go:71] Attempting to schedule: &{{ } {pause-pod-1616b21  default /api/v1beta1/pods/pause-pod-1616b21?namespace=default 78111d6c-b0c2-11e4-a4b9-42010af0b14c 4022 2015-02-10 01:16:41 +0000 UTC map[name:pause-pod-1616b21] map[]} {[] [{pause-container kubernetes/pause []  [{ 12490 12490 TCP }] [] {map[]} [] <nil> <nil> /dev/termination-log false IfNotPresent {[] []}}] {0xaec2a0 <nil> <nil>} ClusterFirst map[] } {Unknown     map[]}}\r\nkube-scheduler.log:I0210 01:32:01.629157    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-1.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:32:01.629259    5233 priorities.go:67] pause-pod-1616b21 -> kubernetes-minion-2.c.cloud-containers-yjhong.internal: Least Requested Priority, AbsoluteRequested: (0, 0) / (1000, 4026531840) Score: (10, 10)\r\nkube-scheduler.log:I0210 01:32:01.629317    5233 factory.go:274] Attempting to bind pause-pod-1616b21 to kubernetes-minion-2.c.cloud-containers-yjhong.internal\r\n```\r\n\r\nNote that the delete request returned OK (200).\r\n```DELETE /api/v1beta1/pods/pause-pod-1616b21?namespace=default: (168.413613ms) 200```\r\nHowever, the key lives and scheduler even attempted to schedule it again (while the pod was in the unknown status).\r\n\r\nThe content of the key/pod in etcd showed that most of the content has been cleared.\r\n```\r\netcdctl --peers=10.240.177.76:4001 get registry/pods/default/pause-pod-1616b21\r\n{\"kind\":\"Pod\",\"creationTimestamp\":null,\"apiVersion\":\"v1beta1\",\"desiredState\":{\"manifest\":{\"version\":\"v1beta2\",\"id\":\"\",\"volumes\":null,\"containers\":null,\"restartPolicy\":{}}},\"currentState\":{\"manifest\":{\"version\":\"\",\"id\":\"\",\"volumes\":null,\"containers\":null,\"restartPolicy\":{}},\"host\":\"kubernetes-minion-2.c.cloud-containers-yjhong.internal\"}}\r\n```\r\n\r\nI may add a check after the deletion to make sure that the key **was deleted** when etcd claimed to have done so."}}