{"action":"opened","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9833","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9833/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9833/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9833/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/9833","id":88573457,"number":9833,"title":"After replication controller is deleted, some nodes do not kill running containers","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0-candidate","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12/labels","id":1086011,"number":12,"title":"v1.0-candidate","description":"These are candidates for inclusion to v1.0 but will be pushed to v1.0-bubble if they're triaged as not meeting the v1.0 bar as a required launch blocker.","creator":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":57,"state":"open","created_at":"2015-04-27T19:03:36Z","updated_at":"2015-06-16T00:11:58Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2015-06-16T00:11:58Z","updated_at":"2015-06-16T00:11:58Z","closed_at":null,"body":"Discovered this while debugging with @ArtfulCoder \r\n\r\nSome nodes do not kill containers after associated replication controller is deleted.\r\n\r\n**Repro steps:**\r\n* You may use a standard GCE 4 node cluster `n1-standard-1 (1 vCPU, 3.8 GB memory)`\r\n* Download [`sleepypod.yaml`](https://gist.github.com/saad-ali/7d04ff21534729c6c42a)\r\n  * It defines a replication controller with 300 replicas with 10 containers per pod using a *tiny* container which contains a statically compiled C program that just starts up and goes to sleep for a very long time.\r\n* Execute `./cluster/kubectl.sh create -f sleepypod.yaml` to create the replica controller\r\n* Execute `./cluster/kubectl.sh describe rc sleepypod` repeatedly to check how many pods are scheduled.\r\n  * Monitor `Pods Status`\r\n  * Wait until the number of scheduled pods stops increasing (depending on the size of your cluster, you will not be able to schedule all 300 replicas, that's ok).\r\n* Call `./cluster/kubectl.sh delete -f sleepypod.yaml` to delete the replica controller\r\n* Call `./cluster/kubectl.sh get pods` verify there are no `sleepypod-...` in the response\r\n* SSH into each node machine and execute `sudo ps | grep sleepy`\r\n\r\n**Expected result:**\r\n* All nodes should have no running `sleepypod-...` containers\r\n\r\n**Actual result:**\r\n* One or more nodes will not terminate the `sleepypod:` and associated `pause` containers running on it:\r\n```\r\n...\r\n5a43b2d1f759        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer10.b9d85f66_sleepypod-mekmp_default_9fa3ef74-1393-11e5-aded-42010af07014_7d082525                                \r\n06ab4cabc565        gcr.io/google_containers/pause:0.8.0                 \"/pause\"               4 hours ago         Up 4 hours                              k8s_POD.e4cc795_sleepypod-112c1_default_9fe56947-1393-11e5-aded-42010af07014_b2310383                                               \r\n4c70725574ed        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer4.63e5f39_sleepypod-eckpv_default_9f9ba8ea-1393-11e5-aded-42010af07014_e0f13706                                  \r\n3e491a34c918        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer3.2a15f38_sleepypod-eckpv_default_9f9ba8ea-1393-11e5-aded-42010af07014_9d181ac0                                  \r\neb9543fc63b9        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer5.9db5f3a_sleepypod-q9dpx_default_9f9f8436-1393-11e5-aded-42010af07014_47c74ea6                                  \r\n532f0ba08f95        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer1.fb585f36_sleepypod-81gxh_default_9fc28c32-1393-11e5-aded-42010af07014_708dac19                                 \r\nd1798807a41f        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer1.fb585f36_sleepypod-zpgxi_default_9fa6ca8a-1393-11e5-aded-42010af07014_4fdaf5f0                                 \r\na35f6cf4e3f5        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer1.fb585f36_sleepypod-i1pw2_default_9fa948c9-1393-11e5-aded-42010af07014_791453ee                                 \r\ne3a2d3b387d3        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer8.14b25f3d_sleepypod-bj214_default_9faf6559-1393-11e5-aded-42010af07014_e97eebb8                                 \r\n81b51e9f56eb        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer6.d785f3b_sleepypod-x67qk_default_9fb8cd56-1393-11e5-aded-42010af07014_c290eba3                                  \r\n564fd2380bc0        gcr.io/google_containers/pause:0.8.0                 \"/pause\"               4 hours ago         Up 4 hours                              k8s_POD.e4cc795_sleepypod-81gxh_default_9fc28c32-1393-11e5-aded-42010af07014_deaf6b10                                               \r\n4db197a952f9        saadali/sleepy:v0.2                                  \"/sleepy\"              4 hours ago         Up 4 hours                              k8s_sleepycontainer1.fb585f36_sleepypod-di4rg_default_9fabd5ae-1393-11e5-aded-42010af07014_df47b02d                                 \r\n...\r\n```\r\n\r\nRestarting kubelet (`service kubelet restart`) or restarting docker causes the running containers to all get killed.\r\n"}}