{"action":"closed","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5371","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5371/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5371/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5371/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/5371","id":60835998,"number":5371,"title":"Kubelet hangs after instance creation","user":{"login":"sevki","id":429977,"avatar_url":"https://avatars.githubusercontent.com/u/429977?v=3","gravatar_id":"","url":"https://api.github.com/users/sevki","html_url":"https://github.com/sevki","followers_url":"https://api.github.com/users/sevki/followers","following_url":"https://api.github.com/users/sevki/following{/other_user}","gists_url":"https://api.github.com/users/sevki/gists{/gist_id}","starred_url":"https://api.github.com/users/sevki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sevki/subscriptions","organizations_url":"https://api.github.com/users/sevki/orgs","repos_url":"https://api.github.com/users/sevki/repos","events_url":"https://api.github.com/users/sevki/events{/privacy}","received_events_url":"https://api.github.com/users/sevki/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/support","name":"priority/support","color":"c7def8"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":{"login":"erictune","id":8105590,"avatar_url":"https://avatars.githubusercontent.com/u/8105590?v=3","gravatar_id":"","url":"https://api.github.com/users/erictune","html_url":"https://github.com/erictune","followers_url":"https://api.github.com/users/erictune/followers","following_url":"https://api.github.com/users/erictune/following{/other_user}","gists_url":"https://api.github.com/users/erictune/gists{/gist_id}","starred_url":"https://api.github.com/users/erictune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erictune/subscriptions","organizations_url":"https://api.github.com/users/erictune/orgs","repos_url":"https://api.github.com/users/erictune/repos","events_url":"https://api.github.com/users/erictune/events{/privacy}","received_events_url":"https://api.github.com/users/erictune/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2015-03-12T15:22:19Z","updated_at":"2015-03-18T15:11:14Z","closed_at":"2015-03-18T15:11:14Z","body":"It seems that the kubelet becomes unresponsive after a docker instance is created\r\n\r\n```\r\nMar 12 12:18:00 server1.c.something-live.internal kubelet[715]: I0312 12:18:00.241113     715 etcd.go:82] Watch closed (&api.Status{TypeMeta:api.TypeMeta{Kind:\"\", APIVersion:\"\"}, ListMeta:api.ListMeta{SelfLink:\"\", ResourceVersion:\"\"}, Status:\"Failure\", Message:\"401: The event in requested index is outdated and cleared (the requested history has been cleared [16202858/15852374]) [16203857]\", Reason:\"\", Details:(*api.StatusDetails)(nil), Code:0}). Reopening.\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.241309     715 logs.go:40] etcd DEBUG: get /registry/nodes/10.240.116.118/boundpods [http://10.240.186.130:4001]\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.241369     715 logs.go:40] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/10.240.116.118/boundpods?consistent=true&recursive=false&sorted=false\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.241381     715 logs.go:40] etcd DEBUG: send.request.to http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&recursive=false&sorted=false | method GET\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.242991     715 logs.go:40] etcd DEBUG: recv.response.fromhttp://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&recursive=false&sorted=false\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.243533     715 logs.go:40] etcd DEBUG: recv.success.http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&recursive=false&sorted=false\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.243996     715 logs.go:40] etcd DEBUG: watch /registry/nodes/10.240.116.118/boundpods [http://10.240.186.130:4001]\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.244014     715 logs.go:40] etcd DEBUG: get /registry/nodes/10.240.116.118/boundpods [http://10.240.186.130:4001]\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.244040     715 logs.go:40] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203860\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.244086     715 logs.go:40] etcd DEBUG: send.request.to http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203860 | method GET\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.251769     715 logs.go:40] etcd DEBUG: recv.response.fromhttp://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203860\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.253318     715 logs.go:40] etcd DEBUG: recv.success.http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203860\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.253410     715 logs.go:40] etcd DEBUG: get /registry/nodes/10.240.116.118/boundpods [http://10.240.186.130:4001]\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.253440     715 logs.go:40] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203861\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.253471     715 logs.go:40] etcd DEBUG: send.request.to http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203861 | method GET\r\nMar 12 12:18:01 server1.c.something-live.internal kubelet[715]: I0312 12:18:01.258111     715 logs.go:40] etcd DEBUG: recv.response.fromhttp://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203861\r\nMar 12 12:18:04 server1.c.something-live.internal kubelet[715]: I0312 12:18:04.918977     715 logs.go:40] etcd DEBUG: recv.success.http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203861\r\nMar 12 12:18:04 server1.c.something-live.internal kubelet[715]: I0312 12:18:04.919140     715 logs.go:40] etcd DEBUG: get /registry/nodes/10.240.116.118/boundpods [http://10.240.186.130:4001]\r\nMar 12 12:18:04 server1.c.something-live.internal kubelet[715]: I0312 12:18:04.919179     715 logs.go:40] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203866\r\nMar 12 12:18:04 server1.c.something-live.internal kubelet[715]: I0312 12:18:04.919188     715 logs.go:40] etcd DEBUG: send.request.to http://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203866 | method GET\r\nMar 12 12:18:04 server1.c.something-live.internal kubelet[715]: I0312 12:18:04.925299     715 logs.go:40] etcd DEBUG: recv.response.fromhttp://10.240.186.130:4001/v2/keys/registry/nodes/10.240.116.118/boundpods?consistent=true&wait=true&waitIndex=16203866\r\nMar 12 12:18:11 server1.c.something-live.internal kubelet[715]: I0312 12:18:11.277375     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-2nrtl\", UID:\"2c7eba9f-c7de-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"spec.containers{something}\"}): reason: 'killing' Killing f1f345f41e8ceecab864b145ec5ea7c49d82ae3d151f8fdc8c6076aa6f017fec - /k8s_something.39b9c423_something-controller-2nrtl.default.etcd_2c7eba9f-c7de-11e4-803e-42010af0ba82_8c57099c\r\nMar 12 12:18:11 server1.c.something-live.internal kubelet[715]: I0312 12:18:11.289000     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-2nrtl\", UID:\"2c7eba9f-c7de-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"implicitly required container POD\"}): reason: 'killing' Killing ca7408c5b0d1b1b9189ccaa6e953db5183fa7b4a990e4f8d970a64c39d897f45 - /k8s_POD.2f6eec6a_something-controller-2nrtl.default.etcd_2c7eba9f-c7de-11e4-803e-42010af0ba82_9f54f968\r\nMar 12 12:18:11 server1.c.something-live.internal kubelet[715]: I0312 12:18:11.303648     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"implicitly required container POD\"}): reason: 'pulled' Successfully pulled image \"kubernetes/pause:latest\"\r\nMar 12 12:18:12 server1.c.something-live.internal kubelet[715]: I0312 12:18:12.713767     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"implicitly required container POD\"}): reason: 'created' Created with docker id 8cef949b9498dc88a8c5204476750c21706c4c51377ac26add43a7704255d273\r\nMar 12 12:18:13 server1.c.something-live.internal kubelet[715]: I0312 12:18:13.861159     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"implicitly required container POD\"}): reason: 'started' Started with docker id 8cef949b9498dc88a8c5204476750c21706c4c51377ac26add43a7704255d273\r\nMar 12 12:18:14 server1.c.something-live.internal kubelet[715]: I0312 12:18:14.054215     715 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider\r\nMar 12 12:18:14 server1.c.something-live.internal kubelet[715]: I0312 12:18:14.054461     715 provider.go:91] Refreshing cache for provider: *gcp_credentials.dockerConfigKeyProvider\r\nMar 12 12:18:14 server1.c.something-live.internal kubelet[715]: E0312 12:18:14.465805     715 metadata.go:109] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg\r\nMar 12 12:18:14 server1.c.something-live.internal kubelet[715]: I0312 12:18:14.465847     715 provider.go:91] Refreshing cache for provider: *gcp_credentials.dockerConfigUrlKeyProvider\r\nMar 12 12:18:14 server1.c.something-live.internal kubelet[715]: E0312 12:18:14.468857     715 metadata.go:121] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url\r\nMar 12 12:19:13 server1.c.something-live.internal kubelet[715]: I0312 12:19:13.275855     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"spec.containers{something}\"}): reason: 'pulled' Successfully pulled image “image/somesort”\r\nMar 12 12:19:13 server1.c.something-live.internal kubelet[715]: I0312 12:19:13.788724     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:\"something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"spec.containers{something}\"}): reason: 'created' Created with docker id 28e718a309dd820ac2b3e97c72eb94c0bcdd6c4a9375b82f742cf9f0e0175ec8\r\nMar 12 12:19:13 server1.c.something-live.internal kubelet[715]: I0312 12:19:13.883755     715 event.go:137] Event(api.ObjectReference{Kind:\"BoundPod\", Namespace:\"default\", Name:”something-controller-40noe\", UID:\"d5be75bf-c8b1-11e4-803e-42010af0ba82\", APIVersion:\"v1beta1\", ResourceVersion:\"\", FieldPath:\"spec.containers{something}\"}): reason: 'started' Started with docker id 28e718a309dd820ac2b3e97c72eb94c0bcdd6c4a9375b82f742cf9f0e0175ec8\r\n```"}}