{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6115","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6115/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6115/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6115/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6115","id":64868938,"number":6115,"title":"kubelet /stats reports 500 when a container is not found","user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P3","name":"priority/P3","color":"fef2c0"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-27T23:14:49Z","updated_at":"2015-03-27T23:20:16Z","closed_at":null,"body":"There seems to be a delay between the container creation and the cadvisor being notified of such a container change (through inotify) and registers the container. This leads to a case where cadvisor might report the container not found while it actually exists. Because [docker is able to find the container](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/kubelet.go#L2029), kubelet returns the raw error (instead of `ErrContainerNotFound`).\r\n\r\n```\r\nI0327 22:13:10.418373   30839 event.go:138] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"bar-controller-wbo6m\", UID:\"63459b0e-d4ce-11e4-98a4-42010af04d55\", APIVersion:\"v1beta1\", ResourceVersion:\"110606\", FieldPath:\"spec.containers{bar}\"}): reason: 'created' Created with docker id d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\r\nI0327 22:13:12.153278   30839 event.go:138] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"default\", Name:\"bar-controller-wbo6m\", UID:\"63459b0e-d4ce-11e4-98a4-42010af04d55\", APIVersion:\"v1beta1\", ResourceVersion:\"110606\", FieldPath:\"spec.containers{bar}\"}): reason: 'started' Started with docker id d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\r\nI0327 22:13:20.911539   30839 server.go:140] HTTP InternalServerError: Internal Error: unable to find Docker container \"d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\"\r\n(SKIPPED THE BACKTRACE)\r\nI0327 22:13:28.415407   30839 factory.go:71] Using factory \"docker\" for container \"/docker/d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\"\r\nI0327 22:13:28.426950   30839 manager.go:659] Added container: \"/docker/d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2\" (aliases: [k8s_bar.b80fe20f_bar-controller-wbo6m_default_63459b0e-d4ce-11e4-98a4-42010af04d55_0844342d d97df25e70f82640924c01e9a60ed5b18b26e154660321ecd9d4bbb2808c0ee2], namespace: \"docker\")\r\n```\r\n\r\nThis happened when kubelet was trying to start 10~20 containers in a short period of time on a regular node created by e2e. Since it was a relative high load, this scenario should rarely happen. That said, it'd still be nice if Kubelet can report ContainerNotFound instead of exposing the 500 errors."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/87118564","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6115#issuecomment-87118564","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6115","id":87118564,"user":{"login":"vmarmol","id":985208,"avatar_url":"https://avatars.githubusercontent.com/u/985208?v=3","gravatar_id":"","url":"https://api.github.com/users/vmarmol","html_url":"https://github.com/vmarmol","followers_url":"https://api.github.com/users/vmarmol/followers","following_url":"https://api.github.com/users/vmarmol/following{/other_user}","gists_url":"https://api.github.com/users/vmarmol/gists{/gist_id}","starred_url":"https://api.github.com/users/vmarmol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vmarmol/subscriptions","organizations_url":"https://api.github.com/users/vmarmol/orgs","repos_url":"https://api.github.com/users/vmarmol/repos","events_url":"https://api.github.com/users/vmarmol/events{/privacy}","received_events_url":"https://api.github.com/users/vmarmol/received_events","type":"User","site_admin":false},"created_at":"2015-03-27T23:20:16Z","updated_at":"2015-03-27T23:20:16Z","body":"That's certainly not a race we expect to lose, but the behavior you mention should be what we do."}}