{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16601","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16601/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16601/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16601/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16601","id":114323676,"number":16601,"title":"Docker stuck in a bad state on of our soak test cluster's nodes","user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars.githubusercontent.com/u/7085343?v=3","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-30T18:02:34Z","updated_at":"2015-10-30T18:31:32Z","closed_at":null,"body":"This has been causing all the soak test runs on the cluster to fail overnight.\r\n\r\nGeneral summary of what I see on the node:\r\n* Docker is busted on one of the nodes.\r\n* Trying to talk to it gets the error: dial unix /var/run/docker.sock: resource temporarily unavailable\r\n* Supervisor's logs indicate that its checks of docker are failing, but it fails to restart docker:\r\nDocker failed!\r\nStopping Docker: docker.\r\nStarting Docker: docker failed!\r\nwaiting a minute for startup\r\n* docker.log was spammed with the same message repeating over and over: /usr/bin/docker already running.\r\n* Supervisor's logs indicate that its checks of kubelet are failing, so it's restarting it every minute\r\n* kubelet.log indicates that it can't hit docker either: Failed to execute Info() call to the Docker client: Get http://unix.sock/info: dial unix /var/run/docker.sock: resource temporarily unavailable\r\n* kubelet.log also has (potentially benign) errors around not being able to configure cbr0: ConfigureCBR0 requested, but PodCIDR not set. Will not configure CBR0 right now\r\n* The kubelet-checker.sh script isn't running on this minion, despite being running on the other nodes in the cluster\r\n\r\n@kubernetes/goog-node - anyone available to check into this a little more thoroughly? It'd be nice to figure out why the node ended up in this state. You can SSH to the node using\r\n```\r\ngcloud compute ssh --project=kubernetes-jenkins --zone=us-central1-f gce-soak-weekly-minion-1fdd\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/152610823","html_url":"https://github.com/kubernetes/kubernetes/issues/16601#issuecomment-152610823","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16601","id":152610823,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-10-30T18:31:32Z","updated_at":"2015-10-30T18:31:32Z","body":"which version of kubernetes? "}}