{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10368","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10368/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10368/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10368/events","html_url":"https://github.com/kubernetes/kubernetes/issues/10368","id":91079132,"number":10368,"title":"kubelet does not recover from machine reboot","user":{"login":"timstclair","id":3272040,"avatar_url":"https://avatars.githubusercontent.com/u/3272040?v=3","gravatar_id":"","url":"https://api.github.com/users/timstclair","html_url":"https://github.com/timstclair","followers_url":"https://api.github.com/users/timstclair/followers","following_url":"https://api.github.com/users/timstclair/following{/other_user}","gists_url":"https://api.github.com/users/timstclair/gists{/gist_id}","starred_url":"https://api.github.com/users/timstclair/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timstclair/subscriptions","organizations_url":"https://api.github.com/users/timstclair/orgs","repos_url":"https://api.github.com/users/timstclair/repos","events_url":"https://api.github.com/users/timstclair/events{/privacy}","received_events_url":"https://api.github.com/users/timstclair/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/cluster-lifecycle","name":"area/cluster-lifecycle","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-06-25T21:22:26Z","updated_at":"2015-09-22T12:45:40Z","closed_at":"2015-07-02T17:39:17Z","body":"I started a cluster running on GCE last week, using `go run hack/e2e.go -build -up`. Today I noticed today that both of my minion nodes were stuck in the NotReady state:\r\n```\r\n$ kubectl get nodes\r\nNAME                           LABELS                                                STATUS\r\ne2e-test-stclair-minion-cu9a   kubernetes.io/hostname=e2e-test-stclair-minion-cu9a   NotReady\r\ne2e-test-stclair-minion-i5b9   kubernetes.io/hostname=e2e-test-stclair-minion-i5b9   NotReady\r\n```\r\nInspecting the logs yields the same error message over and over again:\r\n```\r\nstclair@e2e-test-stclair-minion-cu9a:~$ tail /var/log/kubelet.log      \r\nW0625 21:18:55.371135    2928 request.go:292] field selector: v1 - pods - spec.host - e2e-test-stclair-minion-cu9a: need to check if this is versioned correctly.\r\nE0625 21:18:55.371813    2928 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)\r\nE0625 21:18:55.371940    2928 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)\r\nE0625 21:18:55.737900    2928 kubelet.go:1591] error getting node: node e2e-test-stclair-minion-cu9a not found\r\nE0625 21:18:55.748663    2928 kubelet.go:1212] Failed creating a mirror pod \"fluentd-elasticsearch-e2e-test-stclair-minion-cu9a_default\": the server has asked for the client to provide credentials (post pods)\r\nE0625 21:18:56.202222    2928 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)\r\nW0625 21:18:56.371997    2928 request.go:292] field selector: v1 - nodes - metadata.name - e2e-test-stclair-minion-cu9a: need to check if this is versioned correctly.\r\nW0625 21:18:56.372199    2928 request.go:292] field selector: v1 - pods - spec.host - e2e-test-stclair-minion-cu9a: need to check if this is versioned correctly.\r\nE0625 21:18:56.372828    2928 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)\r\nE0625 21:18:56.373378    2928 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)\r\n```\r\nSame with the proxy:\r\n```\r\nstclair@e2e-test-stclair-minion-cu9a:~$ tail -n 5 /var/log/kube-proxy.log \r\nE0625 21:18:44.957584    2968 api.go:180] Unable to load endpoints: the server has asked for the client to provide credentials (get endpoints)\r\nE0625 21:19:31.023914    2968 api.go:108] Unable to load services: the server has asked for the client to provide credentials (get services)\r\nE0625 21:20:08.662204    2968 api.go:180] Unable to load endpoints: the server has asked for the client to provide credentials (get endpoints)\r\nE0625 21:20:43.884667    2968 api.go:108] Unable to load services: the server has asked for the client to provide credentials (get services)\r\nE0625 21:21:15.108205    2968 api.go:180] Unable to load endpoints: the server has asked for the client to provide credentials (get endpoints)\r\n```\r\n\r\nVersion:\r\n```\r\n$ kubectl describe nodes e2e-test-stclair-minion-cu9a\r\nW0625 14:05:10.463312   16838 request.go:291] field selector: v1beta3 - events - involvedObject.uid - e2e-test-stclair-minion-cu9a: need to check if this is versioned correctly.\r\nW0625 14:05:10.463428   16838 request.go:291] field selector: v1beta3 - events - involvedObject.name - e2e-test-stclair-minion-cu9a: need to check if this is versioned correctly.\r\nW0625 14:05:10.463441   16838 request.go:291] field selector: v1beta3 - events - involvedObject.namespace - : need to check if this is versioned correctly.\r\nW0625 14:05:10.463451   16838 request.go:291] field selector: v1beta3 - events - involvedObject.kind - Node: need to check if this is versioned correctly.\r\nName:\t\t\te2e-test-stclair-minion-cu9a\r\nLabels:\t\t\tkubernetes.io/hostname=e2e-test-stclair-minion-cu9a\r\nCreationTimestamp:\tThu, 18 Jun 2015 18:10:36 -0700\r\nConditions:\r\n  Type\t\tStatus\t\tLastHeartbeatTime\t\t\tLastTransitionTime\t\t\tReason\t\t\t\t\tMessage\r\n  Ready \tUnknown \tFri, 19 Jun 2015 13:52:08 -0700 \tFri, 19 Jun 2015 13:52:53 -0700 \tKubelet stopped posting node status. \t\r\nAddresses:\t10.240.179.67,146.148.78.88\r\nCapacity:\r\n cpu:\t\t1\r\n memory:\t3800808Ki\r\n pods:\t\t100\r\nVersion:\r\n Kernel Version:\t\t3.16.0-0.bpo.4-amd64\r\n OS Image:\t\t\tDebian GNU/Linux 7 (wheezy)\r\n Container Runtime Version:\tdocker://1.6.2\r\n Kubelet Version:\t\tv0.19.0-634-g2fab19a748fabb-dirty\r\n Kube-Proxy Version:\t\tv0.19.0-634-g2fab19a748fabb-dirty\r\nPodCIDR:\t\t\t10.245.1.0/24\r\nExternalID:\t\t\t1522119118968509696\r\nPods:\t\t\t\t(0 in total)\r\nNo events.\r\n```"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/142277467","html_url":"https://github.com/kubernetes/kubernetes/issues/10368#issuecomment-142277467","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/10368","id":142277467,"user":{"login":"bliss","id":73997,"avatar_url":"https://avatars.githubusercontent.com/u/73997?v=3","gravatar_id":"","url":"https://api.github.com/users/bliss","html_url":"https://github.com/bliss","followers_url":"https://api.github.com/users/bliss/followers","following_url":"https://api.github.com/users/bliss/following{/other_user}","gists_url":"https://api.github.com/users/bliss/gists{/gist_id}","starred_url":"https://api.github.com/users/bliss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bliss/subscriptions","organizations_url":"https://api.github.com/users/bliss/orgs","repos_url":"https://api.github.com/users/bliss/repos","events_url":"https://api.github.com/users/bliss/events{/privacy}","received_events_url":"https://api.github.com/users/bliss/received_events","type":"User","site_admin":false},"created_at":"2015-09-22T12:45:40Z","updated_at":"2015-09-22T12:45:40Z","body":"Good day!\r\nIn spite of the problem has been marked as fixed on kubernetes at least 1.0.3 it's still persist. Just simple KVM setup, master and two nodes. After a node reboot it's never gets back and has NotReady status. Node kubelet status is OK. To bring a node back to cluster I have to restart node kubelet."}}