{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8068","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8068/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8068/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8068/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/8068","id":75304082,"number":8068,"title":"kubelet can't \"get\" node from apiserver for up to ~2 minutes","user":{"login":"mbforbes","id":1170062,"avatar_url":"https://avatars.githubusercontent.com/u/1170062?v=3","gravatar_id":"","url":"https://api.github.com/users/mbforbes","html_url":"https://github.com/mbforbes","followers_url":"https://api.github.com/users/mbforbes/followers","following_url":"https://api.github.com/users/mbforbes/following{/other_user}","gists_url":"https://api.github.com/users/mbforbes/gists{/gist_id}","starred_url":"https://api.github.com/users/mbforbes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbforbes/subscriptions","organizations_url":"https://api.github.com/users/mbforbes/orgs","repos_url":"https://api.github.com/users/mbforbes/repos","events_url":"https://api.github.com/users/mbforbes/events{/privacy}","received_events_url":"https://api.github.com/users/mbforbes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster-lifecycle","name":"team/cluster-lifecycle","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-11T17:50:13Z","updated_at":"2015-05-11T17:53:23Z","closed_at":null,"body":"This makes cluster startup pretty painful and have wide variance, causing problems for monitoring and testing cluster turnup. After the kubelet can first talk to the apiserver, in can be from ~0 seconds to ~2 minutes until the kubelet can \"get\" its own node. Only then can it start pulling images to get pods running.\r\n\r\nWe see repeating errors for up to 2 minutes that look like:\r\n\r\n```\r\nE0505 02:03:19.746214    4659 kubelet.go:1534] error getting node: node <node name> not found\r\nE0505 02:03:26.003989    4659 kubelet.go:1710] error updating node status, will retry: error getting node \"<node name>\": minion \"<node name>\" not found\r\n```\r\n\r\nthis has, I believe, been plaguing us for quite a while. It would be really great if we could figure out why. The master logs, if I recall correctly, don't have anything interesting in them (similar errors). (The most recent cluster I had that had these problems is gone, but I'll check further in the next.)\r\n\r\n+cc @roberthbailey (hoping that \"kubelet registering with master\" change will fix this)\r\n+cc @zmerlynn (has looked into this too)\r\n+cc @davidopp (master)\r\n+cc @dchen1107 (node)"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/100997675","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/8068#issuecomment-100997675","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/8068","id":100997675,"user":{"login":"zmerlynn","id":4942464,"avatar_url":"https://avatars.githubusercontent.com/u/4942464?v=3","gravatar_id":"","url":"https://api.github.com/users/zmerlynn","html_url":"https://github.com/zmerlynn","followers_url":"https://api.github.com/users/zmerlynn/followers","following_url":"https://api.github.com/users/zmerlynn/following{/other_user}","gists_url":"https://api.github.com/users/zmerlynn/gists{/gist_id}","starred_url":"https://api.github.com/users/zmerlynn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zmerlynn/subscriptions","organizations_url":"https://api.github.com/users/zmerlynn/orgs","repos_url":"https://api.github.com/users/zmerlynn/repos","events_url":"https://api.github.com/users/zmerlynn/events{/privacy}","received_events_url":"https://api.github.com/users/zmerlynn/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T17:53:23Z","updated_at":"2015-05-11T17:53:23Z","body":"When I've seen it, it's actually been fairly bimodal. Either the kubelet gets stuck for 2m doing this, or it doesn't. I actually have a dataset of cluster-up timings on GKE that show a weird pile of \"fast\" up timing (1:30-ish) and \"slow\" up timing (3:30-ish), and when I stopped and debugged one of the slow ones, it was this bug."}}