{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7795","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7795/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7795/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7795/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/7795","id":73429608,"number":7795,"title":"Kubelet crash","user":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0-candidate","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12/labels","id":1086011,"number":12,"title":"v1.0-candidate","description":"These are candidates for inclusion to v1.0 but will be pushed to v1.0-bubble if they're triaged as not meeting the v1.0 bar as a required launch blocker.","creator":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":6,"state":"open","created_at":"2015-04-27T19:03:36Z","updated_at":"2015-05-05T20:56:18Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-05-05T20:56:18Z","updated_at":"2015-05-05T21:35:26Z","closed_at":null,"body":"I think this is the root cause of the problems in #7697.\r\n\r\nMy cluster does not appear to be recovering from this.\r\n\r\n```\r\nI0505 16:02:47.639343   30805 server.go:233] Using root directory: /var/lib/kubelet\r\nI0505 16:02:47.752861   30805 manager.go:106] cAdvisor running in container: \"/\"\r\nI0505 16:02:47.753274   30805 fs.go:87] Filesystem partitions: map[/dev/disk/by-uuid/96b86d97-40a1-4181-82c1-1c5558bac1e6:{mountpoint:/ major:8 minor:1}]\r\nI0505 16:02:47.755417   30805 machine.go:223] Couldn't collect info from any of the files in \"/etc/machine-id,/var/lib/dbus/machine-id\"\r\nI0505 16:02:47.755461   30805 manager.go:127] Machine: {NumCores:1 CpuFrequency:2299958 MemoryCapacity:1787035648 MachineID: SystemUUID:554F4E90-C7FF-B2FC-D9F4-17B0ACABA641 BootID:40079502-aaf2-4b93-b76d-d2d3fc3b215b Filesystems:[{Device:/dev/disk/by-uuid/96b86d97-40a1-4181-82c1-1c5558bac1e6 Capacity:105553100800}] DiskMap:map\r\n[8:0:{Name:sda Major:8 Minor:0 Size:107374182400 Scheduler:cfq}] NetworkDevices:[{Name:cbr0 MacAddress:6e:dd:13:e1:66:20 Speed:0 Mtu:1460} {Name:eth0 MacAddress:42:01:0a:f0:90:de Speed:0 Mtu:1460}] Topology:[{Id:0 Memory:1787035648 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Lev\r\nel:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:47185920 Type:Unified Level:3}]}]}\r\nruntime/cgo: pthread_create failed: Resource temporarily unavailable\r\nSIGABRT: abort\r\nPC=0x7f6bfa192165\r\ngoroutine 0 [idle]:\r\ngoroutine 6 [syscall]:\r\nruntime.notetsleepg(0x18a1b38, 0x6fbb38fd9, 0x0)\r\n        /usr/src/go/src/runtime/lock_futex.go:201 +0x52 fp=0xc208017f68 sp=0xc208017f40\r\nruntime.timerproc()\r\n        /usr/src/go/src/runtime/time.go:207 +0xfa fp=0xc208017fe0 sp=0xc208017f68\r\nruntime.goexit()\r\n        /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc208017fe8 sp=0xc208017fe0\r\ncreated by runtime.addtimerLocked\r\n        /usr/src/go/src/runtime/time.go:113 +0x1ba\r\ngoroutine 1 [runnable, locked to thread]:\r\nos.Open(0x108a110, 0x19, 0x2, 0x2, 0xc208030780)\r\n        /usr/src/go/src/os/file.go:238\r\nos.hostname(0x0, 0x0, 0x0, 0x0)\r\n        /usr/src/go/src/os/sys_linux.go:10 +0x9f\r\nos.Hostname(0x0, 0x0, 0x0, 0x0)\r\n        /usr/src/go/src/os/doc.go:93 +0x43\r\ngithub.com/golang/glog.init·2()\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog_file.go:58 +0x1f\r\ngithub.com/golang/glog.init()\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog_file.go:124 +0x23b\r\ngithub.com/GoogleCloudPlatform/kubernetes/pkg/util.init()\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/validation.go:97 +0x86\r\nmain.init()\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/cmd/kubelet/kubelet.go:50 +0x4c\r\ngoroutine 17 [syscall, locked to thread]:\r\nruntime.goexit()\r\n        /usr/src/go/src/runtime/asm_amd64.s:2232 +0x1\r\ngoroutine 5 [chan receive]:\r\ngithub.com/golang/glog.(*loggingT).flushDaemon(0x18a2060)\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78\r\ncreated by github.com/golang/glog.init·1\r\n        /go/src/github.com/GoogleCloudPlatform/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7\r\nrax     0x0\r\nrbx     0x7f6bfa4e5d98\r\nrcx     0xffffffffffffffff\r\nrdx     0x6\r\nrdi     0x792b\r\nrsi     0x792f\r\nrbp     0xbcfb12\r\nrsp     0x7f6bf995d9d8\r\nr8      0xa\r\nr9      0x7f6bf995e700\r\nr10     0x8\r\nr11     0x206\r\nr12     0x1a80010\r\nr13     0x3\r\nr14     0x7f6bfa927040\r\nr15     0x3\r\nrip     0x7f6bfa192165\r\nrflags  0x206\r\ncs      0x33\r\nfs      0x0\r\ngs      0x0\r\n```\r\n\r\nTo dawn to find an owner."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/99229807","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/7795#issuecomment-99229807","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7795","id":99229807,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-05-05T21:35:26Z","updated_at":"2015-05-05T21:35:26Z","body":"Can I access the node in bad state? Looks like kubelet was crashed at init time. Monit should restart kubelet every 2 mins. Is this a crashloop? When you said this is the root cause of #7697 because this happens to a lot of nodes in a cluster? so that rc or node controller ping-pong the workloads? "}}