{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3286","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3286/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3286/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3286/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3286","id":53687620,"number":3286,"title":"Mysterious pod deploy error and cannot successfully debug","user":{"login":"jasonkuhrt","id":284476,"avatar_url":"https://avatars.githubusercontent.com/u/284476?v=3","gravatar_id":"","url":"https://api.github.com/users/jasonkuhrt","html_url":"https://github.com/jasonkuhrt","followers_url":"https://api.github.com/users/jasonkuhrt/followers","following_url":"https://api.github.com/users/jasonkuhrt/following{/other_user}","gists_url":"https://api.github.com/users/jasonkuhrt/gists{/gist_id}","starred_url":"https://api.github.com/users/jasonkuhrt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasonkuhrt/subscriptions","organizations_url":"https://api.github.com/users/jasonkuhrt/orgs","repos_url":"https://api.github.com/users/jasonkuhrt/repos","events_url":"https://api.github.com/users/jasonkuhrt/events{/privacy}","received_events_url":"https://api.github.com/users/jasonkuhrt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-01-07T21:58:34Z","updated_at":"2015-01-07T22:18:30Z","closed_at":null,"body":"There is a 99.9999% chance I've made a mistake in trying to deploy a pod to my kubernetes cluster but I cannot figure out the problem. It is currently very hard to debug issues like this in Kubernetes.\r\n\r\nI am using GKE which I believe is using Kubernetes 0.6.x?\r\n\r\nOk so my issue is that I've deployed a pod that looks like:\r\n\r\n\r\n```yaml\r\n# pod.yaml\r\nkind: Pod\r\napiVersion: v1beta1\r\nid: cloud-api-stream\r\nlabels:\r\n  name: cloud-api-stream\r\ndesiredState:\r\n  manifest:\r\n    version: v1beta1\r\n    id: cloud-api-stream\r\n    containers:\r\n      - name: cloud-api-stream\r\n        image: quay.io/littlebits/cloud-server-api-stream\r\n        ports:\r\n          - name: cloud-api-stream\r\n            containerPort: 7200\r\n```\r\n\r\nBut once deployed it has this description:\r\n\r\n```\r\n$ kubectl --insecure-skip-tls-verify=true --server=https://x.x.x.x describe pod cloud-api-stream\r\n\r\nName:\t\t\t\tcloud-api-stream\r\nImage(s):\t\t\tquay.io/littlebits/cloud-server-api-stream\r\nHost:\t\t\t\tk8s-cluster-test1-node-2.c.littlebits-electronics.internal/130.211.184.163\r\nLabels:\t\t\t\tname=cloud-api-stream\r\nStatus:\t\t\t\tFailed\r\nReplication Controllers:\t<none>\r\nNo events.\r\n```\r\n\r\nHere are the last 100 lines of `kubelet` logs from node-2\r\n\r\n```\r\njasonkuhrt@k8s-cluster-test1-node-2:~$ tail -n 100 /var/log/kubelet.log\r\nE0107 21:13:57.231172 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:13:58.231454 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:13:58.231531 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:13:58.231545 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:13:58.233462 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:13:58.233494 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:13:58.233551 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:13:58.233569 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:13:58.233606 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:13:58.233620 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433 | method GET\r\nI0107 21:13:58.234455 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:18:58.234079 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nE0107 21:18:58.234183 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:18:59.234458 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:18:59.234535 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:18:59.234547 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:18:59.236467 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:18:59.236500 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:18:59.236542 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:18:59.236553 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:18:59.236579 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:18:59.236588 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433 | method GET\r\nI0107 21:18:59.237440 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:23:59.237098 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nE0107 21:23:59.237241 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:24:00.237495 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:24:00.237572 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:24:00.237584 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:24:00.239348 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:24:00.239383 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:24:00.239426 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:24:00.239436 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:24:00.239453 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:24:00.239468 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433 | method GET\r\nI0107 21:24:00.240291 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:24:16.661031 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=433\r\nI0107 21:24:16.661256 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:24:16.661312 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:24:16.661326 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:24:16.673163 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:29:16.660706 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nE0107 21:29:16.660809 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:29:17.661045 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:29:17.661125 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:29:17.661155 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:29:17.662857 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:29:17.662884 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:29:17.662981 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:29:17.662995 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:29:17.663012 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:29:17.663021 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:29:17.665035 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:34:17.667082 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nE0107 21:34:17.667183 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:34:18.667468 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:34:18.667541 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:34:18.667552 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:34:18.681068 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:34:18.681127 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:34:18.681237 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:34:18.681249 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:34:18.681271 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:34:18.681281 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:34:18.690927 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:39:18.676113 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nE0107 21:39:18.676222 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:39:19.676498 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:39:19.676573 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:39:19.676584 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:39:19.678339 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:39:19.678535 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:39:19.678645 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:39:19.678657 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:39:19.678675 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:39:19.678690 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:39:19.681412 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:44:19.678962 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nE0107 21:44:19.679091 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:44:20.679314 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:44:20.679388 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:44:20.679403 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:44:20.681242 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:44:20.681270 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:44:20.681359 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:44:20.681396 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:44:20.681414 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:44:20.681424 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:44:20.683358 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:49:20.681707 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nE0107 21:49:20.681817 13487 etcd.go:71] Watch error: &{{  0001-01-01 00:00:00 +0000 UTC      map[]} Failure unexpected end of JSON input  <nil> 0}\r\nI0107 21:49:21.682038 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:49:21.682120 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:49:21.682132 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false | method GET\r\nI0107 21:49:21.702822 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:49:21.702870 13487 logs.go:39] etcd DEBUG: recv.success.http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&recursive=false&sorted=false\r\nI0107 21:49:21.702980 13487 logs.go:39] etcd DEBUG: watch /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:49:21.702993 13487 logs.go:39] etcd DEBUG: get /registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet [http://10.240.65.178:4001]\r\nI0107 21:49:21.703014 13487 logs.go:39] etcd DEBUG: Connecting to etcd: attempt1forkeys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\nI0107 21:49:21.703024 13487 logs.go:39] etcd DEBUG: send.request.to http://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438 | method GET\r\nI0107 21:49:21.712337 13487 logs.go:39] etcd DEBUG: recv.response.fromhttp://10.240.65.178:4001/v2/keys/registry/hosts/k8s-cluster-test1-node-2.c.littlebits-electronics.internal/kubelet?consistent=true&wait=true&waitIndex=438\r\n```\r\n\r\nThere is one other wrinkle. I realized earlier today that my error was that I did not provide a `.dockercfg` for `kubetet` for my image on `quay.io` that requires authentication. I knew this because I saw 403 errors in `kubelet` logs! I tried to fix this by placing a correct `.dockercfg` in each node's root `/` directory. Things still did not work. Then I realized that I need to (really?) restart the `kubelet` process via `sudo service kubelet restart`. After doing this however things have *still* not worked. All I see is that the `pod` `status` is \"failed\" and a whole bunch of not useful logs from `kubelet` (compared to before where I was at least about to figure out that it was failing on `quay.io` with `403`).\r\n\r\nThanks for your consideration!"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/69102791","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3286#issuecomment-69102791","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3286","id":69102791,"user":{"login":"jasonkuhrt","id":284476,"avatar_url":"https://avatars.githubusercontent.com/u/284476?v=3","gravatar_id":"","url":"https://api.github.com/users/jasonkuhrt","html_url":"https://github.com/jasonkuhrt","followers_url":"https://api.github.com/users/jasonkuhrt/followers","following_url":"https://api.github.com/users/jasonkuhrt/following{/other_user}","gists_url":"https://api.github.com/users/jasonkuhrt/gists{/gist_id}","starred_url":"https://api.github.com/users/jasonkuhrt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasonkuhrt/subscriptions","organizations_url":"https://api.github.com/users/jasonkuhrt/orgs","repos_url":"https://api.github.com/users/jasonkuhrt/repos","events_url":"https://api.github.com/users/jasonkuhrt/events{/privacy}","received_events_url":"https://api.github.com/users/jasonkuhrt/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T22:18:30Z","updated_at":"2015-01-07T22:18:30Z","body":"Wow... tired I guess. Here's the actually cmd result:\r\n\r\n```\r\n$ sudo docker logs 92c364736b0f\r\n\r\njasonkuhrt@k8s-cluster-test1-node-2:~$ sudo docker logs 408e7d45d52e\r\n\r\n> cloud-server-api-stream@0.2.11 start /usr/src/app\r\n> node lib/index\r\n\r\n{\"name\":\"app\",\"hostname\":\"cloud-api-stream\",\"pid\":15,\"component\":\"auth\",\"level\":30,\"msg\":\"loaded\",\"time\":\"2015-01-07T22:17:22.460Z\",\"v\":0}\r\n{\"name\":\"app\",\"hostname\":\"cloud-api-stream\",\"pid\":15,\"component\":\"wsserver\",\"level\":30,\"config\":{\"oauth\":{\"auth_token_uri\":\"http://localhost:3000/access_tokens\",\"auth_uri\":\"http://localhost:3000/oauth/authorize\"},\"port\":7200,\"log_level\":\"debug\",\"cloudsetup_uri\":\"http://device-setup-local.littlebitscloud.cc:7000\",\"api_http_uri\":\"http://localhost:8000\",\"redis\":{\"database_number\":1}},\"msg\":\"Launching\",\"time\":\"2015-01-07T22:17:22.812Z\",\"v\":0}\r\n\r\nevents.js:72\r\n        throw er; // Unhandled 'error' event\r\n              ^\r\nError: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED\r\n    at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:142:13)\r\n    at RedisClient.on_error (/usr/src/app/node_modules/redis/index.js:180:10)\r\n    at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:95:14)\r\n    at Socket.emit (events.js:95:17)\r\n    at net.js:441:14\r\n    at process._tickCallback (node.js:442:13)\r\n\r\nnpm ERR! Linux 3.16-0.bpo.2-amd64\r\nnpm ERR! argv \"node\" \"/usr/local/bin/npm\" \"start\"\r\nnpm ERR! node v0.10.35\r\nnpm ERR! npm  v2.1.16\r\nnpm ERR! code ELIFECYCLE\r\nnpm ERR! cloud-server-api-stream@0.2.11 start: `node lib/index`\r\nnpm ERR! Exit status 8\r\nnpm ERR!\r\nnpm ERR! Failed at the cloud-server-api-stream@0.2.11 start script 'node lib/index'.\r\nnpm ERR! This is most likely a problem with the cloud-server-api-stream package,\r\nnpm ERR! not with npm itself.\r\nnpm ERR! Tell the author that this fails on your system:\r\nnpm ERR!     node lib/index\r\nnpm ERR! You can get their info via:\r\nnpm ERR!     npm owner ls cloud-server-api-stream\r\nnpm ERR! There is likely additional logging output above.\r\n\r\nnpm ERR! Please include the following file with any support request:\r\nnpm ERR!     /usr/src/app/npm-debug.log\r\n```\r\n\r\nSo here I see the logical application-level error! Great. Now, how can we make this way easier?"}}