{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16201","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16201/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16201/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16201/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16201","id":113109514,"number":16201,"title":"Heapster crash looping in version skewed cluster (v1.1 master, v1.0 node)","user":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/upgrade","name":"area/upgrade","color":"bfd4f2"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P0","name":"priority/P0","color":"e11d21"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/13","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.1","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/13/labels","id":1220916,"number":13,"title":"v1.1","description":"Release 1.1.\r\n\r\nP1 features:\r\n* 250 nodes\r\n* kube-proxy perf. improvements (via IPTables)\r\n* Docker 1.8.x\r\n* Horizontal pod auto-scaling\r\n* Job Object\r\n* Schema validation in kubectl\r\n* Experimental API and API groups\r\n* Graceful pod termination\r\n","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":46,"closed_issues":223,"state":"open","created_at":"2015-07-22T19:17:12Z","updated_at":"2015-10-24T03:34:42Z","due_on":"2015-09-21T07:00:00Z","closed_at":null},"comments":5,"created_at":"2015-10-23T22:17:15Z","updated_at":"2015-10-24T03:34:42Z","closed_at":null,"body":"Heapster pod: \r\n```console\r\n$  kubernetes git:(master) k --namespace=kube-system get po heapster-v10-hscvw      -o json\r\n{\r\n    \"kind\": \"Pod\",\r\n    \"apiVersion\": \"v1\",\r\n    \"metadata\": {\r\n        \"name\": \"heapster-v10-hscvw\",\r\n        \"generateName\": \"heapster-v10-\",\r\n        \"namespace\": \"kube-system\",\r\n        \"selfLink\": \"/api/v1/namespaces/kube-system/pods/heapster-v10-hscvw\",\r\n        \"uid\": \"6998dd02-79bf-11e5-92c1-42010af00002\",\r\n        \"resourceVersion\": \"6878\",\r\n        \"creationTimestamp\": \"2015-10-23T19:51:12Z\",\r\n        \"labels\": {\r\n            \"k8s-app\": \"heapster\",\r\n            \"kubernetes.io/cluster-service\": \"true\",\r\n            \"version\": \"v10\"\r\n        },\r\n        \"annotations\": {\r\n            \"kubernetes.io/created-by\": \"{\\\"kind\\\":\\\"SerializedReference\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"reference\\\":{\\\"kind\\\":\\\"ReplicationController\\\",\\\"namespace\\\":\\\"kube-system\\\",\\\"name\\\":\\\"heapster-v10\\\",\\\"uid\\\":\\\"6991c9d1-79bf-11e5-92c1-42010af00002\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"resourceVersion\\\":\\\"34\\\"}}\\n\"\r\n        }\r\n    },\r\n    \"spec\": {\r\n        \"volumes\": [\r\n            {\r\n                \"name\": \"ssl-certs\",\r\n                \"hostPath\": {\r\n                    \"path\": \"/etc/ssl/certs\"\r\n                }\r\n            },\r\n            {\r\n                \"name\": \"default-token-2vb6z\",\r\n                \"secret\": {\r\n                    \"secretName\": \"default-token-2vb6z\"\r\n                }\r\n            }\r\n        ],\r\n        \"containers\": [\r\n            {\r\n                \"name\": \"heapster\",\r\n                \"image\": \"gcr.io/google_containers/heapster:v0.18.2\",\r\n                \"command\": [\r\n                    \"/heapster\",\r\n                    \"--source=kubernetes:''\",\r\n                    \"--sink=gcl\",\r\n                    \"--sink=gcmautoscaling\",\r\n                    \"--sink=influxdb:http://monitoring-influxdb:8086\",\r\n                        \"memory\": \"300Mi\"\r\n                    }\r\n                },\r\n                \"volumeMounts\": [\r\n                    {\r\n                        \"name\": \"ssl-certs\",\r\n                        \"readOnly\": true,\r\n                        \"mountPath\": \"/etc/ssl/certs\"\r\n                    },\r\n                    {\r\n                        \"name\": \"default-token-2vb6z\",\r\n                        \"readOnly\": true,\r\n                        \"mountPath\": \"/var/run/secrets/kubernetes.io/serviceaccount\"\r\n                    }\r\n                ],\r\n                \"terminationMessagePath\": \"/dev/termination-log\",\r\n                \"imagePullPolicy\": \"IfNotPresent\"\r\n            }\r\n        ],\r\n        \"restartPolicy\": \"Always\",\r\n        \"terminationGracePeriodSeconds\": 30,\r\n        \"dnsPolicy\": \"ClusterFirst\",\r\n        \"serviceAccountName\": \"default\",\r\n        \"serviceAccount\": \"default\",\r\n        \"nodeName\": \"k-minion-wziz\",\r\n        \"securityContext\": {}\r\n    },\r\n    \"status\": {\r\n        \"phase\": \"Running\",\r\n        \"conditions\": [\r\n            {\r\n                \"type\": \"Ready\",\r\n                \"status\": \"False\",\r\n                \"lastProbeTime\": null,\r\n                \"lastTransitionTime\": \"2015-10-23T22:13:26Z\",\r\n                \"reason\": \"ContainersNotReady\",\r\n                \"message\": \"containers with unready status: [heapster]\"\r\n            }\r\n        ],\r\n        \"hostIP\": \"10.240.0.4\",\r\n        \"podIP\": \"10.244.3.3\",\r\n        \"startTime\": \"2015-10-23T19:52:15Z\",\r\n        \"containerStatuses\": [\r\n            {\r\n                \"name\": \"heapster\",\r\n                \"state\": {\r\n                    \"waiting\": {\r\n                        \"reason\": \"CrashLoopBackOff\",\r\n                        \"message\": \"Back-off 5m0s restarting failed container=heapster pod=heapster-v10-hscvw_kube-system\"\r\n                    }\r\n                },\r\n                \"lastState\": {\r\n                    \"terminated\": {\r\n                        \"exitCode\": 255,\r\n                        \"reason\": \"Error\",\r\n                        \"startedAt\": \"2015-10-23T22:13:16Z\",\r\n                        \"finishedAt\": \"2015-10-23T22:13:17Z\",\r\n                        \"containerID\": \"docker://6eb13b48f081a72bc0e02e7acfc2c7e6190f7b8ef1e563a9df341dc33937c52f\"\r\n                    }\r\n                },\r\n                \"ready\": false,\r\n                \"restartCount\": 33,\r\n                \"image\": \"gcr.io/google_containers/heapster:v0.18.2\",\r\n                \"imageID\": \"docker://899df50acd84baf1f21fe6936b7fba2b9417586483bf335e2a7654c15c6e0589\",\r\n                \"containerID\": \"docker://6eb13b48f081a72bc0e02e7acfc2c7e6190f7b8ef1e563a9df341dc33937c52f\"\r\n            }\r\n        ]\r\n    }\r\n}\r\n```\r\n\r\nAnd logs:\r\n\r\n```console\r\n$  kubernetes git:(master) k --namespace=kube-system logs --previous heapster-v10-hscvw \r\nI1023 22:13:16.564111       1 heapster.go:55] /heapster --source=kubernetes:'' --sink=gcl --sink=gcmautoscaling --sink=influxdb:http://monitoring-influxdb:8086 --stats_resolution=30s --sink_frequency=1m\r\nI1023 22:13:16.564207       1 heapster.go:56] Heapster version 0.18.2\r\nI1023 22:13:16.564656       1 kube_factory.go:169] Using Kubernetes client with master \"https://10.0.0.1:443\" and version \"v1\"\r\nI1023 22:13:16.564691       1 kube_factory.go:170] Using kubelet port 10255\r\nI1023 22:13:16.586895       1 driver.go:243] Project ID for GCL sink is: \"mikedanese-k8s\"\r\nI1023 22:13:16.586947       1 driver.go:263] creating GCL sink\r\nE1023 22:13:17.675570       1 core.go:97] [GCM] list metrics failed request &{Method:GET URL:https://www.googleapis.com/cloudmonitoring/v2beta2/projects/mikedanese-k8s/metricDescriptors Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[application/json] Authorization:[Bearer ya29.FQIVkt9SUOVgCxwJFtG3h9M9mrP9xX-ak3yRIP2s-z4qvfoAkl-KdBpipYIxbgO85YBy]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:www.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil>} failed with status \"403 Forbidden\" and response: &{Status:403 Forbidden StatusCode:403 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[application/json; charset=UTF-8] Vary:[Origin X-Origin] X-Content-Type-Options:[nosniff] Server:[GSE] X-Xss-Protection:[1; mode=block] Alternate-Protocol:[443:quic,p=1] Alt-Svc:[quic=\":443\"; p=\"1\"; ma=604800] Date:[Fri, 23 Oct 2015 22:13:17 GMT] Cache-Control:[private, max-age=0] X-Frame-Options:[SAMEORIGIN] Expires:[Fri, 23 Oct 2015 22:13:17 GMT]] Body:0x4c208264e40 ContentLength:-1 TransferEncoding:[chunked] Close:false Trailer:map[] Request:0x4c208033ee0 TLS:0x4c2081af480}, Body: \"{\\n \\\"error\\\": {\\n  \\\"errors\\\": [\\n   {\\n    \\\"domain\\\": \\\"usageLimits\\\",\\n    \\\"reason\\\": \\\"accessNotConfigured\\\",\\n    \\\"message\\\": \\\"Access Not Configured. The API (Cloud Monitoring API) is not enabled for your project. Please use the Google Developers Console to update your configuration.\\\",\\n    \\\"extendedHelp\\\": \\\"https://console.developers.google.com\\\"\\n   }\\n  ],\\n  \\\"code\\\": 403,\\n  \\\"message\\\": \\\"Access Not Configured. The API (Cloud Monitoring API) is not enabled for your project. Please use the Google Developers Console to update your configuration.\\\"\\n }\\n}\\n\"\r\nI1023 22:13:17.675695       1 driver.go:284] created GCM Autocaling sink\r\nF1023 22:13:17.675716       1 heapster.go:62] request &{Method:GET URL:https://www.googleapis.com/cloudmonitoring/v2beta2/projects/mikedanese-k8s/metricDescriptors Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[application/json] Authorization:[Bearer ya29.FQIVkt9SUOVgCxwJFtG3h9M9mrP9xX-ak3yRIP2s-z4qvfoAkl-KdBpipYIxbgO85YBy]] Body:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:www.googleapis.com Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil>} failed with status \"403 Forbidden\" and response: &{Status:403 Forbidden StatusCode:403 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[application/json; charset=UTF-8] Vary:[Origin X-Origin] X-Content-Type-Options:[nosniff] Server:[GSE] X-Xss-Protection:[1; mode=block] Alternate-Protocol:[443:quic,p=1] Alt-Svc:[quic=\":443\"; p=\"1\"; ma=604800] Date:[Fri, 23 Oct 2015 22:13:17 GMT] Cache-Control:[private, max-age=0] X-Frame-Options:[SAMEORIGIN] Expires:[Fri, 23 Oct 2015 22:13:17 GMT]] Body:0x4c208264e40 ContentLength:-1 TransferEncoding:[chunked] Close:false Trailer:map[] Request:0x4c208033ee0 TLS:0x4c2081af480}, Body: \"{\\n \\\"error\\\": {\\n  \\\"errors\\\": [\\n   {\\n    \\\"domain\\\": \\\"usageLimits\\\",\\n    \\\"reason\\\": \\\"accessNotConfigured\\\",\\n    \\\"message\\\": \\\"Access Not Configured. The API (Cloud Monitoring API) is not enabled for your project. Please use the Google Developers Console to update your configuration.\\\",\\n    \\\"extendedHelp\\\": \\\"https://console.developers.google.com\\\"\\n   }\\n  ],\\n  \\\"code\\\": 403,\\n  \\\"message\\\": \\\"Access Not Configured. The API (Cloud Monitoring API) is not enabled for your project. Please use the Google Developers Console to update your configuration.\\\"\\n }\\n}\\n\"\r\n```\r\n\r\ncc @vishh @davidopp @ihmccreery \r\n\r\np0 for triage"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/150748370","html_url":"https://github.com/kubernetes/kubernetes/issues/16201#issuecomment-150748370","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16201","id":150748370,"user":{"login":"mikedanese","id":1787169,"avatar_url":"https://avatars.githubusercontent.com/u/1787169?v=3","gravatar_id":"","url":"https://api.github.com/users/mikedanese","html_url":"https://github.com/mikedanese","followers_url":"https://api.github.com/users/mikedanese/followers","following_url":"https://api.github.com/users/mikedanese/following{/other_user}","gists_url":"https://api.github.com/users/mikedanese/gists{/gist_id}","starred_url":"https://api.github.com/users/mikedanese/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikedanese/subscriptions","organizations_url":"https://api.github.com/users/mikedanese/orgs","repos_url":"https://api.github.com/users/mikedanese/repos","events_url":"https://api.github.com/users/mikedanese/events{/privacy}","received_events_url":"https://api.github.com/users/mikedanese/received_events","type":"User","site_admin":false},"created_at":"2015-10-24T03:34:41Z","updated_at":"2015-10-24T03:34:41Z","body":"Need to cherry pick so leave open"}}