{"action":"created","issue":{"url":"https://api.github.com/repos/deis/workflow/issues/99","labels_url":"https://api.github.com/repos/deis/workflow/issues/99/labels{/name}","comments_url":"https://api.github.com/repos/deis/workflow/issues/99/comments","events_url":"https://api.github.com/repos/deis/workflow/issues/99/events","html_url":"https://github.com/deis/workflow/issues/99","id":122597925,"number":99,"title":"server error talking to controller when launching during a `git push` based deploy","user":{"login":"arschles","id":70865,"avatar_url":"https://avatars.githubusercontent.com/u/70865?v=3","gravatar_id":"","url":"https://api.github.com/users/arschles","html_url":"https://github.com/arschles","followers_url":"https://api.github.com/users/arschles/followers","following_url":"https://api.github.com/users/arschles/following{/other_user}","gists_url":"https://api.github.com/users/arschles/gists{/gist_id}","starred_url":"https://api.github.com/users/arschles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arschles/subscriptions","organizations_url":"https://api.github.com/users/arschles/orgs","repos_url":"https://api.github.com/users/arschles/repos","events_url":"https://api.github.com/users/arschles/events{/privacy}","received_events_url":"https://api.github.com/users/arschles/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-12-16T20:48:46Z","updated_at":"2015-12-16T23:48:57Z","closed_at":null,"body":"On a `git push` from the [`example-go` repo](https://github.com/deis/example-go) (a buildpack build), the following output results:\r\n\r\n```console\r\nENG000656:example-go aaronschlesinger$ git push deis master\r\nThe authenticity of host '[deis.micro-kube.local]:2222 ([172.17.8.100]:2222)' can't be established.\r\nECDSA key fingerprint is SHA256:0ZiwB2heubH7jTpArCHN6Mf009u6TtgDDS99ZwUg/KM.\r\nAre you sure you want to continue connecting (yes/no)? yes\r\nWarning: Permanently added '[deis.micro-kube.local]:2222,[172.17.8.100]:2222' (ECDSA) to the list of known hosts.\r\nCounting objects: 75, done.\r\nDelta compression using up to 8 threads.\r\nCompressing objects: 100% (40/40), done.\r\nWriting objects: 100% (75/75), 18.20 KiB | 0 bytes/s, done.\r\nTotal 75 (delta 30), reused 75 (delta 30)\r\nAdded host ‘http://10.3.0.36:9000’ successfully.\r\nmc: <ERROR> Unable to make bucket ‘http://10.3.0.36:9000/git’. The requested bucket name is not available.\r\n‘wicker-radiator.tar.gz’ -> ‘http://10.3.0.36:9000/git/home/wicker-radiator:git-5450cbcd/tar’\r\nTotal: 2.76 KB, Transferred: 2.76 KB, Speed: 820.28 KB/s\r\n-----> stored tarfile in http://10.3.0.36:9000/git/home/wicker-radiator:git-5450cbcd/tar\r\n-----> creating builder pod in namespace deis\r\npod \"wicker-radiator-git-5450cbcd\" created\r\nWaiting for build to complete...\r\n............................................................\r\nBuild complete.\r\n-----> Launching... \r\n2015/12/16 19:29:58 failed retrieving config from controller: <h1>Server Error (500)</h1>\r\nTo ssh://git@deis.micro-kube.local:2222/wicker-radiator.git\r\n * [new branch]      master -> master\r\n```\r\n\r\nNote the following:\r\n\r\n1. I was able to successfully push and launch a new app yesterday evening\r\n2. This push was done on a fresh deis cluster, fresh app\r\n3. The first two pushes failed, and I saw a traceback in workflow logs (see below), but _subsequent pushes seem to succeed_.\r\n\r\n```console\r\n10.2.76.1 \"POST /v2/hooks/build HTTP/1.1\" 500 27 \"deis-builder\"\r\n10.2.76.1 \"GET /v2/ HTTP/1.1\" 401 58 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"POST /v2/auth/register/ HTTP/1.1\" 400 43 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"GET /v2/ HTTP/1.1\" 401 58 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"POST /v2/auth/login/ HTTP/1.1\" 200 52 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"POST /v2/keys/ HTTP/1.1\" 201 983 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"POST /v2/keys/ HTTP/1.1\" 400 41 \"Deis Client v2.0.0-dev\"\r\nINFO wicker-radiator: config wicker-radiator-fe4aa59 updated\r\nINFO wicker-radiator: release wicker-radiator-v1 created\r\nINFO [wicker-radiator]: arschles created initial release\r\n10.2.76.1 \"POST /v2/apps/ HTTP/1.1\" 201 208 \"Deis Client v2.0.0-dev\"\r\n10.2.76.1 \"POST /v2/hooks/push HTTP/1.1\" 201 448 \"curl/7.39.0\"\r\n10.2.76.1 \"POST /v2/hooks/config HTTP/1.1\" 200 203 \"deis-builder\"\r\nINFO wicker-radiator: build wicker-radiator-630cd8a created\r\nINFO wicker-radiator: release wicker-radiator-v2 created\r\nINFO [wicker-radiator]: arschles deployed 5450cbc\r\nINFO Pulling Docker image 10.3.0.185:5000/wicker-radiator:git-5450cbcd\r\nINFO wicker-radiator: 500 Server Error: Internal Server Error (\"invalid registry endpoint https://10.3.0.185:5000/v0/: unable to ping registry endpoint https://10.3.0.185:5000/v0/\r\nv2 ping attempt failed with error: Get https://10.3.0.185:5000/v2/: tls: oversized record received with length 20527\r\n v1 ping attempt failed with error: Get https://10.3.0.185:5000/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry 10.3.0.185:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/10.3.0.185:5000/ca.crt\")\r\nINFO [wicker-radiator]: 500 Server Error: Internal Server Error (\"invalid registry endpoint https://10.3.0.185:5000/v0/: unable to ping registry endpoint https://10.3.0.185:5000/v0/\r\nv2 ping attempt failed with error: Get https://10.3.0.185:5000/v2/: tls: oversized record received with length 20527\r\n v1 ping attempt failed with error: Get https://10.3.0.185:5000/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry 10.3.0.185:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/10.3.0.185:5000/ca.crt\")\r\nINFO wicker-radiator: arschles scaled containers web=1\r\nINFO [wicker-radiator]: arschles scaled containers web=1\r\nDEBUG scale wicker-radiator_v2.web, img wicker-radiator:v2, params {u'num': 1, u'tags': {}, u'aname': u'wicker-radiator', u'version': u'v2', u'memory': {}, u'cpu': {}}, cmd \"start web\"\r\nDEBUG create wicker-radiator_v2.web, img wicker-radiator:v2, params {u'version': u'v2', u'tags': {}, u'aname': u'wicker-radiator', u'num': 1, u'memory': {}, u'cpu': {}}, cmd \"start web\"\r\nERROR wicker-radiator: wicker-radiator_v2.web (scale): u'containerID'\r\nERROR [wicker-radiator]: wicker-radiator_v2.web (scale): u'containerID'\r\nERROR Internal Server Error: /v2/hooks/build\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.7/site-packages/django/core/handlers/base.py\", line 112, in get_response\r\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/django/views/decorators/csrf.py\", line 57, in wrapped_view\r\n    return view_func(*args, **kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/rest_framework/viewsets.py\", line 85, in view\r\n    return self.dispatch(request, *args, **kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/rest_framework/views.py\", line 407, in dispatch\r\n    response = self.handle_exception(exc)\r\n  File \"/usr/lib/python2.7/site-packages/rest_framework/views.py\", line 404, in dispatch\r\n    response = handler(request, *args, **kwargs)\r\n  File \"/app/api/views.py\", line 403, in create\r\n    super(BuildHookViewSet, self).create(request, *args, **kwargs)\r\n  File \"/app/api/views.py\", line 129, in create\r\n    return super(BaseDeisViewSet, self).create(request, *args, **kwargs)\r\n  File \"/usr/lib/python2.7/site-packages/rest_framework/mixins.py\", line 21, in create\r\n    self.perform_create(serializer)\r\n  File \"/app/api/viewsets.py\", line 21, in perform_create\r\n    self.post_save(obj)\r\n  File \"/app/api/views.py\", line 410, in post_save\r\n    build.create(self.user)\r\n  File \"/app/api/models.py\", line 738, in create\r\n    self.app.deploy(user, new_release)\r\n  File \"/app/api/models.py\", line 466, in deploy\r\n    self._default_scale(user, release)\r\n  File \"/app/api/models.py\", line 510, in _default_scale\r\n    self.scale(user, structure)\r\n  File \"/app/api/models.py\", line 296, in scale\r\n    self._scale_containers(scale_types, to_remove)\r\n  File \"/app/api/models.py\", line 329, in _scale_containers\r\n    **kwargs\r\n  File \"/app/scheduler/k8s.py\", line 295, in scale\r\n    self.create(name, image, command, **kwargs)\r\n  File \"/app/scheduler/k8s.py\", line 406, in create\r\n    self._create_service(name, app_name, app_type, data)\r\n  File \"/app/scheduler/k8s.py\", line 436, in _create_service\r\n    container_id = actual_pod['status']['containerStatuses'][0]['containerID'].split(\"//\")[1]\r\nKeyError: u'containerID'\r\n10.2.76.1 \"POST /v2/hooks/build HTTP/1.1\" 500 27 \"deis-builder\"\r\n10.2.76.1 \"POST /v2/hooks/push HTTP/1.1\" 201 448 \"curl/7.39.0\"\r\n10.2.76.1 \"POST /v2/hooks/config HTTP/1.1\" 200 203 \"deis-builder\"\r\nINFO wicker-radiator: build wicker-radiator-52947c0 created\r\nINFO wicker-radiator: release wicker-radiator-v3 created\r\nINFO [wicker-radiator]: arschles deployed 6b2d12b\r\nINFO Pulling Docker image 10.3.0.185:5000/wicker-radiator:git-6b2d12b8\r\nINFO wicker-radiator: 500 Server Error: Internal Server Error (\"invalid registry endpoint https://10.3.0.185:5000/v0/: unable to ping registry endpoint https://10.3.0.185:5000/v0/\r\nv2 ping attempt failed with error: Get https://10.3.0.185:5000/v2/: tls: oversized record received with length 20527\r\n v1 ping attempt failed with error: Get https://10.3.0.185:5000/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry 10.3.0.185:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/10.3.0.185:5000/ca.crt\")\r\nINFO [wicker-radiator]: 500 Server Error: Internal Server Error (\"invalid registry endpoint https://10.3.0.185:5000/v0/: unable to ping registry endpoint https://10.3.0.185:5000/v0/\r\nv2 ping attempt failed with error: Get https://10.3.0.185:5000/v2/: tls: oversized record received with length 20527\r\n v1 ping attempt failed with error: Get https://10.3.0.185:5000/v1/_ping: tls: oversized record received with length 20527. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry 10.3.0.185:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/10.3.0.185:5000/ca.crt\")\r\nDEBUG deploy wicker-radiator_v3.web, img wicker-radiator:v3, params {u'version': u'v3', u'tags': {}, u'aname': u'wicker-radiator', u'num': 0, u'memory': {}, u'cpu': {}}, cmd \"start web\"\r\nINFO wicker-radiator: arschles scaled containers web=1\r\nINFO [wicker-radiator]: arschles scaled containers web=1\r\n10.2.76.1 \"POST /v2/hooks/build HTTP/1.1\" 200 65 \"deis-builder\"\r\n```"},"comment":{"url":"https://api.github.com/repos/deis/workflow/issues/comments/165291238","html_url":"https://github.com/deis/workflow/issues/99#issuecomment-165291238","issue_url":"https://api.github.com/repos/deis/workflow/issues/99","id":165291238,"user":{"login":"arschles","id":70865,"avatar_url":"https://avatars.githubusercontent.com/u/70865?v=3","gravatar_id":"","url":"https://api.github.com/users/arschles","html_url":"https://github.com/arschles","followers_url":"https://api.github.com/users/arschles/followers","following_url":"https://api.github.com/users/arschles/following{/other_user}","gists_url":"https://api.github.com/users/arschles/gists{/gist_id}","starred_url":"https://api.github.com/users/arschles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arschles/subscriptions","organizations_url":"https://api.github.com/users/arschles/orgs","repos_url":"https://api.github.com/users/arschles/repos","events_url":"https://api.github.com/users/arschles/events{/privacy}","received_events_url":"https://api.github.com/users/arschles/received_events","type":"User","site_admin":false},"created_at":"2015-12-16T23:48:57Z","updated_at":"2015-12-16T23:48:57Z","body":"I think since we have docs in place and this is not a bug with deis, we can close. thoughts?"}}