{"action":"opened","issue":{"url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/626","labels_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/626/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/626/comments","events_url":"https://api.github.com/repos/mesosphere/kubernetes-mesos/issues/626/events","html_url":"https://github.com/mesosphere/kubernetes-mesos/issues/626","id":117422785,"number":626,"title":"Scheduling frequently failed with ExitCode:2","user":{"login":"karlkfi","id":3980984,"avatar_url":"https://avatars.githubusercontent.com/u/3980984?v=3","gravatar_id":"","url":"https://api.github.com/users/karlkfi","html_url":"https://github.com/karlkfi","followers_url":"https://api.github.com/users/karlkfi/followers","following_url":"https://api.github.com/users/karlkfi/following{/other_user}","gists_url":"https://api.github.com/users/karlkfi/gists{/gist_id}","starred_url":"https://api.github.com/users/karlkfi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karlkfi/subscriptions","organizations_url":"https://api.github.com/users/karlkfi/orgs","repos_url":"https://api.github.com/users/karlkfi/repos","events_url":"https://api.github.com/users/karlkfi/events{/privacy}","received_events_url":"https://api.github.com/users/karlkfi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-11-17T18:49:00Z","updated_at":"2015-11-17T18:49:00Z","closed_at":null,"body":"Deploy k8s, etcd (may not be critical to duplication) & cassandra:\r\n\r\n```\r\ndcos config prepend package.sources https://github.com/mesosphere/multiverse/archive/version-1.x.zip\r\ndcos package update --validate\r\ndcos package install cassandra\r\ndcos package install etcd\r\ncat >/tmp/options.json <<EOF\r\n{\r\n  \"kubernetes\": {\r\n    \"etcd-mesos-framework-name\": \"etcd\"\r\n  }\r\n}\r\nEOF\r\ndcos package install --options=/tmp/options.json kubernetes\r\n```\r\n\r\nDeploy [OinkerGo](https://github.com/karlkfi/oinker-go):\r\n\r\n```\r\ndcos kubectl create -f ~/go/src/github.com/karlkfi/oinker-go/kubernetes.yaml\r\n```\r\n\r\nCheck the status of the pods:\r\n\r\n```\r\n$ dcos kubectl get pod -l=app=oinker\r\nNAME              READY     STATUS       RESTARTS   AGE\r\noinker-go-2qpwc   0/1       Pending      0          14s\r\noinker-go-75cjf   0/1       Pending      0          14s\r\noinker-go-7url5   0/1       Pending      0          14s\r\noinker-go-7xanx   0/1       Pending      0          14s\r\noinker-go-8ur4h   0/1       Pending      0          14s\r\noinker-go-dal6r   0/1       Pending      0          14s\r\noinker-go-eu2xw   0/1       Pending      0          14s\r\noinker-go-kk9yr   1/1       Running      0          43s\r\noinker-go-m5yc8   0/1       Running      0          14s\r\noinker-go-rxflp   0/1       Running      0          14s\r\noinker-go-sukc1   0/1       Running      0          14s\r\noinker-go-tei77   0/1       Running      0          14s\r\noinker-go-uyged   1/1       Running      0          43s\r\noinker-go-v9r41   0/1       Running      0          14s\r\noinker-go-vyu2h   0/1       ExitCode:2   1          14s\r\noinker-go-ygmxz   0/1       Pending      0          14s\r\noinker-go-yni2u   1/1       Running      0          43s\r\noinker-go-zfb72   0/1       Pending      0          14s\r\n```\r\n\r\nWait a few seconds and try again:\r\n\r\n```\r\n$ dcos kubectl get pod -l=app=oinker\r\nNAME              READY     STATUS       RESTARTS   AGE\r\noinker-go-2qpwc   0/1       Pending      0          29s\r\noinker-go-75cjf   1/1       Running      0          29s\r\noinker-go-7url5   0/1       Running      1          29s\r\noinker-go-7xanx   0/1       Running      0          29s\r\noinker-go-8ur4h   0/1       ExitCode:2   1          29s\r\noinker-go-dal6r   0/1       ExitCode:2   0          29s\r\noinker-go-eu2xw   0/1       Running      0          29s\r\noinker-go-kk9yr   1/1       Running      0          58s\r\noinker-go-m5yc8   0/1       ExitCode:2   2          29s\r\noinker-go-rxflp   0/1       ExitCode:2   2          29s\r\noinker-go-sukc1   0/1       ExitCode:2   2          29s\r\noinker-go-tei77   0/1       ExitCode:2   1          29s\r\noinker-go-uyged   1/1       Running      0          58s\r\noinker-go-v9r41   1/1       Running      0          29s\r\noinker-go-vyu2h   0/1       Running      3          29s\r\noinker-go-ygmxz   0/1       Running      0          29s\r\noinker-go-yni2u   1/1       Running      0          58s\r\noinker-go-zfb72   0/1       ExitCode:2   1          29s\r\n```\r\n\r\nEvents on one of the failing pods:\r\n\r\n```\r\nEvents:\r\n  FirstSeen\t\t\t\tLastSeen\t\t\tCount\tFrom\t\t\tSubobjectPath\t\t\t\tReason\t\t\tMessage\r\n  Tue, 17 Nov 2015 10:34:13 -0800\tTue, 17 Nov 2015 10:34:15 -0800\t3\t{scheduler }\t\t\t\t\t\t\tfailedScheduling\tError scheduling: No suitable offers for pod/task\r\n```\r\n\r\nEventually the pods all get scheduled, but it can take 0-12 restarts.\r\n\r\n```\r\n$ dcos kubectl get pod -l=app=oinker\r\nNAME              READY     STATUS    RESTARTS   AGE\r\noinker-go-2qpwc   1/1       Running   2          9m\r\noinker-go-75cjf   1/1       Running   0          9m\r\noinker-go-7url5   1/1       Running   2          9m\r\noinker-go-7xanx   1/1       Running   2          9m\r\noinker-go-8ur4h   1/1       Running   2          9m\r\noinker-go-dal6r   1/1       Running   2          9m\r\noinker-go-eu2xw   1/1       Running   2          9m\r\noinker-go-kk9yr   1/1       Running   0          9m\r\noinker-go-m5yc8   1/1       Running   2          9m\r\noinker-go-rxflp   1/1       Running   2          9m\r\noinker-go-sukc1   1/1       Running   2          9m\r\noinker-go-tei77   1/1       Running   2          9m\r\noinker-go-uyged   1/1       Running   0          9m\r\noinker-go-v9r41   1/1       Running   0          9m\r\noinker-go-vyu2h   0/1       API error (500): Cannot start container 3b838db332dbb27dc67758126f6f77251ecafb42d8d7015d36a3e9e33ceeee20: cannot join network of a non running container: 8b3bdc9ad26f89df56f78df1a2af8b2736058c407705504832243834c8c5d854\r\n                  8         9m\r\noinker-go-ygmxz   1/1       Running   2         9m\r\noinker-go-yni2u   1/1       Running   0         9m\r\noinker-go-zfb72   1/1       Running   2         9m\r\n```\r\n\r\n(I think this `cannot join network of a non running container` is a different bug.)"}}