{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11634","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11634/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11634/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11634/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/11634","id":96196858,"number":11634,"title":"DNS Service Resolution (v1.0.1)","user":{"login":"kenjones-cisco","id":7967510,"avatar_url":"https://avatars.githubusercontent.com/u/7967510?v=3","gravatar_id":"","url":"https://api.github.com/users/kenjones-cisco","html_url":"https://github.com/kenjones-cisco","followers_url":"https://api.github.com/users/kenjones-cisco/followers","following_url":"https://api.github.com/users/kenjones-cisco/following{/other_user}","gists_url":"https://api.github.com/users/kenjones-cisco/gists{/gist_id}","starred_url":"https://api.github.com/users/kenjones-cisco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kenjones-cisco/subscriptions","organizations_url":"https://api.github.com/users/kenjones-cisco/orgs","repos_url":"https://api.github.com/users/kenjones-cisco/repos","events_url":"https://api.github.com/users/kenjones-cisco/events{/privacy}","received_events_url":"https://api.github.com/users/kenjones-cisco/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/networking","name":"area/networking","color":"006b75"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/support","name":"kind/support","color":"eb6420"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-07-21T00:26:44Z","updated_at":"2015-07-21T13:56:22Z","closed_at":null,"body":"In an Openstack cluster running Centos7 (1 master, 3 nodes) with DNS addon enabled.\r\n\r\n*Node status*\r\n```\r\n[centos@cchub-control-01 ~]$ kubectl get nodes\r\nNAME              LABELS    STATUS\r\ncchub-worker-01   <none>    Ready\r\ncchub-worker-02   <none>    Ready\r\ncchub-worker-03   <none>    Ready\r\n```\r\n\r\n*Pods, ReplicationControllers, Services:*\r\n```\r\n[centos@cchub-control-01 ~]$ kubectl get rc,svc,po --all-namespaces\r\nNAMESPACE     CONTROLLER    CONTAINER(S)   IMAGE(S)                                         SELECTOR                      REPLICAS\r\nkube-system   kube-dns-v8   etcd           gcr.io/google_containers/etcd:2.0.9              k8s-app=kube-dns,version=v8   1\r\n                            kube2sky       gcr.io/google_containers/kube2sky:1.11\r\n                            skydns         gcr.io/google_containers/skydns:2015-03-11-001\r\n                            healthz        gcr.io/google_containers/exechealthz:1.0\r\nNAMESPACE     NAME         LABELS                                                                           SELECTOR           IP(S)          PORT(S)\r\ndefault       kubernetes   component=apiserver,provider=kubernetes                                          <none>             192.168.0.1    443/TCP\r\nkube-system   kube-dns     k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS   k8s-app=kube-dns   192.168.0.10   53/UDP\r\n                                                                                                                                              53/TCP\r\nNAMESPACE     NAME                READY     STATUS    RESTARTS   AGE\r\ndefault       busybox             1/1       Running   0          3m\r\nkube-system   kube-dns-v8-mwe6s   4/4       Running   0          27m\r\n```\r\n\r\n*Master Processes:*\r\n```\r\n[centos@cchub-control-01 ~]$ sudo systemctl status etcd kube-apiserver kube-controller-manager kube-scheduler -l\r\netcd.service - Etcd Server\r\n   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled)\r\n  Drop-In: /etc/systemd/system/etcd.service.d\r\n           └─10-peers.conf\r\n   Active: active (running) since Mon 2015-07-20 23:44:50 UTC; 14min ago\r\n Main PID: 11334 (etcd)\r\n   CGroup: /system.slice/etcd.service\r\n           └─11334 /usr/bin/etcd\r\n\r\nJul 20 23:44:50 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:50 etcdserver: restart member e353ae4d515a115f in cluster 5a635de6c3d012fd at commit index 74\r\nJul 20 23:44:50 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:50 raft: e353ae4d515a115f became follower at term 2\r\nJul 20 23:44:50 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:50 raft: newRaft e353ae4d515a115f [peers: [], term: 2, commit: 74, applied: 0, lastindex: 74, lastterm: 2]\r\nJul 20 23:44:50 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:50 etcdserver: added local member e353ae4d515a115f [http://10.7.7.87:7001] to cluster 5a635de6c3d012fd\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 raft: e353ae4d515a115f is starting a new election at term 2\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 raft: e353ae4d515a115f became candidate at term 3\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 raft: e353ae4d515a115f received vote from e353ae4d515a115f at term 3\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 raft: e353ae4d515a115f became leader at term 3\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 raft.node: e353ae4d515a115f elected leader e353ae4d515a115f at term 3\r\nJul 20 23:44:52 cchub-control-01.novalocal bash[11334]: 2015/07/20 23:44:52 etcdserver: published {Name:cchub-control-01 ClientURLs:[http://cchub-control-01:4001]} to cluster 5a635de6c3d012fd\r\n\r\nkube-apiserver.service - Kubernetes API Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:44:52 UTC; 14min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11408 (kube-apiserver)\r\n   CGroup: /system.slice/kube-apiserver.service\r\n           └─11408 /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://cchub-control-01:4001 --insecure-bind-address=0.0.0.0 --allow_privileged=true --service-cluster-ip-range=192.168.0.0/16\r\n\r\nJul 20 23:44:51 cchub-control-01.novalocal kube-apiserver[11408]: [restful] 2015/07/20 23:44:51 log.go:30: [restful/swagger] listing is available at https://10.7.7.87:6443/swaggerapi/\r\nJul 20 23:44:51 cchub-control-01.novalocal kube-apiserver[11408]: [restful] 2015/07/20 23:44:51 log.go:30: [restful/swagger] https://10.7.7.87:6443/swaggerui/ is mapped to folder /swagger-ui/\r\nJul 20 23:44:52 cchub-control-01.novalocal kube-apiserver[11408]: I0720 23:44:52.307464   11408 server.go:441] Serving securely on 0.0.0.0:6443\r\nJul 20 23:44:52 cchub-control-01.novalocal kube-apiserver[11408]: I0720 23:44:52.307487   11408 server.go:483] Serving insecurely on 0.0.0.0:8080\r\nJul 20 23:44:52 cchub-control-01.novalocal kube-apiserver[11408]: I0720 23:44:52.795100   11408 server.go:456] Using self-signed cert (/var/run/kubernetes/apiserver.crt, /var/run/kubernetes/apiserver.key)\r\nJul 20 23:44:52 cchub-control-01.novalocal systemd[1]: Started Kubernetes API Server.\r\nJul 20 23:46:05 cchub-control-01.novalocal kube-apiserver[11408]: I0720 23:46:05.242851   11408 trace.go:57] Trace \"Update *api.Node\" (started 2015-07-20 23:46:03.814711463 +0000 UTC):\r\nJul 20 23:46:05 cchub-control-01.novalocal kube-apiserver[11408]: [1.428023443s] [1.428023443s] END\r\nJul 20 23:47:09 cchub-control-01.novalocal kube-apiserver[11408]: I0720 23:47:09.874136   11408 trace.go:57] Trace \"Update *api.Node\" (started 2015-07-20 23:47:08.418956476 +0000 UTC):\r\nJul 20 23:47:09 cchub-control-01.novalocal kube-apiserver[11408]: [1.455151103s] [1.455151103s] END\r\n\r\nkube-controller-manager.service - Kubernetes Controller Manager\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:44:53 UTC; 14min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11453 (kube-controller)\r\n   CGroup: /system.slice/kube-controller-manager.service\r\n           └─11453 /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://cchub-control-01:8080\r\n\r\nJul 20 23:44:53 cchub-control-01.novalocal systemd[1]: Starting Kubernetes Controller Manager...\r\nJul 20 23:44:53 cchub-control-01.novalocal systemd[1]: Started Kubernetes Controller Manager.\r\nJul 20 23:44:53 cchub-control-01.novalocal kube-controller-manager[11453]: I0720 23:44:53.189987   11453 plugins.go:69] No cloud provider specified.\r\nJul 20 23:44:53 cchub-control-01.novalocal kube-controller-manager[11453]: I0720 23:44:53.190060   11453 nodecontroller.go:114] Sending events to api server.\r\nJul 20 23:44:53 cchub-control-01.novalocal kube-controller-manager[11453]: E0720 23:44:53.190140   11453 controllermanager.go:198] Failed to start service controller: ServiceController should not be run without a cloudprovider.\r\n\r\nkube-scheduler.service - Kubernetes Scheduler Plugin\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:44:53 UTC; 14min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11496 (kube-scheduler)\r\n   CGroup: /system.slice/kube-scheduler.service\r\n           └─11496 /usr/bin/kube-scheduler --logtostderr=true --v=0 --master=http://cchub-control-01:8080\r\n\r\nJul 20 23:44:53 cchub-control-01.novalocal systemd[1]: Starting Kubernetes Scheduler Plugin...\r\nJul 20 23:44:53 cchub-control-01.novalocal systemd[1]: Started Kubernetes Scheduler Plugin.\r\nJul 20 23:44:53 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:44:53.423902   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:44:54 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:44:54.426966   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:44:56 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:44:56.432990   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:45:00 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:45:00.439380   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:45:08 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:45:08.442412   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:45:24 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:45:24.448159   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:45:56 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:45:56.454069   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'failedScheduling' no nodes available to schedule pods\r\nJul 20 23:46:56 cchub-control-01.novalocal kube-scheduler[11496]: I0720 23:46:56.535197   11496 event.go:203] Event(api.ObjectReference{Kind:\"Pod\", Namespace:\"kube-system\", Name:\"kube-dns-v8-mwe6s\", UID:\"4f750354-2f39-11e5-a5b1-fa163eca8887\", APIVersion:\"v1\", ResourceVersion:\"15\", FieldPath:\"\"}): reason: 'scheduled' Successfully assigned kube-dns-v8-mwe6s to cchub-worker-02\r\n```\r\n\r\n*DNS Responses:*\r\n```\r\n[centos@cchub-control-01 ~]$ kubectl create -f busybox.yaml\r\npods/busybox\r\n[centos@cchub-control-01 ~]$ kubectl get pods busybox\r\nNAME      READY     STATUS    RESTARTS   AGE\r\nbusybox   0/1       Pending   0          9s\r\n[centos@cchub-control-01 ~]$ kubectl get pods busybox\r\nNAME      READY     STATUS    RESTARTS   AGE\r\nbusybox   1/1       Running   0          21s\r\n[centos@cchub-control-01 ~]$ kubectl exec busybox -- nslookup kubernetes\r\nServer:    192.168.0.10\r\nAddress 1: 192.168.0.10\r\n\r\nnslookup: can't resolve 'kubernetes'\r\nerror: Error executing remote command: Error executing command in container: Error executing in Docker Container: 1\r\n[centos@cchub-control-01 ~]$ kubectl exec busybox -- nslookup kubernetes.default.svc.cluster.local\r\nServer:    192.168.0.10\r\nAddress 1: 192.168.0.10\r\n\r\nnslookup: can't resolve 'kubernetes.default.svc.cluster.local'\r\nerror: Error executing remote command: Error executing command in container: Error executing in Docker Container: 1\r\n```\r\n\r\n*Node 1 NAT:*\r\n```\r\n[centos@cchub-worker-01 ~]$ iptables -t nat -L -n -v\r\niptables v1.4.21: can't initialize iptables table `nat': Permission denied (you must be root)\r\nPerhaps iptables or your kernel needs to be upgraded.\r\n[centos@cchub-worker-01 ~]$ sudo iptables -t nat -L -n -v\r\nChain PREROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    1    60 KUBE-PORTALS-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    1    60 KUBE-NODEPORT-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n\r\nChain INPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain OUTPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n  165 11502 KUBE-PORTALS-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    0     0 KUBE-NODEPORT-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL\r\n\r\nChain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0\r\n    2   262 MASQUERADE  all  --  *      *       4.0.0.0/16           0.0.0.0/0            /* flannel_nat */\r\n\r\nChain DOCKER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-PORTALS-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 redir ports 40971\r\n    0     0 REDIRECT   udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 redir ports 52779\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 redir ports 46955\r\n\r\nChain KUBE-PORTALS-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 to:10.7.7.88:40971\r\n    0     0 DNAT       udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 to:10.7.7.88:52779\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 to:10.7.7.88:46955\r\n```\r\n\r\n*Node 1 Processes:*\r\n```\r\n[centos@cchub-worker-01 ~]$ sudo systemctl status kube-proxy\r\nkube-proxy.service - Kubernetes Kube-Proxy Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:37 UTC; 4min 22s ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11860 (kube-proxy)\r\n   CGroup: /system.slice/kube-proxy.service\r\n           └─11860 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://cchub-control-01:8080\r\n\r\nJul 20 23:46:37 cchub-worker-01.novalocal systemd[1]: Started Kubernetes Kube-Proxy Server.\r\n[centos@cchub-worker-01 ~]$ sudo systemctl status flanneld\r\nflanneld.service - Flanneld overlay address etcd agent\r\n   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:35 UTC; 4min 35s ago\r\n  Process: 11588 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)\r\n Main PID: 11582 (flanneld)\r\n   CGroup: /system.slice/flanneld.service\r\n           └─11582 /usr/bin/flanneld -etcd-endpoints=http://cchub-control-01:4001 -etcd-prefix=/coreos.com/network -iface=eth0\r\n\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: recv.success\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: I0720 23:46:35.382275 11582 udp.go:247] Subnet added: 4.0.26.0/24\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: I0720 23:46:35.382386 11582 udp.go:247] Subnet added: 4.0.82.0/24\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: I0720 23:46:35.382397 11582 udp.go:247] Subnet added: 4.0.24.0/24\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: rawWatch /coreos.com/network/subnets []\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: get /coreos.com/network/subnets [http://cchub-control-01:4001]\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: Connecting to etcd: attempt 1 for keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: send.request.to http://cchub-control-01:4001/v2/keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70 | method GET\r\nJul 20 23:46:35 cchub-worker-01.novalocal flanneld[11582]: go-etcd2015/07/20 23:46:35 DEBUG: recv.response.from\r\nJul 20 23:46:35 cchub-worker-01.novalocal systemd[1]: Started Flanneld overlay address etcd agent.\r\n[centos@cchub-worker-01 ~]$ sudo systemctl status docker\r\ndocker.service - Docker Application Container Engine\r\n   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)\r\n  Drop-In: /usr/lib/systemd/system/docker.service.d\r\n           └─flannel.conf\r\n   Active: active (running) since Mon 2015-07-20 23:46:36 UTC; 4min 42s ago\r\n     Docs: http://docs.docker.com\r\n Main PID: 11703 (docker)\r\n   CGroup: /system.slice/docker.service\r\n           └─11703 /usr/bin/docker -d --selinux-enabled --bip=4.0.82.1/24 --mtu=1472\r\n\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"-job version() = OK (0)\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:51:18 cchub-worker-01.novalocal docker[11703]: time=\"2015-07-20T23:51:18Z\" level=info msg=\"-job containers() = OK (0)\"\r\n```\r\n\r\n*Node 2 NAT:*\r\n```\r\n[centos@cchub-worker-02 ~]$ sudo iptables -t nat -L -n -v\r\nChain PREROUTING (policy ACCEPT 6 packets, 460 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n  471 36060 KUBE-PORTALS-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    1    60 KUBE-NODEPORT-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n\r\nChain INPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain OUTPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n  415 26339 KUBE-PORTALS-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    0     0 KUBE-NODEPORT-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL\r\n\r\nChain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0\r\n  474 36499 MASQUERADE  all  --  *      *       4.0.0.0/16           0.0.0.0/0            /* flannel_nat */\r\n\r\nChain DOCKER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-PORTALS-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 redir ports 50724\r\n    0     0 REDIRECT   udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 redir ports 60509\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 redir ports 60549\r\n\r\nChain KUBE-PORTALS-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 to:10.7.7.9:50724\r\n    0     0 DNAT       udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 to:10.7.7.9:60509\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 to:10.7.7.9:60549\r\n```\r\n\r\n*Node 2 Processes:*\r\n```\r\n[centos@cchub-worker-02 ~]$ sudo systemctl status kubelet kube-proxy flanneld docker -l\r\nkubelet.service - Kubernetes Kubelet Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:37 UTC; 7min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11977 (kubelet)\r\n   CGroup: /system.slice/kubelet.service\r\n           └─11977 /usr/bin/kubelet --logtostderr=true --v=0 --api_servers=http://cchub-control-01:8080 --address=0.0.0.0 --hostname_override=cchub-worker-02 --allow_privileged=true --cluster_dns=192.168.0.10 --cluster_domain=cluster.local\r\n\r\nJul 20 23:46:37 cchub-worker-02.novalocal systemd[1]: Started Kubernetes Kubelet Server.\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: W0720 23:46:37.803559   11977 server.go:462] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: W0720 23:46:37.803608   11977 server.go:424] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:37.803687   11977 manager.go:127] cAdvisor running in container: \"/\"\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:37.804049   11977 fs.go:93] Filesystem partitions: map[/dev/vda1:{mountpoint:/ major:253 minor:1}]\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:37.806421   11977 manager.go:156] Machine: {NumCores:4 CpuFrequency:2799998 MemoryCapacity:16660070400 MachineID:64c4f0e99e19435e8480912fc5f820f4 SystemUUID:B0967161-5848-475E-9A81-709FA08B7DD6 BootID:9d4bdad6-677c-42c9-8574-4d49394d65a3 Filesystems:[{Device:/dev/vda1 Capacity:52810084352}] DiskMap:map[253:0:{Name:vda Major:253 Minor:0 Size:53687091200 Scheduler:deadline} 253:16:{Name:vdb Major:253 Minor:16 Size:5368709120 Scheduler:deadline} 253:32:{Name:vdc Major:253 Minor:32 Size:5368709120 Scheduler:deadline} 252:0:{Name:dm-0 Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:fa:16:3e:41:90:a2 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:17179459584 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:1 Memory:0 Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:2 Memory:0 Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]} {Id:3 Memory:0 Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:37.809240   11977 manager.go:163] Version: {KernelVersion:3.10.0-123.el7.x86_64 ContainerOsVersion:CentOS Linux 7 (Core) DockerVersion:1.6.2 CadvisorVersion:0.15.1}\r\nJul 20 23:46:37 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:37.809866   11977 plugins.go:69] No cloud provider specified.\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.878594   11977 docker.go:295] Connecting to docker on unix:///var/run/docker.sock\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.878743   11977 server.go:661] Watching apiserver\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.961465   11977 plugins.go:56] Registering credential provider: .dockercfg\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.966806   11977 server.go:623] Started kubelet\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: E0720 23:46:38.966922   11977 kubelet.go:682] Image garbage collection failed: unable to find data for container /\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.969052   11977 server.go:63] Starting to listen on 0.0.0.0:10250\r\nJul 20 23:46:38 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:38.974599   11977 kubelet.go:702] Running in container \"/kubelet\"\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.047874   11977 factory.go:226] System is using systemd\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.048257   11977 factory.go:234] Registering Docker factory\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.048678   11977 factory.go:89] Registering Raw factory\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.097966   11977 kubelet.go:806] Node cchub-worker-02 was previously registered\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.121172   11977 manager.go:946] Started watching for new ooms in manager\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.121266   11977 oomparser.go:199] OOM parser using kernel log file: \"/var/log/messages\"\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.122695   11977 manager.go:243] Starting recovery of all containers\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.126631   11977 manager.go:248] Recovery completed\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.128493   11977 status_manager.go:76] Starting to sync pod status with apiserver\r\nJul 20 23:46:39 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:39.128527   11977 kubelet.go:1725] Starting kubelet main sync loop.\r\nJul 20 23:46:56 cchub-worker-02.novalocal kubelet[11977]: I0720 23:46:56.676768   11977 provider.go:91] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider\r\n\r\nkube-proxy.service - Kubernetes Kube-Proxy Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:37 UTC; 7min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11897 (kube-proxy)\r\n   CGroup: /system.slice/kube-proxy.service\r\n           └─11897 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://cchub-control-01:8080\r\n\r\nJul 20 23:46:37 cchub-worker-02.novalocal systemd[1]: Started Kubernetes Kube-Proxy Server.\r\n\r\nflanneld.service - Flanneld overlay address etcd agent\r\n   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:35 UTC; 7min ago\r\n  Process: 11624 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)\r\n Main PID: 11618 (flanneld)\r\n   CGroup: /system.slice/flanneld.service\r\n           └─11618 /usr/bin/flanneld -etcd-endpoints=http://cchub-control-01:4001 -etcd-prefix=/coreos.com/network -iface=eth0\r\n\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: Connecting to etcd: attempt 1 for keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=69\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: send.request.to http://cchub-control-01:4001/v2/keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=69 | method GET\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: I0720 23:46:35.379775 11618 udp.go:247] Subnet added: 4.0.82.0/24\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: rawWatch /coreos.com/network/subnets []\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: get /coreos.com/network/subnets [http://cchub-control-01:4001]\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: Connecting to etcd: attempt 1 for keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: send.request.to http://cchub-control-01:4001/v2/keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70 | method GET\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: recv.response.from\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: recv.success\r\nJul 20 23:46:35 cchub-worker-02.novalocal flanneld[11618]: go-etcd2015/07/20 23:46:35 DEBUG: recv.response.from\r\n\r\ndocker.service - Docker Application Container Engine\r\n   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)\r\n  Drop-In: /usr/lib/systemd/system/docker.service.d\r\n           └─flannel.conf\r\n   Active: active (running) since Mon 2015-07-20 23:46:36 UTC; 7min ago\r\n     Docs: http://docs.docker.com\r\n Main PID: 11738 (docker)\r\n   CGroup: /system.slice/docker.service\r\n           └─11738 /usr/bin/docker -d --selinux-enabled --bip=4.0.24.1/24 --mtu=1472\r\n\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"-job container_inspect(ec69606bfe5d2b9768ee6cf9ceb59e02f7a8249ab2e0129c0911d32968ca38ab) = OK (0)\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"GET /containers/d470f6673c9643e3d5ea21402628f7e9c92b32cfee97e56937f6c3597a639cd1/json\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"+job container_inspect(d470f6673c9643e3d5ea21402628f7e9c92b32cfee97e56937f6c3597a639cd1)\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"-job container_inspect(d470f6673c9643e3d5ea21402628f7e9c92b32cfee97e56937f6c3597a639cd1) = OK (0)\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:53:56 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:56Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:53:57 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:57Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:53:57 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:57Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:53:57 cchub-worker-02.novalocal docker[11738]: time=\"2015-07-20T23:53:57Z\" level=info msg=\"-job containers() = OK (0)\"\r\n```\r\n\r\n*Node 3 NAT:*\r\n```\r\n[centos@cchub-worker-03 ~]$ sudo iptables -t nat -L -n -v\r\nChain PREROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    1    60 KUBE-PORTALS-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    1    60 KUBE-NODEPORT-CONTAINER  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n\r\nChain INPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain OUTPUT (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n  183 12816 KUBE-PORTALS-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */\r\n    0     0 KUBE-NODEPORT-HOST  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */\r\n    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL\r\n\r\nChain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0\r\n    2   262 MASQUERADE  all  --  *      *       4.0.0.0/16           0.0.0.0/0            /* flannel_nat */\r\n\r\nChain DOCKER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-NODEPORT-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n\r\nChain KUBE-PORTALS-CONTAINER (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 redir ports 34481\r\n    0     0 REDIRECT   udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 redir ports 44886\r\n    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 redir ports 60545\r\n\r\nChain KUBE-PORTALS-HOST (1 references)\r\n pkts bytes target     prot opt in     out     source               destination\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.1          /* default/kubernetes: */ tcp dpt:443 to:10.7.7.89:34481\r\n    0     0 DNAT       udp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns */ udp dpt:53 to:10.7.7.89:44886\r\n    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            192.168.0.10         /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 to:10.7.7.89:60545\r\n```\r\n\r\n*Node 3 Processes:*\r\n```\r\n[centos@cchub-worker-03 ~]$ sudo systemctl status kubelet kube-proxy flanneld docker\r\nkubelet.service - Kubernetes Kubelet Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:37 UTC; 6min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11932 (kubelet)\r\n   CGroup: /system.slice/kubelet.service\r\n           └─11932 /usr/bin/kubelet --logtostderr=true --v=0 --api_servers=http://cchub-control-01:8080 --address=0.0.0.0 --hostname_override=cchub-worker-03 --allow_privileged=true --cluster_dns=192.168.0.10 --cluster_domain=cluster.local\r\n\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.823581   11932 factory.go:226] System is using systemd\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.824226   11932 factory.go:234] Registering Docker factory\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.824865   11932 factory.go:89] Registering Raw factory\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.854374   11932 kubelet.go:806] Node cchub-worker-03 was previously registered\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.900461   11932 manager.go:946] Started watching for new ooms in manager\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.900497   11932 oomparser.go:199] OOM parser using kernel log file: \"/var/log/messages\"\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.901009   11932 manager.go:243] Starting recovery of all containers\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.903832   11932 manager.go:248] Recovery completed\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.905720   11932 status_manager.go:76] Starting to sync pod status with apiserver\r\nJul 20 23:46:38 cchub-worker-03.novalocal kubelet[11932]: I0720 23:46:38.905749   11932 kubelet.go:1725] Starting kubelet main sync loop.\r\n\r\nkube-proxy.service - Kubernetes Kube-Proxy Server\r\n   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:37 UTC; 6min ago\r\n     Docs: https://github.com/GoogleCloudPlatform/kubernetes\r\n Main PID: 11852 (kube-proxy)\r\n   CGroup: /system.slice/kube-proxy.service\r\n           └─11852 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://cchub-control-01:8080\r\n\r\nJul 20 23:46:37 cchub-worker-03.novalocal systemd[1]: Started Kubernetes Kube-Proxy Server.\r\n\r\nflanneld.service - Flanneld overlay address etcd agent\r\n   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled)\r\n   Active: active (running) since Mon 2015-07-20 23:46:35 UTC; 6min ago\r\n  Process: 11580 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)\r\n Main PID: 11574 (flanneld)\r\n   CGroup: /system.slice/flanneld.service\r\n           └─11574 /usr/bin/flanneld -etcd-endpoints=http://cchub-control-01:4001 -etcd-prefix=/coreos.com/network -iface=eth0\r\n\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: Connecting to etcd: attempt 1 for keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=69\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: send.request.to http://cchub-control-01:4001/v2/keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=69 | method GET\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: I0720 23:46:35.381191 11574 udp.go:247] Subnet added: 4.0.82.0/24\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: rawWatch /coreos.com/network/subnets []\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: get /coreos.com/network/subnets [http://cchub-control-01:4001]\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: Connecting to etcd: attempt 1 for keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: send.request.to http://cchub-control-01:4001/v2/keys/coreos.com/network/subnets?recursive=true&wait=true&waitIndex=70 | method GET\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: recv.response.from\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: recv.success\r\nJul 20 23:46:35 cchub-worker-03.novalocal flanneld[11574]: go-etcd2015/07/20 23:46:35 DEBUG: recv.response.from\r\n\r\ndocker.service - Docker Application Container Engine\r\n   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled)\r\n  Drop-In: /usr/lib/systemd/system/docker.service.d\r\n           └─flannel.conf\r\n   Active: active (running) since Mon 2015-07-20 23:46:36 UTC; 6min ago\r\n     Docs: http://docs.docker.com\r\n Main PID: 11694 (docker)\r\n   CGroup: /system.slice/docker.service\r\n           └─11694 /usr/bin/docker -d --selinux-enabled --bip=4.0.26.1/24 --mtu=1472\r\n\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:53:10 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:10Z\" level=info msg=\"-job containers() = OK (0)\"\r\nJul 20 23:53:11 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:11Z\" level=info msg=\"GET /containers/json\"\r\nJul 20 23:53:11 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:11Z\" level=info msg=\"+job containers()\"\r\nJul 20 23:53:11 cchub-worker-03.novalocal docker[11694]: time=\"2015-07-20T23:53:11Z\" level=info msg=\"-job containers() = OK (0)\"\r\n```\r\n\r\nHost IP Addresses will be in the range `10.7.7.*`\r\nFlannel IP Addresses will be in the range: `4.0.0.0/16`\r\nDNS IP Addresses will be in the range: `192.168.0.0/16`\r\n\r\nI've reviewed the docs on DNS many times, the different setup guides, examples, and most every tutorial I can find via google search.\r\n\r\nBut the service name resolution does not work for me. Any input, feedback or additional approaches to validating my setup.\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/123315083","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/11634#issuecomment-123315083","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/11634","id":123315083,"user":{"login":"ArtfulCoder","id":1220660,"avatar_url":"https://avatars.githubusercontent.com/u/1220660?v=3","gravatar_id":"","url":"https://api.github.com/users/ArtfulCoder","html_url":"https://github.com/ArtfulCoder","followers_url":"https://api.github.com/users/ArtfulCoder/followers","following_url":"https://api.github.com/users/ArtfulCoder/following{/other_user}","gists_url":"https://api.github.com/users/ArtfulCoder/gists{/gist_id}","starred_url":"https://api.github.com/users/ArtfulCoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ArtfulCoder/subscriptions","organizations_url":"https://api.github.com/users/ArtfulCoder/orgs","repos_url":"https://api.github.com/users/ArtfulCoder/repos","events_url":"https://api.github.com/users/ArtfulCoder/events{/privacy}","received_events_url":"https://api.github.com/users/ArtfulCoder/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T13:56:22Z","updated_at":"2015-07-21T13:56:22Z","body":"etcd logs prove that kube2sky can talk to api endpoint..\r\ncan you check the resolv.conf file on busybox pod ?\r\nIt almost feels that the nameserver is not set ..."}}