{"action":"created","issue":{"url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/28","labels_url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/28/comments","events_url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/28/events","html_url":"https://github.com/projectcalico/calico-kubernetes/issues/28","id":103688316,"number":28,"title":"Containers are not able to ping each others : \"No JSON object could be decoded\"","user":{"login":"Smana","id":4101366,"avatar_url":"https://avatars.githubusercontent.com/u/4101366?v=3","gravatar_id":"","url":"https://api.github.com/users/Smana","html_url":"https://github.com/Smana","followers_url":"https://api.github.com/users/Smana/followers","following_url":"https://api.github.com/users/Smana/following{/other_user}","gists_url":"https://api.github.com/users/Smana/gists{/gist_id}","starred_url":"https://api.github.com/users/Smana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Smana/subscriptions","organizations_url":"https://api.github.com/users/Smana/orgs","repos_url":"https://api.github.com/users/Smana/repos","events_url":"https://api.github.com/users/Smana/events{/privacy}","received_events_url":"https://api.github.com/users/Smana/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-08-28T09:53:12Z","updated_at":"2015-08-28T15:46:19Z","closed_at":null,"body":"Hello,\r\n\r\nI think my kubelet service is properly configured now.\r\nWhen i run a pod, the container gets its address from the pool\r\n```\r\nip a show cbr0\r\n5: cbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default \r\n    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.233.1.1/24 scope global cbr0\r\n       valid_lft forever preferred_lft forever\r\n    inet6 fe80::507b:20ff:fe97:c0cf/64 scope link \r\n       valid_lft forever preferred_lft forever\r\n\r\nps aux | grep /usr/bin/docker\r\n/usr/bin/docker -d -H fd:// --bridge=cbr0 --iptables=false --ip-masq=false\r\n\r\ndocker exec k8s_worker.41ff1f2e_redis-slave-nruvk_default_08421053-4d68-11e5-af10-525400c14d0a_77d68d0f ip a show eth0\r\n18: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\r\n    link/ether 52:1b:00:d7:e0:27 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.233.1.5/32 scope global eth0\r\n       valid_lft forever preferred_lft forever\r\n    inet6 fe80::501b:ff:fed7:e027/64 scope link \r\n       valid_lft forever preferred_lft forever\r\n```\r\n\r\nHowever if i run a container with docker i can't ping the address of the container spawned by kubernetes.\r\n```\r\ndocker run --rm -ti base/archlinux /bin/bash\r\n[root@87d9b164ffe9 /]# ip a show eth0\r\n24: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default \r\n    link/ether 02:42:0a:e9:01:08 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.233.1.8/24 scope global eth0\r\n       valid_lft forever preferred_lft forever\r\n    inet6 fe80::42:aff:fee9:108/64 scope link \r\n       valid_lft forever preferred_lft forever\r\n[root@87d9b164ffe9 /]# ping 10.233.1.5\r\nPING 10.233.1.5 (10.233.1.5) 56(84) bytes of data.\r\nFrom 10.233.1.8 icmp_seq=1 Destination Host Unreachable\r\n```\r\nNote : This container (run with docker) can ping another container launched the same way on another node.\r\n\r\nThe difference i can see comes from how the interfaces are created on the host, the kubernetes interfaces are not attached to the bridge cbr0.\r\n```\r\n21: cali09089e184d6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000\r\n    link/ether c6:28:14:7c:36:cd brd ff:ff:ff:ff:ff:ff\r\n    inet 10.115.77.87/32 scope global cali09089e184d6\r\n       valid_lft forever preferred_lft forever\r\n    inet6 fe80::c428:14ff:fe7c:36cd/64 scope link \r\n       valid_lft forever preferred_lft forever\r\n25: veth4582dc5: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cbr0 state UP group default \r\n    link/ether 16:ca:4c:7e:e0:f0 brd ff:ff:ff:ff:ff:ff\r\n    inet6 fe80::14ca:4cff:fe7e:e0f0/64 scope link \r\n       valid_lft forever preferred_lft forever\r\n```\r\n\r\nWhen i look to the kubelet logs i get the following error log : \r\n```\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: I0828 09:34:54.047335   29139 exec.go:130] SetUpPod 'exec' network plugin output: Warning: '-format' is deprecated, it will be replaced by '--format' soon. See usage.\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Warning: '-format' is deprecated, it will be replaced by '--format' soon. See usage.\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: /tmp/_MEIc3W7lo/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: /tmp/_MEIc3W7lo/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Using ETCD_AUTHORITY=10.115.77.86:4001\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Using CALICOCTL_PATH=/usr/local/bin/calicoctl\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Using KUBE_API_ROOT=https://10.115.77.86:443/api/v1/\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Args: ['/usr/libexec/kubernetes/kubelet-plugins/net/exec/calico/calico', 'setup', 'default', 'redis-slave-nruvk', 'cc864fdf31cb3c14f44a14e13c144c41c7a2f06c053e4ab5942320cbc7a008e2']\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Executing Calico pod-creation hook\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Configuring docker container cc864fdf31cb3c14f44a14e13c144c41c7a2f06c053e4ab5942320cbc7a008e2\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Docker-assigned IP was 10.233.1.5\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Deleting eth0\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Configuring Calico networking.\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: IP 10.233.1.5 added to cc864fdf31cb3c14f44a14e13c144c41c7a2f06c053e4ab5942320cbc7a008e2\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Adding IP 10.115.77.87 to interface cali09001bb24d6\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Finished configuring network interface\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Failed to open auth_file ([Errno 2] No such file or directory: '/var/lib/kubelet/kubernetes_auth'), assuming insecure mode\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Traceback (most recent call last):\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"<string>\", line 341, in <module>\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"<string>\", line 46, in create\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"<string>\", line 158, in _configure_profile\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"<string>\", line 189, in _get_pod_config\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"<string>\", line 220, in _get_api_path\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"/code/build/calico_kubernetes/out00-PYZ.pyz/json\", line 338, in loads\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"/code/build/calico_kubernetes/out00-PYZ.pyz/json.decoder\", line 366, in decode\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: File \"/code/build/calico_kubernetes/out00-PYZ.pyz/json.decoder\", line 384, in raw_decode\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: ValueError: No JSON object could be decoded\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: , exit status 255\r\n```\r\n\r\nThanks for your help :)\r\nSmana"},"comment":{"url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/comments/135811217","html_url":"https://github.com/projectcalico/calico-kubernetes/issues/28#issuecomment-135811217","issue_url":"https://api.github.com/repos/projectcalico/calico-kubernetes/issues/28","id":135811217,"user":{"login":"caseydavenport","id":1944093,"avatar_url":"https://avatars.githubusercontent.com/u/1944093?v=3","gravatar_id":"","url":"https://api.github.com/users/caseydavenport","html_url":"https://github.com/caseydavenport","followers_url":"https://api.github.com/users/caseydavenport/followers","following_url":"https://api.github.com/users/caseydavenport/following{/other_user}","gists_url":"https://api.github.com/users/caseydavenport/gists{/gist_id}","starred_url":"https://api.github.com/users/caseydavenport/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caseydavenport/subscriptions","organizations_url":"https://api.github.com/users/caseydavenport/orgs","repos_url":"https://api.github.com/users/caseydavenport/repos","events_url":"https://api.github.com/users/caseydavenport/events{/privacy}","received_events_url":"https://api.github.com/users/caseydavenport/received_events","type":"User","site_admin":false},"created_at":"2015-08-28T15:46:19Z","updated_at":"2015-08-28T15:46:19Z","body":"Hi Smana,\r\n\r\nIt's expected that Kubernetes pods which are networked using Calico should be attached to the caliXXXX interfaces (not the docker bridge) so I don't think that is your issue.\r\n\r\nLooking at your Kubelet logs, it looks like you're configured to use HTTPS for accessing the Kubernetes API on its secure port (443), as indicated by this log:\r\n```\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Using KUBE_API_ROOT=https://10.115.77.86:443/api/v1/\r\n```\r\n\r\nHowever, it doesn't look like you've configured auth on the Kubernetes node:\r\n```\r\nAug 28 09:34:54 kubenode-1 kubelet[29139]: Failed to open auth_file ([Errno 2] No such file or directory: '/var/lib/kubelet/kubernetes_auth'), assuming insecure mode\r\n```\r\n\r\nI suspect this is causing the API request to fail, and Calico is attempting to interpret it as JSON (which fails).\r\n\r\nAt the moment, there are two ways to get this working (though #14 covers supporting more types of authentication):\r\n- If your cluster supports it, switch to using the insecure port on the API. e.g `KUBE_API_ROOT=http://10.155.77.86:8080/api/v1`\r\n- Create a token file at `/var/lib/kubelet/kubernetes_auth` which includes a BearerToken registered with the apiserver, which looks something like this:\r\n```\r\n{\r\n  \"BearerToken\": \"YOUR_TOKEN_HERE\"\r\n}\r\n```\r\n\r\nThe plugin should probably be smarter about logging out when authentication types don't match - I've raised #29 to track improving diagnosability."}}