{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6502","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6502/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6502/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6502/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6502","id":66763429,"number":6502,"title":"MultiNode Cluster Internal network (Flannel) Not Working","user":{"login":"sacashgit","id":6346188,"avatar_url":"https://avatars.githubusercontent.com/u/6346188?v=3","gravatar_id":"","url":"https://api.github.com/users/sacashgit","html_url":"https://github.com/sacashgit","followers_url":"https://api.github.com/users/sacashgit/followers","following_url":"https://api.github.com/users/sacashgit/following{/other_user}","gists_url":"https://api.github.com/users/sacashgit/gists{/gist_id}","starred_url":"https://api.github.com/users/sacashgit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sacashgit/subscriptions","organizations_url":"https://api.github.com/users/sacashgit/orgs","repos_url":"https://api.github.com/users/sacashgit/repos","events_url":"https://api.github.com/users/sacashgit/events{/privacy}","received_events_url":"https://api.github.com/users/sacashgit/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/support","name":"kind/support","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/support","name":"priority/support","color":"c7def8"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-04-07T03:28:11Z","updated_at":"2015-04-09T18:47:26Z","closed_at":null,"body":"Hi \r\nI have created Kubernetes cluster with 1 master and 3 minions (one VM per Node) set up. Following the step by step guide, I could complete the set up, get rid of few issues along the way and apparently every thing seems to be working over the surface which I mean - Kubectl get pods,services,rc do return every thing as expected. However I do have few problems under the hood : \r\n\r\n1. Redis master-slave sync is not working. Slave keeps looking for master:6379 and never finds it.\r\n2. The front end service is not accessible by any means ( I tried localhost:8000 or publicIPs variable by populating with minions public IP addresses). \r\n\r\nbased on the above symptoms, I came to conclusion that while the nodes (In this my VMs)  are communicating with each other very well but internal communication between containers isnt working at all.  How do I go about debugging the problem #1 and #2 listed above ? \r\n\r\nHere are some screen caps to show you what is working and what is not.\r\nVersion Info: \r\nClient Version: version.Info{Major:\"0\", Minor:\"12\", GitVersion:\"v0.12.0\", GitCommit:\"ecca42643b91a7117de8cd385b64e6bafecefd65\", GitTreeState:\"clean\"}\r\nServer Version: version.Info{Major:\"0\", Minor:\"12\", GitVersion:\"v0.12.0\", GitCommit:\"ecca42643b91a7117de8cd385b64e6bafecefd65\", GitTreeState:\"clean\"}\r\n\r\nCommand : kubectl get minions,pods,services,rc\r\n![screen shot 2015-04-06 at 8 18 18 pm](https://cloud.githubusercontent.com/assets/6346188/7016490/366d680a-dc9a-11e4-8380-0c756e9c596a.png)\r\n\r\nSlave Redis Container on Minion (docker logs <slave_container_id>\r\n[8] 07 Apr 03:23:36.114 # Unable to connect to MASTER: Connection timed out\r\n[8] 07 Apr 03:23:37.116 * Connecting to MASTER redis-master:6379\r\n\r\nKube-Proxy.Log shows errors \r\n\r\nE0408 22:59:25.779812    1895 api.go:107] Unable to load services: Get http://MASTER_IP:8080/api/v1beta1/services?namespace=: dial tcp: lookup MASTER_IP: Temporary failure in name resolution\r\nE0408 22:59:28.223032    1895 api.go:179] Unable to load endpoints: Get http://MASTER_IP:8080/api/v1beta1/endpoints?namespace=: dial tcp: lookup MASTER_IP: Temporary failure in name resolution\r\nE0408 23:00:53.253518    1895 api.go:107] Unable to load services: Get http://MASTER_IP:8080/api/v1beta1/services?namespace=: dial tcp: lookup MASTER_IP: Temporary failure in name resolution\r\nE0408 23:00:54.769872    1895 api.go:179] Unable to load endpoints: Get http://MASTER_IP:8080/api/v1beta1/endpoints?namespace=: dial tcp: lookup MASTER_IP: Temporary failure in name resolution\r\n\r\nMaster Redis Looks good (docker logs <master_container_id>)\r\n\r\n[1] 06 Apr 22:05:43.147 # Server started, Redis version 2.8.19\r\n[1] 06 Apr 22:05:43.147 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.\r\n[1] 06 Apr 22:05:43.148 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.\r\n[1] 06 Apr 22:05:43.148 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.\r\n[1] 06 Apr 22:05:43.148 * DB loaded from disk: 0.000 seconds\r\n[1] 06 Apr 22:05:43.148 * The server is now ready to accept connections on port 6379\r\n\r\nFrontend container looks good too.. \r\n\r\n015-04-06 05:06:34,611 CRIT Supervisor running as root (no user in config file)\r\n2015-04-06 05:06:34,611 WARN Included extra file \"/etc/supervisor/conf.d/supervisord-apache2.conf\" during parsing\r\n2015-04-06 05:06:34,636 INFO RPC interface 'supervisor' initialized\r\n2015-04-06 05:06:34,636 WARN cElementTree not installed, using slower XML parser for XML-RPC\r\n2015-04-06 05:06:34,637 CRIT Server 'unix_http_server' running without any HTTP authentication checking\r\n2015-04-06 05:06:34,637 INFO supervisord started with pid 7\r\n2015-04-06 05:06:35,639 INFO spawned: 'apache2' with pid 8\r\n2015-04-06 05:06:36,665 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)\r\n\r\nHow do I debug this to make sure, I have front end service accessible and master/slave synch up working ?\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/91324811","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6502#issuecomment-91324811","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6502","id":91324811,"user":{"login":"sacashgit","id":6346188,"avatar_url":"https://avatars.githubusercontent.com/u/6346188?v=3","gravatar_id":"","url":"https://api.github.com/users/sacashgit","html_url":"https://github.com/sacashgit","followers_url":"https://api.github.com/users/sacashgit/followers","following_url":"https://api.github.com/users/sacashgit/following{/other_user}","gists_url":"https://api.github.com/users/sacashgit/gists{/gist_id}","starred_url":"https://api.github.com/users/sacashgit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sacashgit/subscriptions","organizations_url":"https://api.github.com/users/sacashgit/orgs","repos_url":"https://api.github.com/users/sacashgit/repos","events_url":"https://api.github.com/users/sacashgit/events{/privacy}","received_events_url":"https://api.github.com/users/sacashgit/received_events","type":"User","site_admin":false},"created_at":"2015-04-09T18:47:26Z","updated_at":"2015-04-09T18:47:26Z","body":"Update : I managed to get the application accessible via Public address. The main issue after looking at every thing - looks like, containers are not able to 'resolve' service names for sure.  Redis-slave is not able to find redis-master. The PHP application is not able to find redis-master.  So it is the service discovery or service name resolution is the MAIN issue. I do not have specific DNS service running in this cluster.  What do you think right or round about way of making this Guestbook example work ? I will then rename the issue appropriately. \r\n\r\nSome Logs that suggest the service name resolution is not working \r\n\r\ndocker logs --tail=50 8cb7fa6afc5a [SLAVE log]\r\n[8] 09 Apr 18:39:54.460 * Connecting to MASTER redis-master:6379\r\n[8] 09 Apr 18:39:54.464 # Unable to connect to MASTER: Connection timed out\r\n[8] 09 Apr 18:39:55.467 * Connecting to MASTER redis-master:6379\r\n[8] 09 Apr 18:39:55.473 # Unable to connect to MASTER: Connection timed out\r\n[8] 09 Apr 18:39:56.476 * Connecting to MASTER redis-master:6379\r\n[8] 09 Apr 18:39:56.517 # Unable to connect to MASTER: Connection timed out\r\n\r\nPHP App Error - redis-slave:6379\r\n<b>Fatal error</b>:  Uncaught exception 'Predis\\Connection\\ConnectionException' with message 'php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141\r\nStack trace:"}}