{"action":"created","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/1771","labels_url":"https://api.github.com/repos/rancher/rancher/issues/1771/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/1771/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/1771/events","html_url":"https://github.com/rancher/rancher/issues/1771","id":99936222,"number":1771,"title":"DNS lookup fails for all containers launched with rancher","user":{"login":"bilal","id":60473,"avatar_url":"https://avatars.githubusercontent.com/u/60473?v=3","gravatar_id":"","url":"https://api.github.com/users/bilal","html_url":"https://github.com/bilal","followers_url":"https://api.github.com/users/bilal/followers","following_url":"https://api.github.com/users/bilal/following{/other_user}","gists_url":"https://api.github.com/users/bilal/gists{/gist_id}","starred_url":"https://api.github.com/users/bilal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bilal/subscriptions","organizations_url":"https://api.github.com/users/bilal/orgs","repos_url":"https://api.github.com/users/bilal/repos","events_url":"https://api.github.com/users/bilal/events{/privacy}","received_events_url":"https://api.github.com/users/bilal/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-08-09T22:34:54Z","updated_at":"2015-08-11T07:26:22Z","closed_at":null,"body":"Containers are not able to resolve hosts for linked services and all DNS lookups are failing with Rancher 0.32 and rancher agent instance v0.3.1 (rancher/agent-instance:v0.3.1) and rancher agent v0.7.11 (rancher/agent:v0.7.11). \r\n\r\n*Steps to reproduce:*\r\n\r\n Launch any container with Rancher server 0.32. Execute shell and try pinging an external host (e.g., google.com) or a linked service. The rancher-agent on each host is not affected by this issue and is able to resolve hosts. \r\n\r\n*Other Info:*\r\n\r\nDocker version: 1.6.2\r\nHost provider: AWS EC2\r\n\r\n*Error logs from rancher agent:*\r\n\r\n09/08/2015 03:17:27rancher-dns: no process found\r\n09/08/2015 03:17:27[[scripts.sh:198] docker_env\r\n09/08/2015 03:17:27[[scripts.sh:26] '[' -e /.dockerenv ']'\r\n09/08/2015 03:17:27[[[scripts.sh:27] docker_env_vars\r\n09/08/2015 03:17:27[[[scripts.sh:19] '[' -e /.dockerenv ']'\r\n09/08/2015 03:17:27[[[scripts.sh:20] grep CATTLE_\r\n09/08/2015 03:17:27[[[scripts.sh:20] xargs -0 '-I{}' echo export '\"{}\"'\r\n09/08/2015 03:17:27[[[scripts.sh:20] cat /proc/1/environ\r\n09/08/2015 03:17:27[[scripts.sh:27] eval export '\"CATTLE_CONFIG_URL=http://HOST:PORT/v1\"' export '\"CATTLE_AGENT_INSTANCE_AUTH=Basic' 'BASIC-AUTH\"' export '\"CATTLE_HOME=/var/lib/cattle\"'\r\n09/08/2015 03:17:27[[[scripts.sh:27] export CATTLE_CONFIG_URL=http://HOST:PORT/v1 export 'CATTLE_AGENT_INSTANCE_AUTH=Basic AUTH' export CATTLE_HOME=/var/lib/cattle\r\n09/08/2015 03:17:27[[[scripts.sh:27] CATTLE_CONFIG_URL=http://HOST:PORT/v1\r\n09/08/2015 03:17:27[[[scripts.sh:27] CATTLE_AGENT_INSTANCE_AUTH='Basic AUTH'\r\n09/08/2015 03:17:27[[[scripts.sh:27] CATTLE_HOME=/var/lib/cattle\r\n09/08/2015 03:17:27[[scripts.sh:199] script_env\r\n09/08/2015 03:17:27[[scripts.sh:123] '[' -n http://HOST:PORT/v1 ']'\r\n09/08/2015 03:17:27[[scripts.sh:124] return\r\n09/08/2015 03:17:27[[scripts.sh:201] export CATTLE_HOME=/var/lib/cattle\r\n09/08/2015 03:17:27[[scripts.sh:201] CATTLE_HOME=/var/lib/cattle\r\n09/08/2015 03:17:27[[scripts.sh:202] export CATTLE_CONFIG_URL=http://HOST:PORT/v1\r\n09/08/2015 03:17:27[[scripts.sh:202] CATTLE_CONFIG_URL=http://HOST:PORT/v1\r\n09/08/2015 03:17:27[[scripts.sh:203] export CATTLE_STORAGE_URL=\r\n09/08/2015 03:17:27[[scripts.sh:203] CATTLE_STORAGE_URL=\r\n09/08/2015 03:17:27[apply.sh:21] DNS=\r\n09/08/2015 03:17:27[[apply.sh:23] awk '{print $2}'\r\n09/08/2015 03:17:27[[apply.sh:23] grep '^nameserver'\r\n09/08/2015 03:17:27[[apply.sh:23] sed 's/^# nameserver/nameserver/'\r\n09/08/2015 03:17:27[[apply.sh:23] cat /etc/resolv.conf\r\n09/08/2015 03:17:27[apply.sh:23] for address in '$(cat /etc/resolv.conf  | sed '\\''s/^# nameserver/nameserver/'\\'' | grep ^nameserver | awk '\\''{print $2}'\\'')'\r\n09/08/2015 03:17:27[apply.sh:24] '[' 172.31.0.2 = 169.254.169.250 ']'\r\n09/08/2015 03:17:27[apply.sh:28] '[' -z '' ']'\r\n09/08/2015 03:17:27[apply.sh:29] DNS='\"172.31.0.2\"'\r\n09/08/2015 03:17:27[apply.sh:35] sed -i 's/PARENT_DNS/\"172.31.0.2\"/' content-home/etc/cattle/dns/answers.json\r\n09/08/2015 03:17:27[apply.sh:37] stage_files\r\n09/08/2015 03:17:27[scripts.sh:91] local monit=false\r\n09/08/2015 03:17:27[scripts.sh:93] '[' -d content-home ']'\r\n09/08/2015 03:17:27[scripts.sh:94] find content-home -name '*.sh' -exec chmod +x '{}' ';'\r\n09/08/2015 03:17:27[scripts.sh:95] sed 's/^/INFO: HOME -> /'\r\n09/08/2015 03:17:27[scripts.sh:95] tar xvf - --no-overwrite-dir -C /var/lib/cattle\r\n09/08/2015 03:17:27[scripts.sh:95] tar cf - -C content-home .\r\n09/08/2015 03:17:27[scripts.sh:98] '[' -d content ']'\r\n09/08/2015 03:17:27[scripts.sh:116] '[' false = true ']'\r\n09/08/2015 03:17:27[apply.sh:39] ip addr add 169.254.169.250/32 dev eth0\r\n09/08/2015 03:17:27[apply.sh:39] true\r\n09/08/2015 03:17:27[apply.sh:40] grep -q 169.254.169.254\r\n09/08/2015 03:17:27[apply.sh:40] ip route show\r\n09/08/2015 03:17:27[apply.sh:44] reload_dns\r\n09/08/2015 03:17:27[apply.sh:7] '[' '!' -e /etc/init.d/rancher-dns ']'\r\n09/08/2015 03:17:27[[apply.sh:12] pidof rancher-dns\r\n09/08/2015 03:17:27[apply.sh:12] PID=870\r\n09/08/2015 03:17:27[apply.sh:14] '[' -z 870 ']'\r\n09/08/2015 03:17:27[apply.sh:18] killall -HUP rancher-dns\r\n"},"comment":{"url":"https://api.github.com/repos/rancher/rancher/issues/comments/129738865","html_url":"https://github.com/rancher/rancher/issues/1771#issuecomment-129738865","issue_url":"https://api.github.com/repos/rancher/rancher/issues/1771","id":129738865,"user":{"login":"silviupanaite","id":8484794,"avatar_url":"https://avatars.githubusercontent.com/u/8484794?v=3","gravatar_id":"","url":"https://api.github.com/users/silviupanaite","html_url":"https://github.com/silviupanaite","followers_url":"https://api.github.com/users/silviupanaite/followers","following_url":"https://api.github.com/users/silviupanaite/following{/other_user}","gists_url":"https://api.github.com/users/silviupanaite/gists{/gist_id}","starred_url":"https://api.github.com/users/silviupanaite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/silviupanaite/subscriptions","organizations_url":"https://api.github.com/users/silviupanaite/orgs","repos_url":"https://api.github.com/users/silviupanaite/repos","events_url":"https://api.github.com/users/silviupanaite/events{/privacy}","received_events_url":"https://api.github.com/users/silviupanaite/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T07:26:22Z","updated_at":"2015-08-11T07:26:22Z","body":"Yes I hadn't pulled the right container, also these are container running on the same host, tried a different host, got the same problem.\r\n{\r\n    \"default\": {\r\n        \"recurse\": [\r\n            \"213.186.33.99\"\r\n        ]\r\n    },\r\n    \"10.42.154.9\": {\r\n        \"a\": {\r\n\t\t\"redis.\": {\"answer\": [\"10.42.154.9\"]},\r\n\t\t\"redis.rancher.internal.\": {\"answer\": [\"10.42.154.9\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.129.161\": {\r\n        \"a\": {\r\n\t\t\"redis-outboxpro.\": {\"answer\": [\"10.42.50.159\"]},\r\n\t\t\"redis-outboxpro.rancher.internal.\": {\"answer\": [\"10.42.50.159\"]},\r\n\t\t\"connector-twitter.\": {\"answer\": [\"10.42.129.161\"]},\r\n\t\t\"connector-twitter.rancher.internal.\": {\"answer\": [\"10.42.129.161\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.97.166\": {\r\n        \"a\": {\r\n\t\t\"database.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"database.rancher.internal.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"ruby-daemon.\": {\"answer\": [\"10.42.97.166\"]},\r\n\t\t\"ruby-daemon.rancher.internal.\": {\"answer\": [\"10.42.97.166\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.50.159\": {\r\n        \"a\": {\r\n\t\t\"redis-outboxpro.\": {\"answer\": [\"10.42.50.159\"]},\r\n\t\t\"redis-outboxpro.rancher.internal.\": {\"answer\": [\"10.42.50.159\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.35.122\": {\r\n        \"a\": {\r\n\t\t\"mongodb.\": {\"answer\": [\"10.42.171.76\"]},\r\n\t\t\"mongodb.rancher.internal.\": {\"answer\": [\"10.42.171.76\"]},\r\n\t\t\"daemon.\": {\"answer\": [\"10.42.97.166\"]},\r\n\t\t\"daemon.rancher.internal.\": {\"answer\": [\"10.42.97.166\"]},\r\n\t\t\"apache-php.\": {\"answer\": [\"10.42.35.122\"]},\r\n\t\t\"apache-php.rancher.internal.\": {\"answer\": [\"10.42.35.122\"]},\r\n\t\t\"redis.\": {\"answer\": [\"10.42.154.9\"]},\r\n\t\t\"redis.rancher.internal.\": {\"answer\": [\"10.42.154.9\"]},\r\n\t\t\"database.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"database.rancher.internal.\": {\"answer\": [\"10.42.179.148\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.171.76\": {\r\n        \"a\": {\r\n\t\t\"database.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"database.rancher.internal.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"mongodb.\": {\"answer\": [\"10.42.171.76\"]},\r\n\t\t\"mongodb.rancher.internal.\": {\"answer\": [\"10.42.171.76\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.179.148\": {\r\n        \"a\": {\r\n\t\t\"mysql.\": {\"answer\": [\"10.42.179.148\"]},\r\n\t\t\"mysql.rancher.internal.\": {\"answer\": [\"10.42.179.148\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"10.42.140.2\": {\r\n        \"a\": {\r\n\t\t\"unifiedinbox.\": {\"answer\": [\"10.42.35.122\"]},\r\n\t\t\"unifiedinbox.rancher.internal.\": {\"answer\": [\"10.42.35.122\"]}\r\n    \t}\r\n\r\n    },\r\n\r\n    \"\": {}\r\n}"}}