{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/17689","labels_url":"https://api.github.com/repos/docker/docker/issues/17689/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/17689/comments","events_url":"https://api.github.com/repos/docker/docker/issues/17689/events","html_url":"https://github.com/docker/docker/issues/17689","id":115078124,"number":17689,"title":"Ubuntu 14.04, docker 1.9.0 + dnsmasq result in hostname lookup failure in containers","user":{"login":"UndeadBaneGitHub","id":3450198,"avatar_url":"https://avatars.githubusercontent.com/u/3450198?v=3","gravatar_id":"","url":"https://api.github.com/users/UndeadBaneGitHub","html_url":"https://github.com/UndeadBaneGitHub","followers_url":"https://api.github.com/users/UndeadBaneGitHub/followers","following_url":"https://api.github.com/users/UndeadBaneGitHub/following{/other_user}","gists_url":"https://api.github.com/users/UndeadBaneGitHub/gists{/gist_id}","starred_url":"https://api.github.com/users/UndeadBaneGitHub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/UndeadBaneGitHub/subscriptions","organizations_url":"https://api.github.com/users/UndeadBaneGitHub/orgs","repos_url":"https://api.github.com/users/UndeadBaneGitHub/repos","events_url":"https://api.github.com/users/UndeadBaneGitHub/events{/privacy}","received_events_url":"https://api.github.com/users/UndeadBaneGitHub/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-11-04T15:27:59Z","updated_at":"2015-11-12T13:58:31Z","closed_at":null,"body":"Formal info:\r\n\r\n`docker version`:\r\n    \r\n    Client:\r\n     Version:      1.9.0\r\n     API version:  1.21\r\n     Go version:   go1.4.2\r\n     Git commit:   76d6bc9\r\n     Built:        Tue Nov  3 17:43:42 UTC 2015\r\n     OS/Arch:      linux/amd64\r\n    \r\n    Server:\r\n     Version:      1.9.0\r\n     API version:  1.21\r\n     Go version:   go1.4.2\r\n     Git commit:   76d6bc9\r\n     Built:        Tue Nov  3 17:43:42 UTC 2015\r\n     OS/Arch:      linux/amd64\r\n\r\n`docker info` (however, the bug was initially reproduced on a much larger AWS `m4.2xlarge` instance, this `t2.micro` was used later for isolating the bug):\r\n     \r\n    Containers: 0\r\n    Images: 0\r\n    Server Version: 1.9.0\r\n    Storage Driver: devicemapper\r\n     Pool Name: docker-202:1-398179-pool\r\n     Pool Blocksize: 65.54 kB\r\n     Base Device Size: 107.4 GB\r\n     Backing Filesystem: extfs\r\n     Data file: /dev/loop0\r\n     Metadata file: /dev/loop1\r\n     Data Space Used: 1.821 GB\r\n     Data Space Total: 107.4 GB\r\n     Data Space Available: 28.81 GB\r\n     Metadata Space Used: 1.479 MB\r\n     Metadata Space Total: 2.147 GB\r\n     Metadata Space Available: 2.146 GB\r\n     Udev Sync Supported: true\r\n     Deferred Removal Enabled: false\r\n     Deferred Deletion Enabled: false\r\n     Deferred Deleted Device Count: 0\r\n     Data loop file: /var/lib/docker/devicemapper/devicemapper/data\r\n     Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n     Library Version: 1.02.77 (2012-10-15)\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.13.0-48-generic\r\n    Operating System: Ubuntu 14.04.2 LTS\r\n    CPUs: 1\r\n    Total Memory: 992.5 MiB\r\n    \r\n`uname -a`:\r\n    \r\n    Linux ip-172-31-32-210 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n-------------------------------------------------------\r\n\r\nTested on Amazon instances (`t2.micro` with `Ubuntu 14.04` free instance is enough), shell script for reproducing the bug is attached. The bug is always there, shows up in 100% of cases.\r\n\r\nThe test script is written so that it can be ran on a fresh AWS instance and give the result (the crash), no additional preparation is required - just create a shell file with this script and run it.\r\n\r\nThe bug was caught on mesosphere docker images, so they are referenced in the script, but I suppose, something more lightweight, that also uses hostname lookup (like `php`) can be used.\r\n\r\nNothing like this happened on docker `1.8.3` - the images are started and run without any issues, `dnsmasq` or not.\r\n\r\nThe script can be easily edited to install `1.8.3` docker or not install `dnsmasq` - then the bug goes away. But with `dnsmasq` and the latest version stuff just does not work.\r\n\r\nThe script:\r\n    \r\n    #!/bin/bash\r\n    HOST_IP=$(hostname --ip-address)\r\n    DOCKER_OPTS='DOCKER_OPTS=\"--log-driver=json-file --log-opt max-size=20m --log-opt max-file=5\"'\r\n    \r\n    apt-get update && apt-get install -y libdevmapper-dev libpthread-stubs0-dev libsqlite3-dev libudev-dev\r\n    \r\n    # Without DNSMASQ everything seems to work\r\n    apt-get install -y dnsmasq\r\n    # Sleep is required to let dnsmasq start properly\r\n    sleep 1m\r\n    \r\n    apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D\r\n    \r\n    rm -f /etc/apt/sources.list.d/docker.list\r\n    \r\n    echo \"deb https://apt.dockerproject.org/repo ubuntu-trusty main\" >> /etc/apt/sources.list.d/docker.list\r\n    \r\n    apt-get update && apt-get purge -y lxc-docker*\r\n    \r\n    # Everything works fine with docker 1.8.3\r\n    # apt-get install -y docker-engine=1.8.3-0~trusty\r\n    # ...but fails with 1.9.0\r\n    apt-get install -y docker-engine=1.9.0-0~trusty\r\n    \r\n    GREPPED_DOCKER_OPTS=$(grep \"$DOCKER_OPTS\" /etc/default/docker)\r\n    if [[ $? -eq 1 ]]\r\n    then\r\n    \tsed -e \"\\$a$DOCKER_OPTS\" /etc/default/docker\r\n    \tservice docker restart\r\n    fi\r\n    \r\n    # Start Zookeeper\r\n    docker run -d \\\r\n    \t--name=zookeeper \\\r\n    \t--net=host \\\r\n    \t--name=zookeeper \\\r\n    \tjplock/zookeeper\r\n    \r\n    # Start mesos master\r\n    docker run -d \\\r\n    \t--name=mesos_master \\\r\n    \t--net=host \\\r\n    \tmesosphere/mesos-master:0.25.0-0.2.70.ubuntu1404 \\\r\n    \t\t--ip=$HOST_IP \\\r\n    \t\t--zk=zk://$HOST_IP:2181/mesos \\\r\n    \t\t--work_dir=/var/lib/mesos/master \\\r\n    \t\t--quorum=1\r\n    \r\n    # Start mesos slave\r\n    docker run -d \\\r\n    \t--name=mesos_slave \\\r\n    \t--privileged \\\r\n    \t--net=host \\\r\n    \t-v /sys:/sys \\\r\n    \t-v /usr/bin/docker:/usr/bin/docker:ro \\\r\n    \t-v /var/run/docker.sock:/var/run/docker.sock \\\r\n    \t-v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \\\r\n    \t-v /lib64/libpthread.so.0:/lib/libpthread.so.0:ro \\\r\n    \t-v /lib64/libsqlite3.so.0:/lib/libsqlite3.so.0:ro \\\r\n    \t-v /lib64/libudev.so.1:/lib/libudev.so.1:ro \\\r\n    \t-v /usr/lib/x86_64-linux-gnu/libapparmor.so.1:/usr/lib/x86_64-linux-gnu/libapparmor.so.1:ro \\\r\n    \t-v /opt/intempt/docker/docker_cfg.tar.gz:/etc/docker_cfg.tar.gz:ro \\\r\n    \tmesosphere/mesos-slave:0.25.0-0.2.70.ubuntu1404 \\\r\n    \t\t--ip=$HOST_IP \\\r\n    \t\t--containerizers=docker \\\r\n    \t\t--master=zk://$HOST_IP:2181/mesos \\\r\n    \t\t--work_dir=/var/lib/mesos/slave \\\r\n    \t\t--log_dir=/var/log/mesos/slave \\\r\n    \t\t--resources='mem:1024;cpus:1;ports:[2000-40000]'\r\n    \r\n    \r\n    # Watch the slave crash within 30s or so with the following error:\r\n    \r\n    # F1104 14:53:12.548439     1 slave.cpp:369] Failed to get hostname: Temporary failure in name resolution\r\n    # *** Check failure stack trace: ***\r\n    #     @     0x7f6b5eaad64d  google::LogMessage::Fail()\r\n    #     @     0x7f6b5eaaf48d  google::LogMessage::SendToLog()\r\n    #     @     0x7f6b5eaad23c  google::LogMessage::Flush()\r\n    #     @     0x7f6b5eaafd89  google::LogMessageFatal::~LogMessageFatal()\r\n    #     @     0x7f6b5e52cdc9  mesos::internal::slave::Slave::initialize()\r\n    #     @     0x7f6b5ea5ebb1  process::ProcessManager::resume()\r\n    #     @     0x7f6b5ea5f29a  process::ProcessManager::wait()\r\n    #     @     0x7f6b5ea5fa17  process::wait()\r\n    #     @           0x409ab4  main\r\n    #     @     0x7f6b5cf61ec5  (unknown)\r\n    #     @           0x40a1f8  (unknown)\r\n    # *** Aborted at 1446648792 (unix time) try \"date -d @1446648792\" if you are using GNU date ***\r\n    # PC: @     0x7f6b5cf7a227 (unknown)\r\n    # *** SIGSEGV (@0x0) received by PID 1 (TID 0x7f6b5f47e800) from PID 0; stack trace: ***\r\n    #     @     0x7f6b5d315340 (unknown)\r\n    #     @     0x7f6b5cf7a227 (unknown)\r\n    #     @     0x7f6b5eab5909 google::DumpStackTraceAndExit()\r\n    #     @     0x7f6b5eaad64d google::LogMessage::Fail()\r\n    #     @     0x7f6b5eaaf48d google::LogMessage::SendToLog()\r\n    #     @     0x7f6b5eaad23c google::LogMessage::Flush()\r\n    #     @     0x7f6b5eaafd89 google::LogMessageFatal::~LogMessageFatal()\r\n    #     @     0x7f6b5e52cdc9 mesos::internal::slave::Slave::initialize()\r\n    #     @     0x7f6b5ea5ebb1 process::ProcessManager::resume()\r\n    #     @     0x7f6b5ea5f29a process::ProcessManager::wait()\r\n    #     @     0x7f6b5ea5fa17 process::wait()\r\n    #     @           0x409ab4 main\r\n    #     @     0x7f6b5cf61ec5 (unknown)\r\n    #     @           0x40a1f8 (unknown)\r\n    \r\n    # The master crashes too - zookeeper refuses connections, I suppose, because of the same bug. No crashes happen, if 1.8.3 docker is installed, or there is no dnsmasq"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/156108671","html_url":"https://github.com/docker/docker/issues/17689#issuecomment-156108671","issue_url":"https://api.github.com/repos/docker/docker/issues/17689","id":156108671,"user":{"login":"UndeadBaneGitHub","id":3450198,"avatar_url":"https://avatars.githubusercontent.com/u/3450198?v=3","gravatar_id":"","url":"https://api.github.com/users/UndeadBaneGitHub","html_url":"https://github.com/UndeadBaneGitHub","followers_url":"https://api.github.com/users/UndeadBaneGitHub/followers","following_url":"https://api.github.com/users/UndeadBaneGitHub/following{/other_user}","gists_url":"https://api.github.com/users/UndeadBaneGitHub/gists{/gist_id}","starred_url":"https://api.github.com/users/UndeadBaneGitHub/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/UndeadBaneGitHub/subscriptions","organizations_url":"https://api.github.com/users/UndeadBaneGitHub/orgs","repos_url":"https://api.github.com/users/UndeadBaneGitHub/repos","events_url":"https://api.github.com/users/UndeadBaneGitHub/events{/privacy}","received_events_url":"https://api.github.com/users/UndeadBaneGitHub/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T13:58:31Z","updated_at":"2015-11-12T13:58:31Z","body":"@bpradipt This is really weird, as I've just now created a new t2.micro + Ubuntu 14.04 instance (`Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-5189a661`), copy-pasted the script into `/tmp/test.sh`, chmoded it, ran it - and now I get the same error exactly: both master and slave have crashed with errors, leaving only zookeeper running:\r\n    \r\n    docker ps --format \"{{.ID}}\\t{{.RunningFor}}\\t{{.Status}}\\t{{.Names}}\"\r\n    662ad270a2bf\t3 minutes\tUp 3 minutes\tzookeeper\r\n\r\nI wonder, what can be different. Are you sure, that you have not apply any changes to the script?\r\n\r\nI will also try to create AWS instance under a different account and test it there as well."}}