{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/17485","labels_url":"https://api.github.com/repos/docker/docker/issues/17485/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/17485/comments","events_url":"https://api.github.com/repos/docker/docker/issues/17485/events","html_url":"https://github.com/docker/docker/issues/17485","id":114050430,"number":17485,"title":"Starting docker service before remote filesystems are mounted breaks container bind-mounts","user":{"login":"jperville","id":834791,"avatar_url":"https://avatars.githubusercontent.com/u/834791?v=3","gravatar_id":"","url":"https://api.github.com/users/jperville","html_url":"https://github.com/jperville","followers_url":"https://api.github.com/users/jperville/followers","following_url":"https://api.github.com/users/jperville/following{/other_user}","gists_url":"https://api.github.com/users/jperville/gists{/gist_id}","starred_url":"https://api.github.com/users/jperville/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jperville/subscriptions","organizations_url":"https://api.github.com/users/jperville/orgs","repos_url":"https://api.github.com/users/jperville/repos","events_url":"https://api.github.com/users/jperville/events{/privacy}","received_events_url":"https://api.github.com/users/jperville/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-10-29T13:35:53Z","updated_at":"2015-11-05T14:07:03Z","closed_at":null,"body":"# Description of problem\r\n\r\nThe docker service starts before remote filesystems are mounted on the host. Upon restart of the host, containers may start with broken volumes and the data from the remote filesystem won't be available from within those containers.\r\n\r\nThe fix: modify the `/etc/init/docker.conf` file to `start on (filesystem and net-device-up IFACE!=lo)` instead of `start on (local-filesystems and net-device-up IFACE!=lo)`.\r\n\r\n# System information\r\n\r\n`docker version`: `Docker version 1.8.3, build f4bf5c7`\r\n`docker info`:\r\n\r\n```\r\nContainers: 1\r\nImages: 2178\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: xfs\r\n Dirs: 2180\r\n Dirperm1 Supported: true\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.19.0-31-generic\r\nOperating System: Ubuntu 14.04.3 LTS\r\nCPUs: 8\r\nTotal Memory: 15.58 GiB\r\nName: jp440p\r\nID: CB5Z:P7RF:RIET:BEGL:XC55:566P:2VWO:GL6X:GU75:5ZZN:66WM:SOAH\r\nHttp Proxy: http://10.0.3.84:3128\r\nHttps Proxy: http://10.0.3.84:3128\r\nNo Proxy: localhost,127.0.0.1,172.17.42.1,10.0.3.1,192.168.56.1,192.168.33.1,192.168.122.1,192.168.33.1,192.168.33.10.xip.io,10.0.2.15,/var/run/docker.sock\r\nWARNING: No swap limit support\r\n```\r\n\r\n`uname -a`: `Linux jp440p 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux`\r\n\r\n\r\nEnvironment details (AWS, VirtualBox, physical, etc.):\r\n - ubuntu 14.04 host\r\n - /etc/fstab file referencing remote filesystems (in my case, cephfs mounted at /ceph)\r\n - docker containers declared with `docker run ... --restart=always ... -v /ceph:/ceph`\r\n\r\n# Reproducing\r\n\r\nHow reproducible: always, as long as the remote filesystem is slow enough to mount. We use the Fuse CephFS driver which take a couple seconds to start even host-only network; I'm unsure how reproducible the problem is using NFS which is usually pretty fast to mount.\r\n\r\n\r\nSteps to Reproduce:\r\n1. add a /ceph mount in /etc/fstab (ours looks like `conf=//etc/ceph/ceph.conf,id=cephfs.staging4-compute,keyfile=/etc/ceph/ceph.client.cephfs.staging4-compute.secret /ceph fuse.ceph defaults,_netdev 0 0`)\r\n2. mount /ceph and touch some files there to make it non empty\r\n3. declare some containers which are restarted on boot and use the /ceph mount point: `docker run -d --name=bindmountbug --restart=always -v /ceph:/ceph busybox:ubuntu-14.04 nc -l 1337 -l`\r\n4. verify with `docker exec bindmountbug ls -la /ceph` that the expected files are present\r\n5. reboot the host\r\n6. run `docker exec bindmountbug ls -la /ceph` and compare with results from step 3.\r\n\r\n\r\nActual Results:\r\n - The files created at step 3. are not listed in the output of the `ls -la /ceph` from step 6.\r\n\r\nExpected Results:\r\n - The files created at step 3. are listed in the output of the `ls -la /ceph` from step 6.\r\n\r\n\r\n# Additional info\r\n\r\n\r\nThe bug is explained by the docker service starting on the `local-filesystems` event (on ubuntu 14.04), which is an event which triggers before the remote file systems are mounted. The docker service should be started on the `filesystem` event instead (the filesystem event triggers after all filesystems, both local and remote, are mounted; see `man 7 filesystem`).\r\n\r\nI fixed the issue manually by editing the `/etc/init/upstart.conf` file like this:\r\n\r\n```diff\r\ndiff --git a/init/docker.conf b/init/docker.conf\r\nindex ec50b35..b403408 100644\r\n--- a/init/docker.conf\r\n+++ b/init/docker.conf\r\n@@ -1,6 +1,6 @@\r\n description \"Docker daemon\"\r\n \r\n-start on (local-filesystems and net-device-up IFACE!=lo)\r\n+start on (filesystem and net-device-up IFACE!=lo)\r\n stop on runlevel [!2345]\r\n limit nofile 524288 1048576\r\n limit nproc 524288 1048576\r\n```\r\n\r\nSee also:\r\n - https://github.com/docker/docker/pull/5840 , in which the docker service was modified to start on local-filesystem instead of filesystem (which introduced this issue in the first place)"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/154066509","html_url":"https://github.com/docker/docker/issues/17485#issuecomment-154066509","issue_url":"https://api.github.com/repos/docker/docker/issues/17485","id":154066509,"user":{"login":"cpuguy83","id":799078,"avatar_url":"https://avatars.githubusercontent.com/u/799078?v=3","gravatar_id":"","url":"https://api.github.com/users/cpuguy83","html_url":"https://github.com/cpuguy83","followers_url":"https://api.github.com/users/cpuguy83/followers","following_url":"https://api.github.com/users/cpuguy83/following{/other_user}","gists_url":"https://api.github.com/users/cpuguy83/gists{/gist_id}","starred_url":"https://api.github.com/users/cpuguy83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpuguy83/subscriptions","organizations_url":"https://api.github.com/users/cpuguy83/orgs","repos_url":"https://api.github.com/users/cpuguy83/repos","events_url":"https://api.github.com/users/cpuguy83/events{/privacy}","received_events_url":"https://api.github.com/users/cpuguy83/received_events","type":"User","site_admin":false},"created_at":"2015-11-05T14:07:03Z","updated_at":"2015-11-05T14:07:03Z","body":"It's probably fine.\r\nBut note that if mounts aren't showing up after docker is started, this is indicative of starting docker either with private mount flags or in it's own mount namespace."}}