{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/13775","labels_url":"https://api.github.com/repos/docker/docker/issues/13775/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/13775/comments","events_url":"https://api.github.com/repos/docker/docker/issues/13775/events","html_url":"https://github.com/docker/docker/issues/13775","id":85615229,"number":13775,"title":"[1.7.0-rc*] Data written to ZFS mounts isn't visible in host mounts","user":{"login":"burke","id":1284,"avatar_url":"https://avatars.githubusercontent.com/u/1284?v=3","gravatar_id":"","url":"https://api.github.com/users/burke","html_url":"https://github.com/burke","followers_url":"https://api.github.com/users/burke/followers","following_url":"https://api.github.com/users/burke/following{/other_user}","gists_url":"https://api.github.com/users/burke/gists{/gist_id}","starred_url":"https://api.github.com/users/burke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burke/subscriptions","organizations_url":"https://api.github.com/users/burke/orgs","repos_url":"https://api.github.com/users/burke/repos","events_url":"https://api.github.com/users/burke/events{/privacy}","received_events_url":"https://api.github.com/users/burke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/regression","name":"kind/regression","color":"DE6262"}],"state":"open","locked":false,"assignee":{"login":"calavera","id":1050,"avatar_url":"https://avatars.githubusercontent.com/u/1050?v=3","gravatar_id":"","url":"https://api.github.com/users/calavera","html_url":"https://github.com/calavera","followers_url":"https://api.github.com/users/calavera/followers","following_url":"https://api.github.com/users/calavera/following{/other_user}","gists_url":"https://api.github.com/users/calavera/gists{/gist_id}","starred_url":"https://api.github.com/users/calavera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calavera/subscriptions","organizations_url":"https://api.github.com/users/calavera/orgs","repos_url":"https://api.github.com/users/calavera/repos","events_url":"https://api.github.com/users/calavera/events{/privacy}","received_events_url":"https://api.github.com/users/calavera/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/38","html_url":"https://github.com/docker/docker/milestones/1.7.0","labels_url":"https://api.github.com/repos/docker/docker/milestones/38/labels","id":1010239,"number":38,"title":"1.7.0","description":"Freeze May 26\r\nRelease  June 16","creator":{"login":"dmcgowan","id":169601,"avatar_url":"https://avatars.githubusercontent.com/u/169601?v=3","gravatar_id":"","url":"https://api.github.com/users/dmcgowan","html_url":"https://github.com/dmcgowan","followers_url":"https://api.github.com/users/dmcgowan/followers","following_url":"https://api.github.com/users/dmcgowan/following{/other_user}","gists_url":"https://api.github.com/users/dmcgowan/gists{/gist_id}","starred_url":"https://api.github.com/users/dmcgowan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmcgowan/subscriptions","organizations_url":"https://api.github.com/users/dmcgowan/orgs","repos_url":"https://api.github.com/users/dmcgowan/repos","events_url":"https://api.github.com/users/dmcgowan/events{/privacy}","received_events_url":"https://api.github.com/users/dmcgowan/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":95,"state":"open","created_at":"2015-03-07T01:54:49Z","updated_at":"2015-06-08T18:47:29Z","due_on":"2016-05-16T07:00:00Z","closed_at":null},"comments":12,"created_at":"2015-06-05T19:29:27Z","updated_at":"2015-06-08T21:43:32Z","closed_at":null,"body":"This is a weird one. I'll keep digging into this and flesh out the issue as I discover more, but:\r\n\r\nI have this directory. It's a ZFS dataset.\r\n\r\n```\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# stat -f . | awk '/Type/ { print $NF }'\r\nzfs\r\n```\r\n\r\n```\r\n# zfs list | grep a929 | grep week\r\nlocutus/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01                                             47K   144G    28K  /u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01\r\nlocutus/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01/6ecddc3dd27d5645ec1a2153e7806f99fc1727e0    19K   144G    19K  /u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01/6ecddc3dd27d5645ec1a2153e7806f99fc1727e0\r\n```\r\n\r\nWhen I mount it into a container, the behaviour is quite different on 1.6.0 and 1.7.0, probably due to the Volumes refactor:\r\n\r\n##### 1.6.0:\r\n\r\n```\r\nroot@0beae52c57ab:/app# grep zfs /proc/mounts\r\nlocutus/shopify-f744a0801011b8a849807d3b53df91e719e1bd5ad8523015d984be663482480f/weekof-2015-06-01 /test zfs rw,relatime,xattr,noacl 0 0\r\nlocutus/shopify-f744a0801011b8a849807d3b53df91e719e1bd5ad8523015d984be663482480f/weekof-2015-06-01/03f195f2b7f552b7fa8093364fcb4fc0727b574a /test/03f195f2b7f552b7fa8093364fcb4fc0727b574a zfs rw,relatime,xattr,noacl 0 0\r\nroot@0beae52c57ab:/app#\r\n```\r\n\r\nNote that here, we see nested volumes.\r\n\r\n##### 1.7.0:\r\n\r\nFor some volumes, depending on circumstances I haven't yet been able to figure out, we get this:\r\n\r\n```\r\n# grep zfs /proc/mounts\r\nlocutus/spy-499d8dac8bb0e5d88a981afb20e1fd26fac699799b72cef71f365376e79e89c4 /test zfs rw,relatime,xattr,noacl 0 0\r\nlocutus/spy-499d8dac8bb0e5d88a981afb20e1fd26fac699799b72cef71f365376e79e89c4/weekof-2015-06-01 /test/weekof-2015-06-01 zfs rw,relatime,xattr,noacl 0 0\r\n```\r\n\r\ni.e.: the same, and there is no issue.\r\n\r\nIn other cases, and again, no idea what causes this:\r\n\r\n```\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# docker run -it --entrypoint=/bin/bash -v $(pwd):/test:rw ba5e -l\r\nroot@1469d291f2af:/app# grep zfs /proc/mounts\r\nlocutus /test zfs rw,relatime,xattr,noacl 0 0\r\nroot@1469d291f2af:/app#\r\n```\r\n\r\nSo note, here, that this directory is mounted on the host as a ZFS dataset, with the same behaviour on my 1.6.0 nodes and my 1.7.0 nodes. However, the 1.7.0 nodes sometimes bind it inside the containers as `locutus` (my zpool name) rather than the full dataset name.\r\n\r\nThere is some very interesting behaviour as a result of this:\r\n\r\n```\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# ls\r\n6ecddc3dd27d5645ec1a2153e7806f99fc1727e0\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# docker run -it --entrypoint=/bin/bash -v $(pwd):/test:rw ba5e -l\r\nroot@bde4abe26612:/app# touch /test/what\r\nroot@bde4abe26612:/app# logout\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# ls\r\n6ecddc3dd27d5645ec1a2153e7806f99fc1727e0\r\nroot@borg2:/u/apps/locutus/shared/zfs/shopify-a929cf7d522b5f094635fadbec2fbf96fc973c96764e35303e4bc80dc6974dea/weekof-2015-06-01# docker run -it --entrypoint=/bin/bash -v $(pwd):/test:rw ba5e -l\r\nroot@2ee6bbc4d492:/app# ls /test\r\n6ecddc3dd27d5645ec1a2153e7806f99fc1727e0  what  wip-6ecddc3dd27d5645ec1a2153e7806f99fc1727e0.10704\r\nroot@2ee6bbc4d492:/app#\r\n```\r\n\r\nTo summarize what's happened above:\r\n\r\n1. List files in dataset on host. Get `ds/{a}`.\r\n2. Mount dataset in new container. See `ds/{a}`. Create `ds/{b}` Now container sees `ds/{a,b}`.\r\n3. Exit container. List files on host. See `ds/{a}`.\r\n4. Start another container with the same mount. See `ds/{a,b}`.\r\n\r\nWhat I would expect to see is that the writes from (2) are visible on the host in step 3. I think this is somehow very related to the incorrect-looking mount name in `/proc/mounts`. I would normally blame ZFS for this, but we have strong empirical evidence that this happens only on docker 1.7.0.\r\n\r\n/cc @calavera"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/110150146","html_url":"https://github.com/docker/docker/issues/13775#issuecomment-110150146","issue_url":"https://api.github.com/repos/docker/docker/issues/13775","id":110150146,"user":{"login":"burke","id":1284,"avatar_url":"https://avatars.githubusercontent.com/u/1284?v=3","gravatar_id":"","url":"https://api.github.com/users/burke","html_url":"https://github.com/burke","followers_url":"https://api.github.com/users/burke/followers","following_url":"https://api.github.com/users/burke/following{/other_user}","gists_url":"https://api.github.com/users/burke/gists{/gist_id}","starred_url":"https://api.github.com/users/burke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burke/subscriptions","organizations_url":"https://api.github.com/users/burke/orgs","repos_url":"https://api.github.com/users/burke/repos","events_url":"https://api.github.com/users/burke/events{/privacy}","received_events_url":"https://api.github.com/users/burke/received_events","type":"User","site_admin":false},"created_at":"2015-06-08T21:43:32Z","updated_at":"2015-06-08T21:43:32Z","body":"Hunh, super weird. I'll try to think of anything else weird that could be happening on my system tonight, and try printf-debugging the thing myself tomorrow and Wednesday."}}