{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/13261","labels_url":"https://api.github.com/repos/docker/docker/issues/13261/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/13261/comments","events_url":"https://api.github.com/repos/docker/docker/issues/13261/events","html_url":"https://github.com/docker/docker/issues/13261","id":76852002,"number":13261,"title":"overlay backend allows rmi on in-use layers","user":{"login":"stevenschlansker","id":129097,"avatar_url":"https://avatars.githubusercontent.com/u/129097?v=3","gravatar_id":"","url":"https://api.github.com/users/stevenschlansker","html_url":"https://github.com/stevenschlansker","followers_url":"https://api.github.com/users/stevenschlansker/followers","following_url":"https://api.github.com/users/stevenschlansker/following{/other_user}","gists_url":"https://api.github.com/users/stevenschlansker/gists{/gist_id}","starred_url":"https://api.github.com/users/stevenschlansker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevenschlansker/subscriptions","organizations_url":"https://api.github.com/users/stevenschlansker/orgs","repos_url":"https://api.github.com/users/stevenschlansker/repos","events_url":"https://api.github.com/users/stevenschlansker/events{/privacy}","received_events_url":"https://api.github.com/users/stevenschlansker/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels//system/overlay","name":"/system/overlay","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-05-15T20:41:09Z","updated_at":"2015-05-21T18:29:46Z","closed_at":null,"body":"We switched to overlay yesterday and are already seeing non-deterministic problem with the filesystem backend, across multiple containers and teams. Linux 4.0.2, docker 1.6.0, ext4 filesystem.  I initially thought this might be #11012, but since that was reported against XFS originally I decided to open my own issue.\r\n\r\nWe run Docker containers for our CI builds.  Immediately after we migrated to the overlay storage driver, we started to see problems:\r\n```\r\n++ cut -d . -f 1\r\n./../otpl-deploy-scripts/bin/otpl-deploy: line 218: cut: command not found\r\n```\r\n```\r\n[Step 2/5] jetbrains.buildServer.agent.runner.CannotBuildCommandLineException: Unable to find java executable at: /usr/lib/jvm/java-8-openjdk-amd64/bin/java\r\n    at jetbrains.buildServer.agent.runner.JavaCommandLineBuilder.buildPath2Java(JavaCommandLineBuilder.java:155)\r\n    at jetbrains.buildServer.agent.runner.JavaCommandLineBuilder.build(JavaCommandLineBuilder.java:107)\r\n```\r\n\r\nTrying to enter the container, something is clearly fucked:\r\n```\r\nroot@mesos-slave8-qa-uswest2:~# docker exec -it 0adffe3c5da6 /bin/bash\r\nbash-4.3# ls\r\nbash: ls: command not found\r\nbash-4.3# echo *\r\n*\r\nbash-4.3# pwd\r\n/\r\nbash-4.3# /bin/ls\r\nbash: /bin/ls: No such file or directory\r\nbash-4.3# /bin/*\r\nbash: /bin/*: No such file or directory\r\nbash-4.3# find .\r\n.\r\nbash-4.3# bash  # NOTE HOW THIS SUCCEEDS\r\nbash-4.3# exit\r\nbash-4.3# ps\r\nbash-4.3# find /bin/bash\r\n/bin/bash\r\nbash-4.3# find /bin\r\n/bin\r\n```\r\n\r\nLooking from the outside in:\r\n```\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay# cd 0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls\r\nlower-id  merged  upper  work\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls merged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls upper/\r\ndev  etc  home  mnt  run  tmp  var\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find work\r\nwork\r\nwork/work\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find merged\r\nmerged\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find merged/\r\nmerged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls merged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find upper\r\nupper\r\nupper/.dockerenv\r\nupper/.dockerinit\r\nupper/etc\r\nupper/etc/hosts\r\nupper/etc/mtab\r\n...\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls -lh merged/bin/bash\r\n-rwxr-xr-x 0 root root 998K Oct  7  2014 merged/bin/bash\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls -lh merged/bin/\r\ntotal 0\r\n```\r\n\r\nI looked at the mapping, and wonder if it's something related to the absence of a `root` directory:\r\n\r\n```\r\nroot@mesos-slave8-qa-uswest2:~# cat /proc/mounts  | grep 0adf\r\noverlay /mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83/merged overlay rw,relatime,lowerdir=/mnt/docker/overlay/914daaa4ae3ecae1966c92ef38c62835f5f81fd75ae544409d099345e958c93a/root,upperdir=/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83/upper,workdir=/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83/work 0 0\r\n\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls\r\nlower-id  merged  upper  work\r\n```\r\n\r\n```\r\nroot@mesos-slave8-qa-uswest2:/# docker version\r\nClient version: 1.6.0\r\nClient API version: 1.18\r\nGo version (client): go1.4.2\r\nGit commit (client): 4749651\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.6.0\r\nServer API version: 1.18\r\nGo version (server): go1.4.2\r\nGit commit (server): 4749651\r\nOS/Arch (server): linux/amd64\r\n\r\nroot@mesos-slave8-qa-uswest2:/# docker info\r\nContainers: 40\r\nImages: 360\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nKernel Version: 4.0.2\r\nOperating System: Ubuntu 14.04.2 LTS\r\nCPUs: 8\r\nTotal Memory: 29.45 GiB\r\nName: mesos-slave8-qa-uswest2\r\nID: NMLO:YQYU:3N53:LTCK:CBGB:LYIH:GHTZ:Y5ME:UIUZ:4IKD:VUWQ:BN2L\r\nWARNING: No swap limit support\r\n\r\nroot@mesos-slave8-qa-uswest2:~# uname -a\r\nLinux mesos-slave8-qa-uswest2 4.0.2 #1 SMP Tue May 12 23:29:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/104379245","html_url":"https://github.com/docker/docker/issues/13261#issuecomment-104379245","issue_url":"https://api.github.com/repos/docker/docker/issues/13261","id":104379245,"user":{"login":"stevenschlansker","id":129097,"avatar_url":"https://avatars.githubusercontent.com/u/129097?v=3","gravatar_id":"","url":"https://api.github.com/users/stevenschlansker","html_url":"https://github.com/stevenschlansker","followers_url":"https://api.github.com/users/stevenschlansker/followers","following_url":"https://api.github.com/users/stevenschlansker/following{/other_user}","gists_url":"https://api.github.com/users/stevenschlansker/gists{/gist_id}","starred_url":"https://api.github.com/users/stevenschlansker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevenschlansker/subscriptions","organizations_url":"https://api.github.com/users/stevenschlansker/orgs","repos_url":"https://api.github.com/users/stevenschlansker/repos","events_url":"https://api.github.com/users/stevenschlansker/events{/privacy}","received_events_url":"https://api.github.com/users/stevenschlansker/received_events","type":"User","site_admin":false},"created_at":"2015-05-21T18:29:46Z","updated_at":"2015-05-21T18:29:46Z","body":"I upgraded to 1.6.2:\r\n```\r\nroot@mesos-slave1-qa-uswest2:~# docker version\r\nClient version: 1.6.2\r\nClient API version: 1.18\r\nGo version (client): go1.4.2\r\nGit commit (client): 7c8fca2\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.6.2\r\nServer API version: 1.18\r\nGo version (server): go1.4.2\r\nGit commit (server): 7c8fca2\r\nOS/Arch (server): linux/amd64\r\n```\r\nand retried:\r\n```\r\nroot@mesos-slave1-qa-uswest2:~# docker rmi -f docker-ci-uswest2.otenv.com/shardkeygenerator-server:shardkeygenerator-server_teamcity_37\r\nError response from daemon: Conflict, cannot force delete 3e17baa2fbb1 because the running container 03322bdee759 is using it, stop it and retry\r\n```\r\n\r\nSo it seems this affects 1.6.0 but not 1.6.2"}}