{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/11012","labels_url":"https://api.github.com/repos/docker/docker/issues/11012/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/11012/comments","events_url":"https://api.github.com/repos/docker/docker/issues/11012/events","html_url":"https://github.com/docker/docker/issues/11012","id":58919529,"number":11012,"title":"odd \"no such file or directory\" error with overlay backend","user":{"login":"jmtd","id":63694,"avatar_url":"https://avatars.githubusercontent.com/u/63694?v=3","gravatar_id":"","url":"https://api.github.com/users/jmtd","html_url":"https://github.com/jmtd","followers_url":"https://api.github.com/users/jmtd/followers","following_url":"https://api.github.com/users/jmtd/following{/other_user}","gists_url":"https://api.github.com/users/jmtd/gists{/gist_id}","starred_url":"https://api.github.com/users/jmtd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmtd/subscriptions","organizations_url":"https://api.github.com/users/jmtd/orgs","repos_url":"https://api.github.com/users/jmtd/repos","events_url":"https://api.github.com/users/jmtd/events{/privacy}","received_events_url":"https://api.github.com/users/jmtd/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels//system/overlay","name":"/system/overlay","color":"fbca04"},{"url":"https://api.github.com/repos/docker/docker/labels//system/storage","name":"/system/storage","color":"FBCA04"},{"url":"https://api.github.com/repos/docker/docker/labels/exp/expert","name":"exp/expert","color":"3B8686"},{"url":"https://api.github.com/repos/docker/docker/labels/kind/bug","name":"kind/bug","color":"FF8C94"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-02-25T15:37:08Z","updated_at":"2015-05-15T20:15:11Z","closed_at":null,"body":"I'm getting a very odd \"no such file or directory\" situation with a simple image, built using\r\n\r\n    FROM jmtd/debian:buildd\r\n    MAINTAINER Jonathan Dowland <jmtd@debian.org>\r\n    RUN apt-get update     && apt-get build-dep -y --no-install-recommends squishyball     && apt-get clean\r\n    CMD [\"/bin/bash\"]\r\n\r\nas follows:\r\n\r\n    $ sudo docker build ./\r\n    …\r\n    INFO[0044] lstat /var/lib/docker/overlay/2d582b61ed2f858de066e550b5750fae2fdd142cf85280a19f6c0b3e9fb24cfb/merged/var/lib/apt/lists/http.debian.net_debian_dists_sid_Release: no such file or directory \r\n\r\nThe exact file that is complained about varies from build to build. The container is still available, and the error reproducible:\r\n\r\n    $ sudo docker diff elated_franklin \r\n    FATA[0000] Error response from daemon: lstat /var/lib/docker/overlay/fd47fb7583498a36c779f009ad116621658664f29f3b190eaa2f65f2fc92b78e/merged/var/lib/apt/lists/http.debian.net_debian_dists_sid_Release: no such file or directory \r\n\r\nThis is with\r\n\r\n    # docker version| sed 's/^/    /'\r\n    Client version: 1.5.0-dev\r\n    Client API version: 1.18\r\n    Go version (client): go1.4.2\r\n    Git commit (client): eb88fa7-dirty\r\n    OS/Arch (client): linux/amd64\r\n    Server version: 1.5.0-dev\r\n    Server API version: 1.18\r\n    Go version (server): go1.4.2\r\n    Git commit (server): eb88fa7-dirty\r\n\r\n(this is `cd9769c55e3c0c405854b84c2c4bd273be79b23d` + [this patch](https://git.centos.org/blob/rpms!docker/c7/SOURCES!secrets.patch), which I've pushed [here](https://github.com/jmtd/docker/tree/docker-1.5.0-dev%2Bsecrets) for convenience). I'm using the overlay storage backend, on top of Linux `3.18.7-200.fc21.x86_64`, with `/var/lib/docker` a distinct XFS partition.\r\n\r\nAny clues much appreciated ☺"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/102515473","html_url":"https://github.com/docker/docker/issues/11012#issuecomment-102515473","issue_url":"https://api.github.com/repos/docker/docker/issues/11012","id":102515473,"user":{"login":"stevenschlansker","id":129097,"avatar_url":"https://avatars.githubusercontent.com/u/129097?v=3","gravatar_id":"","url":"https://api.github.com/users/stevenschlansker","html_url":"https://github.com/stevenschlansker","followers_url":"https://api.github.com/users/stevenschlansker/followers","following_url":"https://api.github.com/users/stevenschlansker/following{/other_user}","gists_url":"https://api.github.com/users/stevenschlansker/gists{/gist_id}","starred_url":"https://api.github.com/users/stevenschlansker/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stevenschlansker/subscriptions","organizations_url":"https://api.github.com/users/stevenschlansker/orgs","repos_url":"https://api.github.com/users/stevenschlansker/repos","events_url":"https://api.github.com/users/stevenschlansker/events{/privacy}","received_events_url":"https://api.github.com/users/stevenschlansker/received_events","type":"User","site_admin":false},"created_at":"2015-05-15T20:15:11Z","updated_at":"2015-05-15T20:15:11Z","body":"I can confirm something is seriously fucked up:\r\n```\r\nroot@mesos-slave8-qa-uswest2:~# docker exec -it 0adffe3c5da6 /bin/bash\r\nbash-4.3# ls\r\nbash: ls: command not found\r\nbash-4.3# echo *\r\n*\r\nbash-4.3# pwd\r\n/\r\nbash-4.3# /bin/ls\r\nbash: /bin/ls: No such file or directory\r\nbash-4.3# /bin/*\r\nbash: /bin/*: No such file or directory\r\nbash-4.3# find .\r\n.\r\nbash-4.3# bash  # NOTE HOW THIS SUCCEEDS\r\nbash-4.3# exit\r\nbash-4.3# ps\r\nbash-4.3# find /bin/bash\r\n/bin/bash\r\nbash-4.3# find /bin\r\n/bin\r\n```\r\n\r\nLooking from the outside in:\r\n```\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay# cd 0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls\r\nlower-id  merged  upper  work\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls merged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls upper/\r\ndev  etc  home  mnt  run  tmp  var\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find work\r\nwork\r\nwork/work\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find merged\r\nmerged\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find merged/\r\nmerged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls merged/\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# find upper\r\nupper\r\nupper/.dockerenv\r\nupper/.dockerinit\r\nupper/etc\r\nupper/etc/hosts\r\nupper/etc/mtab\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls -lh merged/bin/bash\r\n-rwxr-xr-x 0 root root 998K Oct  7  2014 merged/bin/bash\r\nroot@mesos-slave8-qa-uswest2:/mnt/docker/overlay/0adffe3c5da6dca485194ed26135361ee6ea46fbec841ef6a1750997a5673d83# ls -lh merged/bin/\r\ntotal 0\r\n```"}}