{"action":"opened","issue":{"url":"https://api.github.com/repos/docker/docker/issues/9930","labels_url":"https://api.github.com/repos/docker/docker/issues/9930/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/9930/comments","events_url":"https://api.github.com/repos/docker/docker/issues/9930/events","html_url":"https://github.com/docker/docker/issues/9930","id":53562689,"number":9930,"title":"Disk space leak from volumes after docker daemon restart","user":{"login":"galdosd","id":349849,"avatar_url":"https://avatars.githubusercontent.com/u/349849?v=3","gravatar_id":"","url":"https://api.github.com/users/galdosd","html_url":"https://github.com/galdosd","followers_url":"https://api.github.com/users/galdosd/followers","following_url":"https://api.github.com/users/galdosd/following{/other_user}","gists_url":"https://api.github.com/users/galdosd/gists{/gist_id}","starred_url":"https://api.github.com/users/galdosd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galdosd/subscriptions","organizations_url":"https://api.github.com/users/galdosd/orgs","repos_url":"https://api.github.com/users/galdosd/repos","events_url":"https://api.github.com/users/galdosd/events{/privacy}","received_events_url":"https://api.github.com/users/galdosd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-01-06T21:01:14Z","updated_at":"2015-01-06T21:01:14Z","closed_at":null,"body":"\r\nSteps to reproduce:\r\n\r\n1) Start a container with a volume\r\n2) Put some data in the volume\r\n3) Reboot the host, kill -9 the docker daemon, or otherwise uncleanly stop the docker daemon\r\n4) Start the docker daemon again. \r\n5) If there are any containers still apparently around according to docker ps -a, docker rm those\r\nPossibly Expected result (best case): The amount of disk space used is the same as before\r\nPossibly Expected result (worst case): More space is used due to stranded volumes, but there is a reliable (not undocumented internals) way to garbage collect those stranded volumes.\r\nActual result: The disk space from the volume in the container that was running at unclean reboot time is still consumed. Specifically, /var/lib/docker/vfs/dir/(hash) will still exist and all files will still be in there. No amount of docker daemon restarting or anything else will get rid of this, nor will it be possible to find the volume through safe docker means (official client or API). There is just a permanent space leak and the only safe way to clean it up is to ensure all desired images/volumes are backed up elsewhere, stop the docker daemon, rm -rf /var/lib/docker, and restart the daemon, and then restore the images/volumes that were backed up.\r\n\r\n\r\nReproducability: \r\n\r\nVery reproducible.\r\nSaw it happen once on 1.3.3 in production (those are the environment details below)\r\nTried it a single time on my local 1.3.2 install and it immediately reproduced the first time.\r\n\r\nEnvironment info:\r\n\r\nThe host is running on a physical Dell PowerEdge R620\r\nI'm using docker 1.3.3 but could not find any related patches in 1.4.x\r\n\r\n\"uname -a; docker version; docker -D info\" output follows:\r\n\r\nLinux hostname 2.6.32-504.1.3.el6.x86_64 #1 SMP Tue Nov 11 14:19:04 CST 2014 x86_64 x86_64 x86_64 GNU/Linux\r\nClient version: 1.3.3\r\nClient API version: 1.15\r\nGo version (client): go1.3.3\r\nGit commit (client): d344625\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.3.3\r\nServer API version: 1.15\r\nGo version (server): go1.3.3\r\nGit commit (server): d344625\r\nContainers: 9\r\nImages: 159\r\nStorage Driver: devicemapper\r\n Pool Name: docker-253:6-234940768-pool\r\n Pool Blocksize: 65.54 kB\r\n Data file: /var/lib/docker/devicemapper/devicemapper/data\r\n Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n Data Space Used: 16.05 GB\r\n Data Space Total: 107.4 GB\r\n Metadata Space Used: 15.2 MB\r\n Metadata Space Total: 2.147 GB\r\n Library Version: 1.02.82-git (2013-10-04)\r\nExecution Driver: native-0.2\r\nKernel Version: 2.6.32-504.1.3.el6.x86_64\r\nOperating System: <unknown>\r\nDebug mode (server): false\r\nDebug mode (client): true\r\nFds: 25\r\nGoroutines: 35\r\nEventsListeners: 0\r\nInit Path: /usr/bin/docker\r\n\r\nMy apologies in advance if this duplicates another issue and my searching was just poor. I can also provide more information if needed.\r\n"}}