{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/13850","labels_url":"https://api.github.com/repos/docker/docker/issues/13850/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/13850/comments","events_url":"https://api.github.com/repos/docker/docker/issues/13850/events","html_url":"https://github.com/docker/docker/issues/13850","id":87043395,"number":13850,"title":"docker 1.7.0-rc2 cannot restart due to graph storage driver failed at Cleanup sometimes","user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/38","html_url":"https://github.com/docker/docker/milestones/1.7.0","labels_url":"https://api.github.com/repos/docker/docker/milestones/38/labels","id":1010239,"number":38,"title":"1.7.0","description":"Freeze May 26\r\nRelease  June 16","creator":{"login":"dmcgowan","id":169601,"avatar_url":"https://avatars.githubusercontent.com/u/169601?v=3","gravatar_id":"","url":"https://api.github.com/users/dmcgowan","html_url":"https://github.com/dmcgowan","followers_url":"https://api.github.com/users/dmcgowan/followers","following_url":"https://api.github.com/users/dmcgowan/following{/other_user}","gists_url":"https://api.github.com/users/dmcgowan/gists{/gist_id}","starred_url":"https://api.github.com/users/dmcgowan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmcgowan/subscriptions","organizations_url":"https://api.github.com/users/dmcgowan/orgs","repos_url":"https://api.github.com/users/dmcgowan/repos","events_url":"https://api.github.com/users/dmcgowan/events{/privacy}","received_events_url":"https://api.github.com/users/dmcgowan/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":106,"state":"open","created_at":"2015-03-07T01:54:49Z","updated_at":"2015-06-10T20:34:31Z","due_on":"2016-05-16T07:00:00Z","closed_at":null},"comments":4,"created_at":"2015-06-10T17:20:46Z","updated_at":"2015-06-10T20:54:54Z","closed_at":null,"body":"Restart docker 1.7.0-rc2 has much higher failure rate comparing to previous version:\r\n\r\n```\r\n# docker version\r\nClient version: 1.7.0-rc2\r\nClient API version: 1.19\r\nGo version (client): go1.4.2\r\nGit commit (client): 7ddecf7\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.7.0-rc2\r\nServer API version: 1.19\r\nGo version (server): go1.4.2\r\nGit commit (server): 7ddecf7\r\nOS/Arch (server): linux/amd64\r\n```\r\n```\r\n# service docker restart\r\n[ ok ] Stopping Docker: docker.\r\n[FAIL] Starting Docker: docker failed!\r\n```\r\n\r\nThe below is the output of 'docker info'\r\n```\r\n# docker info\r\nContainers: 97\r\nImages: 75\r\nStorage Driver: aufs\r\n Root Dir: /var/lib/docker/aufs\r\n Backing Filesystem: extfs\r\n Dirs: 269\r\n Dirperm1 Supported: true\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.16.0-0.bpo.4-amd64\r\nOperating System: Debian GNU/Linux 7 (wheezy) (containerized)\r\nCPUs: 1\r\nTotal Memory: 1.664 GiB\r\nName: e2e-test-dawnchen-minion-m6xr\r\nID: M5AZ:UIJ2:NYHP:XJYO:V6A7:6L5V:A6UE:7SOW:QZM3:PRSK:DQAG:AXBF\r\nWARNING: No swap limit support\r\n```\r\n\r\nChecking docker.log, the failure is always caused by graph storage driver Cleanup(): device or resource busy, but every time, it complains a different container. \r\n\r\n```\r\ntime=\"2015-06-10T16:24:40.919554895Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.025876655Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.131932989Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.238114302Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.344047471Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.450212953Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.557326359Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.663467855Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.769463995Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.878903370Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:41.985180657Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.091416327Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.198276427Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.307627502Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.413813792Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.520339799Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.627273325Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.733724209Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.839447508Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:42.976536547Z\" level=info msg=\"GET /containers/json\"\r\ntime=\"2015-06-10T16:24:45.843043001Z\" level=info msg=\"GET /version\"\r\ntime=\"2015-06-10T16:24:50.094241684Z\" level=info msg=\"Processing signal 'terminated'\"\r\n/usr/bin/docker already running.\r\ntime=\"2015-06-10T16:24:51.225044956Z\" level=error msg=\"Error during graph storage driver.Cleanup(): device or resource busy\"\r\ntime=\"2015-06-10T16:25:01.225632711Z\" level=info msg=\"Container 531c6a22ec40f075cc135e846d88d556982974e5a1873c6eb07fb1a051086852 failed to exit within 10 seconds of SIGTERM - using the force\"\r\n```\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/110911061","html_url":"https://github.com/docker/docker/issues/13850#issuecomment-110911061","issue_url":"https://api.github.com/repos/docker/docker/issues/13850","id":110911061,"user":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"created_at":"2015-06-10T20:54:54Z","updated_at":"2015-06-10T20:54:54Z","body":"I just reproduced the issue by running service docker restart. After another try, now docker is running, and here is the output of docker inspect on bad container (I removed Env), it shouldn't cause issue anyway.\r\n\r\n```\r\n# docker inspect 6272d487b4f7\r\n[\r\n{\r\n    \"Id\": \"6272d487b4f7e03911e88bb8492d53608389b397a63ce18ea61ebda73ecd2f5e\",\r\n    \"Created\": \"2015-06-10T16:27:01.257274407Z\",\r\n    \"Path\": \"/bin/sh\",\r\n    \"Args\": [\r\n        \"-c\",\r\n        \"/usr/sbin/td-agent \\\"$FLUENTD_ARGS\\\" \\u003e /var/log/td-agent/td-agent.log\"\r\n    ],\r\n    \"State\": {\r\n        \"Running\": false,\r\n        \"Paused\": false,\r\n        \"Restarting\": false,\r\n        \"OOMKilled\": false,\r\n        \"Dead\": false,\r\n        \"Pid\": 0,\r\n        \"ExitCode\": 137,\r\n        \"Error\": \"\",\r\n        \"StartedAt\": \"2015-06-10T16:27:01.332015819Z\",\r\n        \"FinishedAt\": \"2015-06-10T20:47:35.223971208Z\"\r\n    },\r\n    \"Image\": \"f89d2752e0c3788e7d1a4932e02ea7d689d2fd1efbcb06c55118c26590545cad\",\r\n    \"NetworkSettings\": {\r\n        \"Bridge\": \"\",\r\n        \"EndpointID\": \"\",\r\n        \"Gateway\": \"\",\r\n        \"GlobalIPv6Address\": \"\",\r\n        \"GlobalIPv6PrefixLen\": 0,\r\n        \"HairpinMode\": false,\r\n        \"IPAddress\": \"\",\r\n        \"IPPrefixLen\": 0,\r\n        \"IPv6Gateway\": \"\",\r\n        \"LinkLocalIPv6Address\": \"\",\r\n        \"LinkLocalIPv6PrefixLen\": 0,\r\n        \"MacAddress\": \"\",\r\n        \"NetworkID\": \"\",\r\n        \"PortMapping\": null,\r\n        \"Ports\": null,\r\n        \"SandboxKey\": \"\",\r\n        \"SecondaryIPAddresses\": null,\r\n        \"SecondaryIPv6Addresses\": null\r\n    },\r\n    \"ResolvConfPath\": \"/var/lib/docker/containers/9e28e74debadbdc30c41c715897c4a25fdb72f80c293de8ac84a5a732c47e970/resolv.conf\",\r\n    \"HostnamePath\": \"/var/lib/docker/containers/9e28e74debadbdc30c41c715897c4a25fdb72f80c293de8ac84a5a732c47e970/hostname\",\r\n    \"HostsPath\": \"/var/lib/docker/containers/9e28e74debadbdc30c41c715897c4a25fdb72f80c293de8ac84a5a732c47e970/hosts\",\r\n    \"LogPath\": \"/var/lib/docker/containers/6272d487b4f7e03911e88bb8492d53608389b397a63ce18ea61ebda73ecd2f5e/6272d487b4f7e03911e88bb8492d53608389b397a63ce18ea61ebda73ecd2f5e-json.log\",\r\n    \"Name\": \"/k8s_fluentd-elasticsearch.fe026601_fluentd-elasticsearch-e2e-test-dawnchen-minion-m6xr_default_5f9950d081564dad2522c56233c6ad72_65514b58\",\r\n    \"RestartCount\": 0,\r\n    \"Driver\": \"aufs\",\r\n    \"ExecDriver\": \"native-0.2\",\r\n    \"MountLabel\": \"\",\r\n    \"ProcessLabel\": \"\",\r\n    \"Volumes\": {\r\n        \"/dev/termination-log\": \"/var/lib/kubelet/pods/5f9950d081564dad2522c56233c6ad72/containers/fluentd-elasticsearch/6272d487b4f7e03911e88bb8492d53608389b397a63ce18ea61ebda73ecd2f5e\",\r\n        \"/var/lib/docker/containers\": \"/var/lib/docker/containers\",\r\n        \"/varlog\": \"/var/log\"\r\n    },\r\n    \"VolumesRW\": {\r\n        \"/dev/termination-log\": true,\r\n        \"/var/lib/docker/containers\": true,\r\n        \"/varlog\": true\r\n    },\r\n    \"AppArmorProfile\": \"\",\r\n    \"ExecIDs\": null,\r\n    \"HostConfig\": {\r\n        \"Binds\": [\r\n            \"/var/log:/varlog\",\r\n            \"/var/lib/docker/containers:/var/lib/docker/containers\",\r\n            \"/var/lib/kubelet/pods/5f9950d081564dad2522c56233c6ad72/containers/fluentd-elasticsearch/6272d487b4f7e03911e88bb8492d53608389b397a63ce18ea61ebda73ecd2f5e:/dev/termination-log\"\r\n        ],\r\n        \"ContainerIDFile\": \"\",\r\n        \"LxcConf\": null,\r\n        \"Memory\": 0,\r\n        \"MemorySwap\": -1,\r\n        \"CpuShares\": 102,\r\n        \"CpuPeriod\": 0,\r\n        \"CpusetCpus\": \"\",\r\n        \"CpusetMems\": \"\",\r\n        \"CpuQuota\": 0,\r\n        \"BlkioWeight\": 0,\r\n        \"OomKillDisable\": false,\r\n        \"Privileged\": false,\r\n        \"PortBindings\": null,\r\n        \"Links\": null,\r\n        \"PublishAllPorts\": false,\r\n        \"Dns\": [\r\n            \"10.0.0.10\",\r\n            \"169.254.169.254\",\r\n            \"10.240.0.1\"\r\n        ],\r\n        \"DnsSearch\": [\r\n            \"default.svc.cluster.local\",\r\n            \"svc.cluster.local\",\r\n            \"cluster.local\",\r\n            \"c.golden-system-455.internal.\",\r\n            \"970079080726.google.internal.\",\r\n            \"google.internal.\"\r\n        ],\r\n        \"ExtraHosts\": null,\r\n        \"VolumesFrom\": null,\r\n        \"Devices\": null,\r\n        \"NetworkMode\": \"container:9e28e74debadbdc30c41c715897c4a25fdb72f80c293de8ac84a5a732c47e970\",\r\n        \"IpcMode\": \"container:9e28e74debadbdc30c41c715897c4a25fdb72f80c293de8ac84a5a732c47e970\",\r\n        \"PidMode\": \"\",\r\n        \"UTSMode\": \"\",\r\n        \"CapAdd\": null,\r\n        \"CapDrop\": null,\r\n        \"RestartPolicy\": {\r\n            \"Name\": \"\",\r\n            \"MaximumRetryCount\": 0\r\n        },\r\n        \"SecurityOpt\": null,\r\n        \"ReadonlyRootfs\": false,\r\n        \"Ulimits\": null,\r\n        \"LogConfig\": {\r\n            \"Type\": \"json-file\",\r\n            \"Config\": {}\r\n        },\r\n        \"CgroupParent\": \"/\"\r\n    },\r\n    \"Config\": {\r\n        \"Hostname\": \"fluentd-elasticsearch-e2e-test-dawnchen-minion-m6xr\",\r\n        \"Domainname\": \"\",\r\n        \"User\": \"\",\r\n        \"AttachStdin\": false,\r\n        \"AttachStdout\": false,\r\n        \"AttachStderr\": false,\r\n        \"PortSpecs\": null,\r\n        \"ExposedPorts\": null,\r\n        \"Tty\": false,\r\n        \"OpenStdin\": false,\r\n        \"StdinOnce\": false,\r\n        \"Env\": [\r\n         ...\r\n        ],\r\n        \"Cmd\": [\r\n            \"/bin/sh\",\r\n            \"-c\",\r\n            \"/usr/sbin/td-agent \\\"$FLUENTD_ARGS\\\" \\u003e /var/log/td-agent/td-agent.log\"\r\n        ],\r\n        \"Image\": \"gcr.io/google_containers/fluentd-elasticsearch:1.5\",\r\n        \"Volumes\": null,\r\n        \"VolumeDriver\": \"\",\r\n        \"WorkingDir\": \"\",\r\n        \"Entrypoint\": null,\r\n        \"NetworkDisabled\": false,\r\n        \"MacAddress\": \"\",\r\n        \"OnBuild\": null,\r\n        \"Labels\": {\r\n            \"io.kubernetes.pod.name\": \"default/fluentd-elasticsearch-e2e-test-dawnchen-minion-m6xr\"\r\n        }\r\n    }\r\n}\r\n]\r\n\r\n```\r\n"}}