{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/17653","labels_url":"https://api.github.com/repos/docker/docker/issues/17653/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/17653/comments","events_url":"https://api.github.com/repos/docker/docker/issues/17653/events","html_url":"https://github.com/docker/docker/issues/17653","id":114825551,"number":17653,"title":"Random \"Cannot start container\" Errors on 1.9.0-rc5 CentOS7","user":{"login":"ryanbauman","id":1096675,"avatar_url":"https://avatars.githubusercontent.com/u/1096675?v=3","gravatar_id":"","url":"https://api.github.com/users/ryanbauman","html_url":"https://github.com/ryanbauman","followers_url":"https://api.github.com/users/ryanbauman/followers","following_url":"https://api.github.com/users/ryanbauman/following{/other_user}","gists_url":"https://api.github.com/users/ryanbauman/gists{/gist_id}","starred_url":"https://api.github.com/users/ryanbauman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryanbauman/subscriptions","organizations_url":"https://api.github.com/users/ryanbauman/orgs","repos_url":"https://api.github.com/users/ryanbauman/repos","events_url":"https://api.github.com/users/ryanbauman/events{/privacy}","received_events_url":"https://api.github.com/users/ryanbauman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/docker/docker/milestones/45","html_url":"https://github.com/docker/docker/milestones/1.9.1","labels_url":"https://api.github.com/repos/docker/docker/milestones/45/labels","id":1385783,"number":45,"title":"1.9.1","description":"","creator":{"login":"tiborvass","id":827131,"avatar_url":"https://avatars.githubusercontent.com/u/827131?v=3","gravatar_id":"","url":"https://api.github.com/users/tiborvass","html_url":"https://github.com/tiborvass","followers_url":"https://api.github.com/users/tiborvass/followers","following_url":"https://api.github.com/users/tiborvass/following{/other_user}","gists_url":"https://api.github.com/users/tiborvass/gists{/gist_id}","starred_url":"https://api.github.com/users/tiborvass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tiborvass/subscriptions","organizations_url":"https://api.github.com/users/tiborvass/orgs","repos_url":"https://api.github.com/users/tiborvass/repos","events_url":"https://api.github.com/users/tiborvass/events{/privacy}","received_events_url":"https://api.github.com/users/tiborvass/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":12,"state":"open","created_at":"2015-10-30T17:08:02Z","updated_at":"2015-11-04T21:05:32Z","due_on":"2015-11-10T05:00:00Z","closed_at":null},"comments":6,"created_at":"2015-11-03T14:12:45Z","updated_at":"2015-11-04T21:32:06Z","closed_at":null,"body":"Since upgrading to 1.9.0 RCs on a CentOS7 host (3.10.0-229.14.1.el7.x86_64), I've been seeing random Cannot start container errors, e.g.:\r\n\r\n    Error response from daemon: Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device\r\n\r\nRelevant log entries are below for reference:\r\n\r\n     Nov  3 08:48:10 dockerhost systemd: Starting docker container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.\r\n    Nov  3 08:48:10 dockerhost systemd: Started docker container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.510245551-05:00\" level=warning msg=\"signal: killed\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.551716997-05:00\" level=debug msg=\"Releasing addresses for endpoint adoring_bhabha's interface on network bridge\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.551781175-05:00\" level=debug msg=\"ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.4)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.553129616-05:00\" level=debug msg=\"[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.553166270-05:00\" level=debug msg=\"[devmapper] Unmount(/data/docker/devicemapper/mnt/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.555363308-05:00\" level=debug msg=\"[devmapper] Unmount done\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.555390143-05:00\" level=debug msg=\"[devmapper] deactivateDevice(479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.555473755-05:00\" level=debug msg=\"[devmapper] removeDevice START(docker-253:4-1074522036-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost systemd: Unit iscsi.service cannot be reloaded because it is inactive.\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.593476956-05:00\" level=debug msg=\"[devmapper] removeDevice END(docker-253:4-1074522036-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.593519481-05:00\" level=debug msg=\"[devmapper] deactivateDevice END(479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.593543175-05:00\" level=debug msg=\"[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b) END\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.594937237-05:00\" level=error msg=\"failed to umount /data/docker/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/shm: invalid argument\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.594989602-05:00\" level=error msg=\"failed to umount /data/docker/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/mqueue: invalid argument\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595014866-05:00\" level=error msg=\"479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: Failed to umount ipc filesystems: failed to cleanup ipc mounts:\\ninvalid argument\\ninvalid argument\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595046108-05:00\" level=debug msg=\"[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b)\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595067916-05:00\" level=debug msg=\"[devmapper] UnmountDevice(hash=479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b) END\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595093237-05:00\" level=error msg=\"Error unmounting device 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: UnmountDevice: device not-mounted id 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595199876-05:00\" level=error msg=\"Handler for POST /v1.21/containers/479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b/start returned error: Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device\"\r\n    Nov  3 08:48:10 dockerhost docker: time=\"2015-11-03T08:48:10.595236064-05:00\" level=error msg=\"HTTP Error\" err=\"Cannot start container 479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b: [8] System error: write /sys/fs/cgroup/devices/system.slice/docker-479acdbe53fc117178f9797bce1eaf3a4c3b68021f22df2f3e0f95bd28afdb6b.scope/cgroup.procs: no such device\" statusCode=500\r\n\r\nI'm able to reproduce this fairly reliably on CentOS7 using the technique mentioned in #17387:\r\n \r\n     (set -e ; while true; do ID=$(docker run -d ibuildthecloud/helloworld:latest); docker rm -fv $ID;done)\r\n\r\nThis issue does not occur with Docker 1.8.3 installed, nor can I reproduce it with 1.9.0-rc5 on a system without systemd using an AUFS storage backend (Ubuntu 14.04).\r\n\r\n# Bug Report Information\r\n    $ docker version\r\n    Client:\r\n     Version:      1.9.0-rc5\r\n     API version:  1.21\r\n     Go version:   go1.4.2\r\n     Git commit:   9318004\r\n     Built:        Tue Nov  3 06:14:13 UTC 2015\r\n     OS/Arch:      linux/amd64\r\n    Server:\r\n     Version:      1.9.0-rc5\r\n     API version:  1.21\r\n     Go version:   go1.4.2\r\n     Git commit:   9318004\r\n     Built:        Tue Nov  3 06:14:13 UTC 2015\r\n     OS/Arch:      linux/amd64\r\n    $ docker info\r\n    Containers: 47\r\n    Images: 731\r\n    Server Version: 1.9.0-rc5\r\n    Storage Driver: devicemapper\r\n    Pool Name: docker-253:4-1074522036-pool\r\n    Pool Blocksize: 65.54 kB\r\n    Base Device Size: 107.4 GB\r\n    Backing Filesystem: xfs\r\n    Data file: /dev/centos_docker/docker_data\r\n    Metadata file: /dev/centos_docker/docker_metadata\r\n    Data Space Used: 88.79 GB\r\n    Data Space Total: 614.9 GB\r\n    Data Space Available: 526.1 GB\r\n    Metadata Space Used: 120 MB\r\n    Metadata Space Total: 17.05 GB\r\n    Metadata Space Available: 16.93 GB\r\n    Udev Sync Supported: true\r\n    Deferred Removal Enabled: false\r\n    Deferred Deletion Enabled: false\r\n    Deferred Deleted Device Count: 0\r\n    Library Version: 1.02.93-RHEL7 (2015-01-28)\r\n    Execution Driver: native-0.2\r\n    Logging Driver: json-file\r\n    Kernel Version: 3.10.0-229.14.1.el7.x86_64\r\n    Operating System: CentOS Linux 7 (Core)\r\n    CPUs: 32\r\n    Total Memory: 188.7 GiB\r\n    Name: dockerhost\r\n    ID: IBQH:7AOM:H6KR:XXK5:5JKT:GODZ:EUEM:7VF2:6THK:W7NQ:SOPU:DL5Y\r\n    Debug mode (server): true\r\n       File Descriptors: 33\r\n       Goroutines: 59\r\n       System Time: 2015-11-03T11:38:44.424354397-05:00\r\n       EventsListeners: 0\r\n       Init SHA1: d676303f7af5bf0ded806f3e60d02b299cd01060\r\n       Init Path: /usr/libexec/docker/dockerinit\r\n       Docker Root Dir: /data/docker\r\n    WARNING: bridge-nf-call-iptables is disabled\r\n    WARNING: bridge-nf-call-ip6tables is disabled\r\n    $ uname -a\r\n     Linux dockerhost 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/153872141","html_url":"https://github.com/docker/docker/issues/17653#issuecomment-153872141","issue_url":"https://api.github.com/repos/docker/docker/issues/17653","id":153872141,"user":{"login":"crosbymichael","id":749551,"avatar_url":"https://avatars.githubusercontent.com/u/749551?v=3","gravatar_id":"","url":"https://api.github.com/users/crosbymichael","html_url":"https://github.com/crosbymichael","followers_url":"https://api.github.com/users/crosbymichael/followers","following_url":"https://api.github.com/users/crosbymichael/following{/other_user}","gists_url":"https://api.github.com/users/crosbymichael/gists{/gist_id}","starred_url":"https://api.github.com/users/crosbymichael/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crosbymichael/subscriptions","organizations_url":"https://api.github.com/users/crosbymichael/orgs","repos_url":"https://api.github.com/users/crosbymichael/repos","events_url":"https://api.github.com/users/crosbymichael/events{/privacy}","received_events_url":"https://api.github.com/users/crosbymichael/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T21:32:06Z","updated_at":"2015-11-04T21:32:06Z","body":"ya, i'll have to setup a centos box and try to reproduce this "}}