{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/10218","labels_url":"https://api.github.com/repos/docker/docker/issues/10218/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/10218/comments","events_url":"https://api.github.com/repos/docker/docker/issues/10218/events","html_url":"https://github.com/docker/docker/issues/10218","id":54883507,"number":10218,"title":"Error response from daemon: Cannot start container c64662f04dde","user":{"login":"zopyx","id":594239,"avatar_url":"https://avatars.githubusercontent.com/u/594239?v=3","gravatar_id":"","url":"https://api.github.com/users/zopyx","html_url":"https://github.com/zopyx","followers_url":"https://api.github.com/users/zopyx/followers","following_url":"https://api.github.com/users/zopyx/following{/other_user}","gists_url":"https://api.github.com/users/zopyx/gists{/gist_id}","starred_url":"https://api.github.com/users/zopyx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zopyx/subscriptions","organizations_url":"https://api.github.com/users/zopyx/orgs","repos_url":"https://api.github.com/users/zopyx/repos","events_url":"https://api.github.com/users/zopyx/events{/privacy}","received_events_url":"https://api.github.com/users/zopyx/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/%2Fsystem%2Fnetworking","name":"/system/networking","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-01-20T13:36:12Z","updated_at":"2015-01-21T12:03:04Z","closed_at":null,"body":"Running the following image from scratch on a new CentOS 7 installation fails badly:\r\n\r\ndocker run \\\r\n  --volume=/:/rootfs:ro \\\r\n  --volume=/var/run:/var/run:rw \\\r\n  --volume=/sys:/sys:ro \\\r\n  --volume=/var/lib/docker/:/var/lib/docker:ro \\\r\n  --publish=8080:8080 \\\r\n  --detach=true \\\r\n  --name=cadvisor \\\r\n  google/cadvisor:latest\r\n\r\na2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83\r\n2015/01/20 14:32:11 Error response from daemon: Cannot start container a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83:  (exit status 1)\r\n--------------\r\n\r\n[root@docker log]# uname -a\r\nLinux docker.zopyx.com 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux\r\n[root@docker log]# docker version\r\nClient version: 1.3.2\r\nClient API version: 1.15\r\nGo version (client): go1.3.3\r\nGit commit (client): 39fa2fa/1.3.2\r\nOS/Arch (client): linux/amd64\r\nServer version: 1.3.2\r\nServer API version: 1.15\r\nGo version (server): go1.3.3\r\nGit commit (server): 39fa2fa/1.3.2\r\n[root@docker log]# docker --version\r\nDocker version 1.3.2, build 39fa2fa/1.3.2\r\n[root@docker log]# docker -D info\r\nContainers: 11\r\nImages: 175\r\nStorage Driver: devicemapper\r\n Pool Name: docker-253:1-203062427-pool\r\n Pool Blocksize: 65.54 kB\r\n Data file: /var/lib/docker/devicemapper/devicemapper/data\r\n Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata\r\n Data Space Used: 9.075 GB\r\n Data Space Total: 107.4 GB\r\n Metadata Space Used: 11.17 MB\r\n Metadata Space Total: 2.147 GB\r\n Library Version: 1.02.84-RHEL7 (2014-03-26)\r\nExecution Driver: native-0.2\r\nKernel Version: 3.10.0-123.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nDebug mode (server): false\r\nDebug mode (client): true\r\nFds: 36\r\nGoroutines: 44\r\nEventsListeners: 0\r\nInit SHA1: c906504aa058139c1d0569ecd0aa5f462a73440f\r\nInit Path: /usr/libexec/docker/dockerinit\r\n\r\n-------------\r\n\r\n\r\nJan 20 08:31:50 localhost docker: [7a34f2cd] +job create(cadvisor)\r\nJan 20 08:31:51 localhost kernel: device-mapper: thin: Creation of new snapshot 192 of device 185 failed.\r\nJan 20 08:31:51 localhost kernel: device-mapper: thin: Creation of new snapshot 193 of device 185 failed.\r\nJan 20 08:31:51 localhost kernel: device-mapper: thin: Creation of new snapshot 194 of device 185 failed.\r\nJan 20 08:31:51 localhost kernel: device-mapper: thin: Creation of new snapshot 195 of device 185 failed.\r\nJan 20 08:31:59 localhost kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: discard\r\nJan 20 08:32:09 localhost docker: [info] GET /images/json?all=False\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] +job images()\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] -job images() = OK (0)\r\nJan 20 08:32:09 localhost docker: [info] GET /images/json?all=True\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] +job images()\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] -job images() = OK (0)\r\nJan 20 08:32:09 localhost docker: [info] GET /containers/json?all=False&size=False\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] +job containers()\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] -job containers() = OK (0)\r\nJan 20 08:32:09 localhost docker: [info] GET /containers/json?all=True&size=False\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] +job containers()\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] -job containers() = OK (0)\r\nJan 20 08:32:09 localhost docker: [info] GET /events?since=1421760710&until=1421760729\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] +job events()\r\nJan 20 08:32:09 localhost docker: [7a34f2cd] -job events() = OK (0)\r\nJan 20 08:32:10 localhost docker: [7a34f2cd] +job log(create, a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83, google/cadvisor:0.8.0)\r\nJan 20 08:32:10 localhost docker: [7a34f2cd] -job log(create, a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83, google/cadvisor:0.8.0) = OK (0)\r\nJan 20 08:32:10 localhost docker: [7a34f2cd] -job create(cadvisor) = OK (0)\r\nJan 20 08:32:10 localhost docker: [info] POST /v1.15/containers/a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83/start\r\nJan 20 08:32:10 localhost docker: [7a34f2cd] +job start(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)\r\nJan 20 08:32:10 localhost kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: discard\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] +job allocate_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job allocate_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83) = OK (0)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] +job allocate_port(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd.allocate_port(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)] Received an unexpected error during port allocation: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 ! -i docker0 -j DNAT --to-destination 172.17.0.12:8080: iptables: No chain/target/match by that name.\r\nJan 20 08:32:11 localhost docker: (exit status 1)\r\nJan 20 08:32:11 localhost docker: iptables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 8080 ! -i docker0 -j DNAT --to-destination 172.17.0.12:8080: iptables: No chain/target/match by that name.\r\nJan 20 08:32:11 localhost docker: (exit status 1)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job allocate_port(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83) = ERR (1)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] +job release_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job release_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83) = OK (0)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] +job release_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job release_interface(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83) = OK (0)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] +job log(die, a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83, google/cadvisor:0.8.0)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job log(die, a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83, google/cadvisor:0.8.0) = OK (0)\r\nJan 20 08:32:11 localhost docker: Cannot start container a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83:  (exit status 1)\r\nJan 20 08:32:11 localhost docker: [7a34f2cd] -job start(a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83) = ERR (1)\r\nJan 20 08:32:11 localhost docker: [error] server.go:1207 Handler for POST /containers/{name:.*}/start returned error: Cannot start container a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83:  (exit status 1)\r\nJan 20 08:32:11 localhost docker: [error] server.go:110 HTTP Error: statusCode=500 Cannot start container a2324698f5efaf98bfd9efc952cca898c58d8b38faf3f9e5272ff0851d048a83:  (exit status 1)\r\n^C\r\n[root@docker log]#"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/70825734","html_url":"https://github.com/docker/docker/issues/10218#issuecomment-70825734","issue_url":"https://api.github.com/repos/docker/docker/issues/10218","id":70825734,"user":{"login":"unclejack","id":1409668,"avatar_url":"https://avatars.githubusercontent.com/u/1409668?v=3","gravatar_id":"","url":"https://api.github.com/users/unclejack","html_url":"https://github.com/unclejack","followers_url":"https://api.github.com/users/unclejack/followers","following_url":"https://api.github.com/users/unclejack/following{/other_user}","gists_url":"https://api.github.com/users/unclejack/gists{/gist_id}","starred_url":"https://api.github.com/users/unclejack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unclejack/subscriptions","organizations_url":"https://api.github.com/users/unclejack/orgs","repos_url":"https://api.github.com/users/unclejack/repos","events_url":"https://api.github.com/users/unclejack/events{/privacy}","received_events_url":"https://api.github.com/users/unclejack/received_events","type":"User","site_admin":false},"created_at":"2015-01-21T12:03:03Z","updated_at":"2015-01-21T12:03:03Z","body":"Similar problems have been encountered with third party firewall scripts and cron jobs which alter netfilter (iptables) rules, and chains.\r\nThis has been observed in https://github.com/docker/docker/issues/9236 and it turned out it was actually APF (Advanced Policy Firewall, https://www.rfxn.com/projects/advanced-policy-firewall/). The user had this on their system and it was making their containers lose Internet connectivity when the cron job would flush rules and add other rules.\r\n\r\nOn such setups where external tools are altering the netfilter rules, there's no way to guarantee that everything will function properly.\r\nEven if Docker would just re-create the chains and rules it needs on systems where a tool is deleting Docker's chain and rules is being used, this wouldn't guarantee any of the following: \r\n* that the rules and chains won't be removed again\r\n* continuous and interrupted Internet connectivity for the containers \r\n* the connectivity between the containers themselves\r\n* that all the rules and chains won't be removed between the execution of one iptables command and the next\r\n\r\nAs far as Docker is concerned, this is what can be done:\r\n1. try to detect the removal of chains / rules which have been created before \r\n2. document this problem in the official documentation to make it easier to debug\r\n\r\nI've also got CentOS 6 and CentOS 7 systems running Docker, but I haven't run into these problems so far. This means that the problem is most likely not caused by something found in a standard install of CentOS 6 and CentOS 7.\r\n\r\nNetfilter would never remove the DOCKER chain on its own while the system is running. This is something done by an external tool.\r\n\r\nYour system is probably running some kind of firewall scripts which re-apply netfilter rules from time to time, possibly with a cron job or using a daemon. Please take a look at your system to try to find the actual tool which does this. Once you've figured out what tool is doing this, you can probably grab the netfilter rules it's setting up and write a new less intrusive firewall script or you could simply modify that firewall script to stop deleting chains and to not flush rules.\r\n\r\nAfter you've taken a look and investigated, please comment again on this issue so that we can investigate this further if required, document the problem in a better way and let other users avoid it.\r\n\r\n"}}