{"action":"created","issue":{"url":"https://api.github.com/repos/docker/libnetwork/issues/793","labels_url":"https://api.github.com/repos/docker/libnetwork/issues/793/labels{/name}","comments_url":"https://api.github.com/repos/docker/libnetwork/issues/793/comments","events_url":"https://api.github.com/repos/docker/libnetwork/issues/793/events","html_url":"https://github.com/docker/libnetwork/issues/793","id":119839928,"number":793,"title":"network namespaces not being cleaned up on cluster-store bounce","user":{"login":"mgoldsborough","id":1759329,"avatar_url":"https://avatars.githubusercontent.com/u/1759329?v=3","gravatar_id":"","url":"https://api.github.com/users/mgoldsborough","html_url":"https://github.com/mgoldsborough","followers_url":"https://api.github.com/users/mgoldsborough/followers","following_url":"https://api.github.com/users/mgoldsborough/following{/other_user}","gists_url":"https://api.github.com/users/mgoldsborough/gists{/gist_id}","starred_url":"https://api.github.com/users/mgoldsborough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgoldsborough/subscriptions","organizations_url":"https://api.github.com/users/mgoldsborough/orgs","repos_url":"https://api.github.com/users/mgoldsborough/repos","events_url":"https://api.github.com/users/mgoldsborough/events{/privacy}","received_events_url":"https://api.github.com/users/mgoldsborough/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-12-02T00:27:00Z","updated_at":"2015-12-03T17:32:14Z","closed_at":null,"body":"Hi,\r\n\r\nI'm messing around with multi-host networking and hit a few snags.  It seems that the networking namespaces are not cleaned up if the cluster-store goes down while containers are running.  I get the following:\r\n\r\n```\r\nnode3:~# docker run -it  --rm --name container3  --net=RED busybox\r\nError response from daemon: Cannot start container e5d573f3201a996c39e562f008303a2e1f69002931a6ad4ecb7bbce771dc4e4f: container already connected to network RED\r\n```\r\n\r\n## Here's how I got to this point\r\n\r\n### Hardware\r\n\r\n* AWS\r\n* All nodes are t2.micros\r\n* All nodes are running stock Ubuntu 14.04 LTS\r\n\r\n### Network\r\n\r\nI have a 3 node setup on AWS with the following IPs:\r\n\r\n* CIDR block: 10.10.0.0/16\r\n* node1: 10.10.1.10 (consul master)\r\n* node2: 10.10.2.10\r\n* node3: 10.10.3.10\r\n* All internal traffic on 10.10.0.0/16 is open.\r\n\r\n### Software\r\n\r\nOn each box I executed the following to get docker installed.\r\n\r\n```\r\n# apt-get install linux-image-3.19.0-33-generic\r\n# curl -sSL https://get.docker.com/ | sh\r\n```\r\n\r\n## Steps to reproduce:\r\n\r\n```\r\nnode1:~# docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap\r\nnode1:~# docker ps\r\nCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                            NAMES\r\nd85135d0781e        progrium/consul     \"/bin/start -server -\"   3 hours ago         Up 16 minutes       53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   consul\r\n\r\n// Great! Consul is running on node1\r\n\r\nnode2:~# docker network create -d overlay --subnet=10.10.10.0/24 RED\r\nnode2:~# docker run -it --name container2 --rm --net RED busybox\r\n\r\n// Dropped to a busybox shell on node2, container named **container2**\r\n\r\nnode3:~# docker run -it --name container3 --rm --net RED busybox\r\n/ # ping container2\r\nPING container2 (10.10.10.2): 56 data bytes\r\n64 bytes from 10.10.10.2: seq=0 ttl=64 time=0.916 ms\r\n64 bytes from 10.10.10.2: seq=1 ttl=64 time=0.816 ms\r\n--- container2 ping statistics ---\r\n2 packets transmitted, 2 packets received, 0% packet loss\r\nround-trip min/avg/max = 0.816/0.866/0.916 ms\r\n\r\n// I can also ping from container2 -> container3.  Everything looks good from here.\r\n// Now I stop consul on node1 to simulate an outage or unreliable network link.\r\n// I do this while both busybox shells are running on node2 and node3.\r\n\r\nnode1:~# docker stop consul\r\n\r\n// back on node2, I exit out of busybox. \r\n// NOTE: This is with consul still stopped on node1!  This is important to reproduce\r\n// After busybox on node2 has existed, I restart consul on node1\r\n\r\nnode1:~# docker start consul\r\n\r\n// With consul back up I try to start a container on the RED network on node2\r\n\r\nnode2:~# docker run -it --name container2 --rm --net RED busybox\r\nError response from daemon: Cannot start container 1a99ec0339ca7116f47ea9c63a4a2868563a851b4f8bf40d18530a85d96f290a: container already connected to network RED\r\n\r\n// I restart docker and try again but it complains about files existing already:\r\n\r\nnode2:~# service docker restart\r\ndocker stop/waiting\r\ndocker start/running, process 16893\r\n\r\n# docker run -it --name container2 --rm --net RED busybox\r\nError response from daemon: Cannot start container 20b1998041019a5dc1e709a7814fd411d946a548f2f0644b1813d2f3c345c90a: subnet sandbox join failed for \"10.10.10.0/24\": error creating vxlan interface: file exists\r\n\r\n// The only way I can get the container to start is to manually unmount and delete the network namespaces in /var/run/docker/netns e.g.\r\n\r\n# service docker stop\r\ndocker stop/waiting\r\nnode2:~# umount /var/run/docker/netns/*\r\nnode2:~# rm /var/run/docker/netns/*\r\nnode2:~# service docker start\r\ndocker start/running, process 17138\r\nnode2:~# docker run -it --name container2 --rm --net RED busybox\r\n/ # <--- busybox shell, success!\r\n```\r\n\r\n## Env info\r\n\r\n### node1 info\r\n\r\n```\r\nnode1:~# docker version\r\nClient:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nnode1:~# uname -a\r\nLinux ip-10-10-1-10 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n### node2 info\r\n\r\n```\r\nnode2:~# docker version\r\nClient:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nnode2:~# uname -a\r\nLinux ip-10-10-2-10 3.19.0-37-generic #42~14.04.1-Ubuntu SMP Mon Nov 23 15:13:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\nnode2:~# cat /etc/default/docker \r\nDOCKER_OPTS=\"-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://10.10.1.10:8500/network --cluster-advertise=eth0:2375\"\r\n```\r\n\r\n### node3 info\r\n\r\n```\r\nnode3:~# docker version\r\nClient:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.9.1\r\n API version:  1.21\r\n Go version:   go1.4.2\r\n Git commit:   a34a1d5\r\n Built:        Fri Nov 20 13:12:04 UTC 2015\r\n OS/Arch:      linux/amd64\r\n\r\nnode3:~# uname -a\r\nLinux ip-10-10-3-10 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\nnode3:~# cat /etc/default/docker \r\nDOCKER_OPTS=\"-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://10.10.1.10:8500/network --cluster-advertise=eth0:2375\"\r\n```\r\n\r\n## Reference:\r\n\r\n* Steps mainly came from [here](https://github.com/docker/dceu_tutorials/blob/master/6-networking.md)\r\n"},"comment":{"url":"https://api.github.com/repos/docker/libnetwork/issues/comments/161724885","html_url":"https://github.com/docker/libnetwork/issues/793#issuecomment-161724885","issue_url":"https://api.github.com/repos/docker/libnetwork/issues/793","id":161724885,"user":{"login":"mgoldsborough","id":1759329,"avatar_url":"https://avatars.githubusercontent.com/u/1759329?v=3","gravatar_id":"","url":"https://api.github.com/users/mgoldsborough","html_url":"https://github.com/mgoldsborough","followers_url":"https://api.github.com/users/mgoldsborough/followers","following_url":"https://api.github.com/users/mgoldsborough/following{/other_user}","gists_url":"https://api.github.com/users/mgoldsborough/gists{/gist_id}","starred_url":"https://api.github.com/users/mgoldsborough/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgoldsborough/subscriptions","organizations_url":"https://api.github.com/users/mgoldsborough/orgs","repos_url":"https://api.github.com/users/mgoldsborough/repos","events_url":"https://api.github.com/users/mgoldsborough/events{/privacy}","received_events_url":"https://api.github.com/users/mgoldsborough/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T17:32:14Z","updated_at":"2015-12-03T17:32:14Z","body":"Okay, thanks.  I'd be happy to help from a testing standpoint but I'll probably wait until the official Docker 1.10 release to update my production servers :D"}}