{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/17041","labels_url":"https://api.github.com/repos/docker/docker/issues/17041/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/17041/comments","events_url":"https://api.github.com/repos/docker/docker/issues/17041/events","html_url":"https://github.com/docker/docker/issues/17041","id":111515990,"number":17041,"title":"docker network disconnect hangs using overlay driver","user":{"login":"rcgoodfellow","id":1010256,"avatar_url":"https://avatars.githubusercontent.com/u/1010256?v=3","gravatar_id":"","url":"https://api.github.com/users/rcgoodfellow","html_url":"https://github.com/rcgoodfellow","followers_url":"https://api.github.com/users/rcgoodfellow/followers","following_url":"https://api.github.com/users/rcgoodfellow/following{/other_user}","gists_url":"https://api.github.com/users/rcgoodfellow/gists{/gist_id}","starred_url":"https://api.github.com/users/rcgoodfellow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rcgoodfellow/subscriptions","organizations_url":"https://api.github.com/users/rcgoodfellow/orgs","repos_url":"https://api.github.com/users/rcgoodfellow/repos","events_url":"https://api.github.com/users/rcgoodfellow/events{/privacy}","received_events_url":"https://api.github.com/users/rcgoodfellow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-10-14T23:36:11Z","updated_at":"2015-10-16T01:24:59Z","closed_at":null,"body":"When I attempt to disconnect a container from an overlay network the command hangs\r\n\r\n__my setup info__\r\n\r\n```\r\n$ uname -a\r\nLinux marina 3.19.0-28-generic #30~14.04.1-Ubuntu SMP Tue Sep 1 09:32:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n```\r\n$ docker info\r\nContainers: 2\r\nImages: 4\r\nEngine Version: 1.9.0-dev\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.19.0-28-generic\r\nOperating System: Ubuntu 14.04.3 LTS\r\nCPUs: 24\r\nTotal Memory: 23.51 GiB\r\nName: marina\r\nID: MMNL:WUCU:BFZS:BDO5:5S55:3S4I:TB4S:RYSN:2OGR:KK4E:RL3H:DHPO\r\nWARNING: No swap limit support\r\nLabels:\r\n com.docker.network.driver.overlay.bind_interface=eth5\r\nExperimental: true\r\nCluster store: consul://localhost:8500\r\n```\r\n```\r\n$ docker version\r\nClient:\r\n Version:      1.9.0-dev\r\n API version:  1.21\r\n Go version:   go1.4.3\r\n Git commit:   ed9434c\r\n Built:        Sat Oct 10 23:54:27 UTC 2015\r\n OS/Arch:      linux/amd64\r\n Experimental: true\r\n\r\nServer:\r\n Version:      1.9.0-dev\r\n API version:  1.21\r\n Go version:   go1.4.3\r\n Git commit:   ed9434c\r\n Built:        Sat Oct 10 23:54:27 UTC 2015\r\n OS/Arch:      linux/amd64\r\n Experimental: true\r\n```\r\n\r\nI am starting docker like this (have also tried with aufs with same results)\r\n```\r\n$ sudo docker daemon -s overlay -H unix:///var/run/docker.sock --cluster-store=consul://localhost:8500 --label=com.docker.network.driver.overlay.bind_interface=eth5\r\n```\r\n\r\n__reproducing the problem__\r\nI start of by creating a few containers and connecting them to an overlay network\r\n```\r\n$ docker network create -d overlay nt\r\n11785bf709d4cf7029e50c52ee277d186bd7efc1ec7e3899e0a8263431aa6efa\r\n$ docker run -itd --name=aa --hostname=aa ubuntu:15.04 bash \r\n38620ac7245deb69a9c4a5d9e1849c49c927c4fe52acbc57f231e1c16b95003e\r\n$ docker run -itd --name=bb --hostname=bb ubuntu:15.04 bash\r\n6edd83fce8356f4378bbf3fc4ae3927747c706d34304a4022cc25532d454a10e\r\n$ docker ps\r\nCONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES\r\n6edd83fce835        ubuntu:15.04        \"bash\"              5 seconds ago       Up 4 seconds                            bb\r\n38620ac7245d        ubuntu:15.04        \"bash\"              24 seconds ago      Up 23 seconds                           aa\r\n$ docker network connect nt aa\r\n$ docker network connect nt bb\r\n$ docker exec -it aa bash\r\nroot@aa:/# apt-get install -y ping\r\n...\r\nroot@aa:/# ping bb\r\nPING bb (172.17.0.2) 56(84) bytes of data.\r\n64 bytes from bb (172.17.0.2): icmp_seq=1 ttl=64 time=0.175 ms\r\n64 bytes from bb (172.17.0.2): icmp_seq=2 ttl=64 time=0.059 ms\r\n^C\r\n--- bb ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 999ms\r\nrtt min/avg/max/mdev = 0.059/0.117/0.175/0.058 ms\r\nroot@aa:/# exit\r\nexit\r\n```\r\nso far so good, now time to disconnect\r\n```\r\n$ docker network disconnect nt bb\r\n```\r\nthis just hangs so I give it a CTL-C and then inspect the network. The endpoint appears to be gone in this view.\r\n```\r\n^C $ \r\n$ docker network inspect nt\r\n{\r\n    \"name\": \"nt\",\r\n    \"id\": \"11785bf709d4cf7029e50c52ee277d186bd7efc1ec7e3899e0a8263431aa6efa\",\r\n    \"driver\": \"overlay\",\r\n    \"containers\": {\r\n        \"38620ac7245deb69a9c4a5d9e1849c49c927c4fe52acbc57f231e1c16b95003e\": {\r\n            \"endpoint\": \"b997263e7953e6d196b60ef057ef4bd06333b4ea79ebdac990434d1d5a0b0c58\",\r\n            \"mac_address\": \"02:42:ac:15:00:01\",\r\n            \"ipv4_address\": \"172.21.0.1/16\",\r\n            \"ipv6_address\": \"\"\r\n        }\r\n    }\r\n}\r\n```\r\nI repeat the process for the other node ``aa``. Have to CTL-C again, but now I do see an empty network which feels like progress. \r\n```\r\n$ docker network inspect nt\r\n{\r\n    \"name\": \"nt\",\r\n    \"id\": \"11785bf709d4cf7029e50c52ee277d186bd7efc1ec7e3899e0a8263431aa6efa\",\r\n    \"driver\": \"overlay\",\r\n    \"containers\": {}\r\n}\r\n```\r\nHowever when I try to remove the network itself it fails saying it has active endpoints.\r\n```\r\n$ docker network rm nt\r\nError response from daemon: network with name nt id 11785bf709d4cf7029e50c52ee277d186bd7efc1ec7e3899e0a8263431aa6efa has active endpoints\r\n```\r\nI have been kinda getting around this by using stopping the docker daemon and using the following\r\n```\r\n$ curl -X DELETE http://localhost:8500/v1/kv/?recurse\r\n```\r\nwhich seems to clear things out. But this process is not really feasible from the tooling and automation we are currently trying to build.\r\n\r\n__other info__\r\n+ this problem does not crop up when using the `bridge` networking driver\r\n+ this process was working before using the exact same underlying OS and machine but with an older docker version which used the `service publish/unpublish` mechanism.\r\n\r\nThanks\r\n\r\nry\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/148570070","html_url":"https://github.com/docker/docker/issues/17041#issuecomment-148570070","issue_url":"https://api.github.com/repos/docker/docker/issues/17041","id":148570070,"user":{"login":"mavenugo","id":5056191,"avatar_url":"https://avatars.githubusercontent.com/u/5056191?v=3","gravatar_id":"","url":"https://api.github.com/users/mavenugo","html_url":"https://github.com/mavenugo","followers_url":"https://api.github.com/users/mavenugo/followers","following_url":"https://api.github.com/users/mavenugo/following{/other_user}","gists_url":"https://api.github.com/users/mavenugo/gists{/gist_id}","starred_url":"https://api.github.com/users/mavenugo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mavenugo/subscriptions","organizations_url":"https://api.github.com/users/mavenugo/orgs","repos_url":"https://api.github.com/users/mavenugo/repos","events_url":"https://api.github.com/users/mavenugo/events{/privacy}","received_events_url":"https://api.github.com/users/mavenugo/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T01:24:59Z","updated_at":"2015-10-16T01:24:59Z","body":"@mbentley ok. I will try to reproduce it. but, since the network states are in the kv-store, the states dont get removed even after reboot. So, I have to check how a restart would help."}}