{"action":"created","issue":{"url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/256","labels_url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/256/labels{/name}","comments_url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/256/comments","events_url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/256/events","html_url":"https://github.com/Metaswitch/calico-docker/issues/256","id":92139820,"number":256,"title":"Container routes disappear if node-to-node subnets are in a configured pool","user":{"login":"spikecurtis","id":5375600,"avatar_url":"https://avatars.githubusercontent.com/u/5375600?v=3","gravatar_id":"","url":"https://api.github.com/users/spikecurtis","html_url":"https://github.com/spikecurtis","followers_url":"https://api.github.com/users/spikecurtis/followers","following_url":"https://api.github.com/users/spikecurtis/following{/other_user}","gists_url":"https://api.github.com/users/spikecurtis/gists{/gist_id}","starred_url":"https://api.github.com/users/spikecurtis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spikecurtis/subscriptions","organizations_url":"https://api.github.com/users/spikecurtis/orgs","repos_url":"https://api.github.com/users/spikecurtis/repos","events_url":"https://api.github.com/users/spikecurtis/events{/privacy}","received_events_url":"https://api.github.com/users/spikecurtis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-30T16:39:18Z","updated_at":"2015-06-30T18:30:17Z","closed_at":null,"body":"Seen in the field.\r\n\r\nUser had two real interfaces configured: `eth0` and `wlan0` as follows:\r\n\r\n    default via 192.168.1.1 dev wlan0  proto static \r\n    10.111.111.101 via 192.168.2.7 dev eth0  proto bird \r\n    172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1 \r\n    192.168.1.0/24 dev wlan0  proto kernel  scope link  src 192.168.1.170  metric 9 \r\n    192.168.2.0/24 dev eth0  proto kernel  scope link  src 192.168.2.8  metric 1 \r\n\r\nConfigured IP pools in Calico were 192.168.0.0/16 (default) and 10.111.111.0/24.  User then starts containers with IP addresses\r\n\r\n    10.111.111.100\r\n    10.111.111.101\r\n    10.111.111.102\r\n\r\nBut, the routes for these appear and then disappear from the routing table.\r\n\r\nFrom the bird log:\r\n\r\n```\r\n2015-06-27 06:54:57 <TRACE> bgp_192_168_2_7: Got KEEPALIVE\r\n2015-06-27 06:55:38 <TRACE> bgp_192_168_2_7: Sending KEEPALIVE\r\n2015-06-27 06:55:43 <TRACE> bgp_192_168_2_7 < added 10.111.111.100/32 dev cali052870981bb\r\n2015-06-27 06:55:43 <TRACE> bgp_192_168_2_7: Sending UPDATE\r\n2015-06-27 06:55:55 <TRACE> bgp_192_168_2_7 < added 10.111.111.102/32 dev cali035f53981c2\r\n2015-06-27 06:55:55 <TRACE> bgp_192_168_2_7: Sending UPDATE\r\n2015-06-27 06:56:11 <TRACE> bgp_192_168_2_7: Got KEEPALIVE\r\n2015-06-27 06:56:55 <TRACE> bgp_192_168_2_7: Sending KEEPALIVE\r\n2015-06-27 06:57:28 <TRACE> bgp_192_168_2_7: Got KEEPALIVE\r\n2015-06-27 06:58:03 <TRACE> bgp_192_168_2_7: Sending KEEPALIVE\r\n2015-06-27 06:58:14 <TRACE> bgp_192_168_2_7 < removed 10.111.111.100/32 dev cali052870981bb\r\n2015-06-27 06:58:14 <TRACE> bgp_192_168_2_7: Sending UPDATE\r\n2015-06-27 06:58:14 <TRACE> bgp_192_168_2_7 < removed 10.111.111.102/32 dev cali035f53981c2\r\n2015-06-27 06:58:14 <TRACE> bgp_192_168_2_7: Sending UPDATE\r\n```\r\n\r\nMy suspicion is that the BIRDs are exporting the `wlan0` and `eth0` routes to each other, since they're filtered in by the 192.168.0.0/16 pool, which makes resolving the container routes break unpredictably.\r\n\r\nWe need to at minimum warn the user if they're configured like this.  It would be good to also sort out the route filter in BIRD to only export real container routes.  (e.g. only /32s for IPv4 and /128s for IPv6)."},"comment":{"url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/comments/117293209","html_url":"https://github.com/Metaswitch/calico-docker/issues/256#issuecomment-117293209","issue_url":"https://api.github.com/repos/Metaswitch/calico-docker/issues/256","id":117293209,"user":{"login":"tnithya","id":8789924,"avatar_url":"https://avatars.githubusercontent.com/u/8789924?v=3","gravatar_id":"","url":"https://api.github.com/users/tnithya","html_url":"https://github.com/tnithya","followers_url":"https://api.github.com/users/tnithya/followers","following_url":"https://api.github.com/users/tnithya/following{/other_user}","gists_url":"https://api.github.com/users/tnithya/gists{/gist_id}","starred_url":"https://api.github.com/users/tnithya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tnithya/subscriptions","organizations_url":"https://api.github.com/users/tnithya/orgs","repos_url":"https://api.github.com/users/tnithya/repos","events_url":"https://api.github.com/users/tnithya/events{/privacy}","received_events_url":"https://api.github.com/users/tnithya/received_events","type":"User","site_admin":false},"created_at":"2015-06-30T18:30:17Z","updated_at":"2015-06-30T18:30:17Z","body":"@spikecurtis  some additional observations\r\n*The problem persists even after the pool contains only 10.111.111.0/24.\r\n*Also, I'm unable to run zookeeper over swarm using the command \r\ndocker -H 192.168.2.7:1000 run -d   -e CALICO_IP=10.111.111.100 -e CALICO_PROFILE=prof -ti -e ZK_SERVERS=\"server.1=10.111.111.100:2888:3888 server.2=10.111.111.101:2888:3888 server.3=10.111.111.102:2888:3888\" -e ZK_ID=1 --add-host=\"zk1:10.111.111.100\" --hostname=zk1 --publish=10.111.111.100:2181:2181/tcp --publish=10.111.111.100:2888:2888/tcp --publish=10.111.111.100:3888:3888/tcp  <image name>\r\nwhere 192.168.2.7 is my swarm manager ip:swarm port\r\nThis throws an exception \"8534a78e5c3b6b8e9902a3b8c192afba420e7d5032822e445abc9544317fff1a\r\nFATA[0000] Error response from daemon: Cannot start container 8534a78e5c3b6b8e9902a3b8c192afba420e7d5032822e445abc9544317fff1a: Error starting userland proxy: listen tcp 10.111.111.100:2181: bind: cannot assign requested address \"\r\n*When I dont run the zookeeper against swarm, I get the container up , with correct ip address and routes into the routing tables. Then this route magically disappears as if it has been filtered"}}