{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/212","labels_url":"https://api.github.com/repos/coreos/bugs/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/212/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/212/events","html_url":"https://github.com/coreos/bugs/issues/212","id":52170744,"number":212,"title":"DHCP With multiple interfaces can cause connectivity issues.","user":{"login":"nickethier","id":766500,"avatar_url":"https://avatars.githubusercontent.com/u/766500?v=3","gravatar_id":"","url":"https://api.github.com/users/nickethier","html_url":"https://github.com/nickethier","followers_url":"https://api.github.com/users/nickethier/followers","following_url":"https://api.github.com/users/nickethier/following{/other_user}","gists_url":"https://api.github.com/users/nickethier/gists{/gist_id}","starred_url":"https://api.github.com/users/nickethier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickethier/subscriptions","organizations_url":"https://api.github.com/users/nickethier/orgs","repos_url":"https://api.github.com/users/nickethier/repos","events_url":"https://api.github.com/users/nickethier/events{/privacy}","received_events_url":"https://api.github.com/users/nickethier/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/coreos/bugs/labels/component/networkd","name":"component/networkd","color":"0052cc"},{"url":"https://api.github.com/repos/coreos/bugs/labels/priority/P0","name":"priority/P0","color":"DF0101"},{"url":"https://api.github.com/repos/coreos/bugs/labels/team/os","name":"team/os","color":"ffffff"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":27,"created_at":"2014-12-16T21:39:21Z","updated_at":"2015-11-13T17:21:33Z","closed_at":null,"body":"Channel: stable\r\nInstall method: iPXE boot\r\nRunning version `494.5`\r\n\r\nIn some cases, depending on what order interfaces come up, the route table can get screwed up from multiple interfaces auto-configuring via DHCP. This relates to #210.\r\n\r\nThis happens before the `user-cloudinit-proc-cmdline.service` unit can run so theres no way for me to get a cloud-config onto the box to disable DHCP. \r\n\r\nHere are the interfaces on the box:\r\n```\r\ncore@localhost ~ $ ip link\r\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT\r\n    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r\n2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000\r\n    link/ether 00:xx:xx:xx:28:6a brd ff:ff:ff:ff:ff:ff\r\n3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000\r\n    link/ether 00:xx:xx:xx:28:6b brd ff:ff:ff:ff:ff:ff\r\n```\r\n\r\nAnd the route table:\r\n```\r\ncore@localhost ~ $ ip route\r\ndefault via 172.17.3.1 dev eno2  metric 1024\r\ndefault via 172.17.3.1 dev eno1  metric 1024\r\n172.17.3.0/24 dev eno2  proto kernel  scope link  src 172.17.3.93\r\n172.17.3.0/24 dev eno1  proto kernel  scope link  src 172.17.3.42\r\n172.17.3.1 dev eno2  scope link  metric 1024\r\n172.17.3.1 dev eno1  scope link  metric 1024\r\n```\r\n\r\nSo in this case `eno2` will work but `eno1` will not due to the routing. But since eno1 is the first interface traffic will by default attempt to use it.\r\n\r\nExample (pinging the ip I fetch my cloud-config from):\r\n```\r\ncore@localhost ~ $ ping 172.31.1.10\r\nPING 172.31.1.10 (172.31.1.10) from 172.17.3.42 eno1: 56(84) bytes of data.\r\n^C\r\n--- 172.31.1.10 ping statistics ---\r\n6 packets transmitted, 0 received, 100% packet loss, time 5000ms\r\n\r\ncore@localhost ~ $ ping -I eno2 172.31.1.10\r\nPING 172.31.1.10 (172.31.1.10) from 172.17.3.93 eno2: 56(84) bytes of data.\r\n64 bytes from 172.31.1.10: icmp_seq=1 ttl=62 time=1.80 ms\r\n64 bytes from 172.31.1.10: icmp_seq=2 ttl=62 time=1.04 ms\r\n64 bytes from 172.31.1.10: icmp_seq=3 ttl=62 time=1.02 ms\r\n64 bytes from 172.31.1.10: icmp_seq=4 ttl=62 time=1.11 ms\r\n64 bytes from 172.31.1.10: icmp_seq=5 ttl=62 time=1.14 ms\r\n64 bytes from 172.31.1.10: icmp_seq=6 ttl=62 time=1.01 ms\r\n^C\r\n--- 172.31.1.10 ping statistics ---\r\n6 packets transmitted, 6 received, 0% packet loss, time 5006ms\r\nrtt min/avg/max/mdev = 1.010/1.189/1.806/0.281 ms\r\n```\r\n\r\nI've worked around this by blacklisting the macs on the second interface on my dhcp server for my lab but this doesn't scale too well. I don't see why you would need more that one interface to be configured via DHCP (iPXE is a good example, it will only auto configure the first interface*), so it would seem the best resolution would be to only enable DHCP for a single interface. I'm not familiar enough with networkd yet to know if this is a possibility.\r\n\r\n\r\n*ipxe ifconf description:\r\n\r\n>Automatically configure a network interface. iPXE will open the first specified network interface and attempt to automatically configure it. If automatic configuration succeeds, the command will terminate and the network interface will be left open. If automatic configuration fails, the network interface will be closed and iPXE will proceed to the next network interface in the list.\r\n>If no network interfaces are explicitly specified, iPXE will try all available network interfaces.\r\n>If no configurators are explicitly specified, iPXE will try all available configurators.\r\n\r\n\r\nos-release:\r\n```\r\ncore@localhost ~ $ cat /etc/os-release\r\nNAME=CoreOS\r\nID=coreos\r\nVERSION=494.5.0\r\nVERSION_ID=494.5.0\r\nBUILD_ID=\r\nPRETTY_NAME=\"CoreOS 494.5.0\"\r\nANSI_COLOR=\"1;32\"\r\nHOME_URL=\"https://coreos.com/\"\r\nBUG_REPORT_URL=\"https://github.com/coreos/bugs/issues\"\r\n```"},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/156493038","html_url":"https://github.com/coreos/bugs/issues/212#issuecomment-156493038","issue_url":"https://api.github.com/repos/coreos/bugs/issues/212","id":156493038,"user":{"login":"nickethier","id":766500,"avatar_url":"https://avatars.githubusercontent.com/u/766500?v=3","gravatar_id":"","url":"https://api.github.com/users/nickethier","html_url":"https://github.com/nickethier","followers_url":"https://api.github.com/users/nickethier/followers","following_url":"https://api.github.com/users/nickethier/following{/other_user}","gists_url":"https://api.github.com/users/nickethier/gists{/gist_id}","starred_url":"https://api.github.com/users/nickethier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nickethier/subscriptions","organizations_url":"https://api.github.com/users/nickethier/orgs","repos_url":"https://api.github.com/users/nickethier/repos","events_url":"https://api.github.com/users/nickethier/events{/privacy}","received_events_url":"https://api.github.com/users/nickethier/received_events","type":"User","site_admin":false},"created_at":"2015-11-13T17:21:33Z","updated_at":"2015-11-13T17:21:33Z","body":"I'd +1 using kernel parameters as well."}}