{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/7245","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/7245/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/7245/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/7245/events","html_url":"https://github.com/kubernetes/kubernetes/pull/7245","id":70499286,"number":7245,"title":"Integrate Calico network plugin with vagrant provisioner","user":{"login":"Symmetric","id":1524910,"avatar_url":"https://avatars.githubusercontent.com/u/1524910?v=3","gravatar_id":"","url":"https://api.github.com/users/Symmetric","html_url":"https://github.com/Symmetric","followers_url":"https://api.github.com/users/Symmetric/followers","following_url":"https://api.github.com/users/Symmetric/following{/other_user}","gists_url":"https://api.github.com/users/Symmetric/gists{/gist_id}","starred_url":"https://api.github.com/users/Symmetric/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Symmetric/subscriptions","organizations_url":"https://api.github.com/users/Symmetric/orgs","repos_url":"https://api.github.com/users/Symmetric/repos","events_url":"https://api.github.com/users/Symmetric/events{/privacy}","received_events_url":"https://api.github.com/users/Symmetric/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cla:%20yes","name":"cla: yes","color":"bfe5bf"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/XS","name":"size/XS","color":"009900"}],"state":"open","locked":false,"assignee":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"milestone":null,"comments":21,"created_at":"2015-04-23T19:59:58Z","updated_at":"2015-09-01T23:15:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/7245","html_url":"https://github.com/kubernetes/kubernetes/pull/7245","diff_url":"https://github.com/kubernetes/kubernetes/pull/7245.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/7245.patch"},"body":"I've been working on a prototype integration of [Calico](https://github.com/Metaswitch/calico-docker) networking in Kubernetes. I've used the new [network plugin API](https://github.com/GoogleCloudPlatform/kubernetes/pull/5069) to set up the Pod infra container with Calico networking, and the basic use-case is working.\r\n\r\nThis adds tenant isolation to Kubernetes, so that pod traffic can be restricted using the Calico distributed firewall. It also means that network traffic between nodes is unencapsulated, since Calico uses an L3 BGP network rather than an L2 overlay network like OVS/flannel.\r\n\r\nAt this point I'd like to get some feedback on the general approach, to see if there's anything that I should be doing differently.\r\n\r\n# Overview\r\nHere's an overview of the structure:\r\n\r\n* Add a NETWORK_MODE to chose between openvswitch and calico.\r\n* Update the salt files to install the Calico infrastructure container on each minion.\r\n* Update the salt files to run a new etcd instance on the master, listening on port 6666\r\n  * Needed for the Calico etcd datastore, which can't be run as a pod since we need Calico to be up before we can start pods.\r\n* On pod creation:\r\n  * Record the existing (docker-allocated) pod IP.\r\n  * Delete the bridged veth.\r\n  * Add a new veth pair with the docker-allocated Pod IP, using Calico networking. The node-end of the veth pair is not attached to a vSwitch or docker bridge, instead it is processed by the Linux routing table (i.e. a vRouter).\r\n\r\n# Questions\r\nThis works fine with direct Pod-Pod communication, but I'm not sure about Services -- in order to firewall a flow properly, we need to see the real source IP when it hits the iptables rules on the destination node. I'm concerned that the Service is going to get in the way of this; it looks like the service proxy is rewriting the source IP of packets to stay in the reverse flow, is that correct?\r\n\r\nIf so, any suggestions for how we could get the Service proxy to play nice with Calico?\r\n\r\nAnd, some general questions on the architecture described above:\r\n* Should I move the calico-node container initialization from the salt files to the plugin's `setup` step?\r\n  * I use salt config for setting up the calico node, so not sure what's preferable there.\r\n\r\nAny other feedback on the design is welcome.\r\n\r\nCC: @thockin @lxpollitt\r\n\r\n# TODO\r\n- [x] Add `setup` handler in network plugin\r\n- [x] Add `teardown` handler in network plugin\r\n- [x] Set default policy profile in Calico plugin, so that pods have global connectivity by default\r\n- [x] Tidy up OVS/calico flags\r\n- [x] Figure out how Services are going to work (wait for https://github.com/GoogleCloudPlatform/kubernetes/issues/6666 and/or https://github.com/GoogleCloudPlatform/kubernetes/pull/9210)"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/136889156","html_url":"https://github.com/kubernetes/kubernetes/pull/7245#issuecomment-136889156","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/7245","id":136889156,"user":{"login":"Symmetric","id":1524910,"avatar_url":"https://avatars.githubusercontent.com/u/1524910?v=3","gravatar_id":"","url":"https://api.github.com/users/Symmetric","html_url":"https://github.com/Symmetric","followers_url":"https://api.github.com/users/Symmetric/followers","following_url":"https://api.github.com/users/Symmetric/following{/other_user}","gists_url":"https://api.github.com/users/Symmetric/gists{/gist_id}","starred_url":"https://api.github.com/users/Symmetric/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Symmetric/subscriptions","organizations_url":"https://api.github.com/users/Symmetric/orgs","repos_url":"https://api.github.com/users/Symmetric/repos","events_url":"https://api.github.com/users/Symmetric/events{/privacy}","received_events_url":"https://api.github.com/users/Symmetric/received_events","type":"User","site_admin":false},"created_at":"2015-09-01T23:15:32Z","updated_at":"2015-09-01T23:15:32Z","body":"@derekwaynecarr -- any chance you can take a look at this? Would be great to minimize the pain of rebasing and get this in...\r\n\r\nJust been resized as XS, so shouldn't take much of your time. Cheers!"}}