{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/9983","labels_url":"https://api.github.com/repos/docker/docker/issues/9983/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/9983/comments","events_url":"https://api.github.com/repos/docker/docker/issues/9983/events","html_url":"https://github.com/docker/docker/issues/9983","id":53800078,"number":9983,"title":"Proposal: Network Drivers","user":{"login":"erikh","id":18923,"avatar_url":"https://avatars.githubusercontent.com/u/18923?v=3","gravatar_id":"","url":"https://api.github.com/users/erikh","html_url":"https://github.com/erikh","followers_url":"https://api.github.com/users/erikh/followers","following_url":"https://api.github.com/users/erikh/following{/other_user}","gists_url":"https://api.github.com/users/erikh/gists{/gist_id}","starred_url":"https://api.github.com/users/erikh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikh/subscriptions","organizations_url":"https://api.github.com/users/erikh/orgs","repos_url":"https://api.github.com/users/erikh/repos","events_url":"https://api.github.com/users/erikh/events{/privacy}","received_events_url":"https://api.github.com/users/erikh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2015-01-08T20:57:23Z","updated_at":"2015-01-12T18:15:46Z","closed_at":null,"body":"## Proposal: Network Driver\r\n\r\n**THIS PROPOSAL IS A WORK IN PROGRESS**\r\n\r\nThis proposal brings new functionality and new interfaces to Docker's\r\nnetworking implementation. Some things are still in flux, but our goal is to\r\nget this approved as there is a sizeable effort internally to make this happen.\r\nWe need reviewers and comments on this proposal from the community.\r\n\r\nYou can see some of the work [here](https://github.com/docker/docker-network)\r\nbut this is a work in progress, and its implementation (and the maturity of it)\r\nshould not reflect the state of this document.\r\n\r\n### Concepts\r\n\r\nMany concepts will be used throughout this and several proposals that are in\r\nprocess.  You may see these repeated with slightly different verbiage or may\r\ncontain at-length descriptions of how these interfaces will be implemented.\r\n\r\nThe following includes a description of the interface and its name. The\r\nsub-bullets provide practical examples of these components with the existing\r\nimplementation of Docker as an example.\r\n\r\n* Driver: executable code which is triggered when certain operations are\r\n  requested.\r\n* Extension: A collection of Drivers that supply different portions of\r\n  functionality to Docker.\r\n* State: A key/value store of parameterized data for consumption and mutation by a Driver.\r\n  * [libpack](https://github.com/docker/libpack) is our initial state\r\n    implementation, but we are also evaluating\r\n    [ecc](https://github.com/socketplane/ecc) and others.\r\n* Sandbox: An isolated environment \r\n  * libcontainer's functionality\r\n* Endpoint: An addressable endpoint used for communication over a specific network.\r\n  * veth pair\r\n* Network: A collection of endpoints that are able to communicate to each other.\r\n  * Our ethernet bridge, iptables rules we use.\r\n\r\n### Container Network Model (or CNM)\r\n\r\nThe container network model is a few axioms about how docker wishes to supply\r\ninteroperation between networks and containers.\r\n\r\n1. All containers on a specific network can communicate with each other freely.\r\n1. Multiple networks are the way to segment traffic between containers and\r\n   should be supported by all drivers.\r\n1. Multiple endpoints per container are the way to join a container to\r\n   multiple networks.\r\n1. An endpoint is added to a sandbox to provide it with network connectivity.\r\n\r\nThis has a few consequences:\r\n\r\n* Network-based service discovery will replace on-disk and ENV discovery. This\r\n  allows discovery to be scoped and more flexible for external implementations.\r\n  Implementation of this is still TBD.\r\n* Links will be deprecated or only exist on the default network.\r\n\r\n### Notion of a Default Network\r\n\r\nSince Docker is a tool heavily used by both operations and development\r\npersonnel to different goals respective to their skill sets, it is critical to\r\nhave a functioning \"out of the box\" implementation for people to use with ease.\r\nDocker will create a \"default\" network (named `default`) for the use of basic\r\nnetworking.\r\n\r\n### Networks and Endpoints\r\n\r\nDocker's new system will allow for N networks, but a 'default' network will be\r\ncreated as a convenience for users, and with the default driver it will\r\nfunction similarly to the existing network solution now.\r\n\r\nMultiple endpoints can be created for a single container, and bound to them at\r\nthe same time. The endpoints may live on different networks and may all belong\r\nto one container. \r\n\r\n### Workflow for a Network Driver\r\n\r\nAt boot, a network driver will be given a replay of its state; this will allow\r\nthe driver to return to being in sync with the state of the system, to create\r\nnew networks, etc. How replays are handled by drivers is intentionally\r\nundefined.\r\n\r\nA network can be requested to be created. In the workflow below, the network\r\nassumes to be created already.\r\n\r\nA network driver will be asked at `docker run` (in order, for a given network):\r\n\r\n1. To create an endpoint within the network\r\n1. To join an endpoint to a sandbox\r\n\r\nThe driver will also provide port mapping/expose functionality (see below for\r\nAPI) and communicate with service discovery (TBD).\r\n\r\n### Network API\r\n\r\n#### Driver abstract interface:\r\n\r\n```go\r\ntype Driver interface {\r\n  // Restore from state\r\n  Restore(netstate state.State) error\r\n\r\n  // Create a new network with the network id and supplied parameters\r\n  AddNetwork(netid string, params []string) error\r\n\r\n  // Remove a network using the network ID.\r\n  RemoveNetwork(netid string) error\r\n\r\n  // Retrieve a network by ID.\r\n  GetNetwork(id string) (Network, error)\r\n\r\n  // Create an endpoint for a container with a name, and place it in the sandbox.\r\n  Link(netid, name string, sb sandbox.Sandbox, replace bool) (Endpoint, error)\r\n\r\n  // Unlink an endpoint by name.\r\n  Unlink(netid, name string, sb sandbox.Sandbox) error\r\n}\r\n```\r\n\r\n#### Network abstract interface:\r\n\r\n*(note that Link and Unlink here are merely for convenience and do not require\r\nan alternative implementation)*\r\n\r\n```go\r\n// A network is a perimeter of IP connectivity between network services.\r\ntype Network interface {\r\n  // Id returns the network's globally unique identifier\r\n  Id() string\r\n\r\n  // List returns the IDs of available networks\r\n  List() []string\r\n\r\n  // Link makes the specified sandbox reachable as a named endpoint on the network.\r\n  // If the endpoint already exists, the call will either fail (replace=false), or\r\n  // unlink the previous endpoint.\r\n  //\r\n  // For example mynet.Link(mysandbox, \"db\", true) will make mysandbox available as\r\n  // \"db\" on mynet, and will replace the other previous endpoint, if any.\r\n  //\r\n  // The same sandbox can be linked to multiple networks.\r\n  // The same sandbox can be linked to the same network as multiple endpoints.\r\n  Link(s sandbox.Sandbox, name string, replace bool) (Endpoint, error)\r\n\r\n  // Unlink removes the specified endpoint, unlinking the corresponding sandbox from the\r\n  // network.\r\n  Unlink(name string) error\r\n}\r\n```\r\n\r\n#### Endpoint abstract interface:\r\n\r\n```go\r\n// An endpoint represents a particular member of a network, registered under a certain name\r\n// and reachable over IP by other endpoints on the same network.\r\ntype Endpoint interface {\r\n  // The name of the endpoint.\r\n  Name() string\r\n\r\n  // Expose a port over the network. Publish it as the port to the host if\r\n  // requested.\r\n  Expose(portspec string, publish bool) error\r\n\r\n  // The network this endpoint belongs to.\r\n  Network() Network\r\n}\r\n```\r\n\r\n### `docker net` tool\r\n\r\n`docker net` is the vehicle we're pushing to manipulate networks. The\r\nbasic commands are described below:\r\n\r\n* `create`: create a network\r\n* `destroy`: destroy a network\r\n* `join`: join a container to a network\r\n* `leave`: remove a container from a network\r\n\r\nThere will be a forthcoming UI extension to `docker run` which also selects a\r\nnetwork at run time. This interface is currently to be determined.\r\n\r\n### Our initial implementation\r\n\r\nOur implementation is very similar to docker's existing implementation, which\r\nwe are dubbing `simplebridge`. This driver creates a bridge for each network,\r\nand a veth pair for each endpoint. Networks may contain a set of vxlan peers\r\nwhich will be attached to the bridge to ensure network connectivity to get\r\nmulti-host links.\r\n\r\n### Other potential implementations\r\n\r\n(There are a few of these and I may have missed someone. Please let me know if\r\nyou've done work against what exists today!)\r\n\r\n* [Socketplane](https://github.com/socketplane/socketplane) has a bridge to\r\n  docker that claims to be ready to support the new extensions architecture.\r\n"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/69616812","html_url":"https://github.com/docker/docker/issues/9983#issuecomment-69616812","issue_url":"https://api.github.com/repos/docker/docker/issues/9983","id":69616812,"user":{"login":"erikh","id":18923,"avatar_url":"https://avatars.githubusercontent.com/u/18923?v=3","gravatar_id":"","url":"https://api.github.com/users/erikh","html_url":"https://github.com/erikh","followers_url":"https://api.github.com/users/erikh/followers","following_url":"https://api.github.com/users/erikh/following{/other_user}","gists_url":"https://api.github.com/users/erikh/gists{/gist_id}","starred_url":"https://api.github.com/users/erikh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikh/subscriptions","organizations_url":"https://api.github.com/users/erikh/orgs","repos_url":"https://api.github.com/users/erikh/repos","events_url":"https://api.github.com/users/erikh/events{/privacy}","received_events_url":"https://api.github.com/users/erikh/received_events","type":"User","site_admin":false},"created_at":"2015-01-12T18:15:46Z","updated_at":"2015-01-12T18:15:46Z","body":"Hi folks. I'm going to attempt to answer some of your questions; I doubt I will get to all of them, so please remind me if I missed a concern you have!\r\n\r\nAdditionally, I'm not the designer here -- that belongs to @shykes and @bfirsh  --  but I'll answer where appropriate. I **have** however taken liberal notes to bring up to them in a design meeting this or next week. Additionally some of the responses are preemptive and still require a response from these stakeholders. I really some of these probably come off as non-answers. Everything on this list (and more) will be brought up with them and I'll provide answers as I get them.\r\n\r\n* For now, at least, advanced multi-tenancy features will be driver-dependent. We are not prescribing any advanced features for many of the reasons @thockin already mentioned, but principally because we're trying to keep scope down in the prototype/v1.\r\n* Flag handling is still up in the air. I've scheduled a meeting with the responsible parties and hope to get back to you all soon.\r\n* I don't see the problem with `ListEndpoints()` or moving `List()` to Driver. Enumerating endpoints might be more problematic (read: slow) because of the way the data structures are currently organized, but I can look into it.\r\n* I will be updating the proposal with clearer descriptions of Network and Endpoint. I will post again when this is done.\r\n* I want network configs beyond flags (as someone has asked for), but I need to get approval for this.\r\n* `Endpoint.Expose` should really be `Endpoint.Publish` and not take the boolean argument. Currently we have no plans to support links (as in, the `docker run` kind) moving forward and are just going to prescribe that the network is trusted. If you wish to implement advanced firewalling, you can either `iptables` in the container or define this behavior in your driver.\r\n* I'll get back to all of you on the notion of `Driver.Link` and `Network.Link`. I think this is probably a side-effect of the prototype and probably shouldn't be in the API this way.\r\n"}}