{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6666","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6666/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6666/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6666/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6666","id":67476052,"number":6666,"title":"Design: DNS revamp","user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind/design","name":"kind/design","color":"50cba5"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P3","name":"priority/P3","color":"fef2c0"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-04-10T00:04:14Z","updated_at":"2015-05-12T02:11:17Z","closed_at":null,"body":"There have been a bunch of discussions about DNS, and I thought it was worthwhile to collect them all.\r\n\r\n@smarterclayton @bgrant0607 @preilly @vishh  \r\n\r\n# Background and current state\r\n\r\nWe currently serve DNS from a pod running in the cluster.  It runs etcd, skydns, and a bridge between them called kube2sky.  The only records we really serve are A records.  We add an A record for every Service object, resolving to the portal IP for that Service.  For example, given a Service named 'mysvc' in namespace 'myns', we serve a DNS A record (a simple address) for `mysvc.myns.kubernetes.local`.\r\n\r\n## SkyDNS schema\r\n\r\nTo implement this we configure SkyDNS, through etcd,  with a single config record per service:\r\n\r\n```\r\n/skydns/local/kubernetes/myns/mysvc =  '{\"host\":\"$PORTAL_IP\", \"port\":$SERVICE_PORT}'\r\n```\r\n\r\n# Motivations for change\r\n\r\n1) We now have \"headless\" services - services that do not have a portal IP.\r\n2) We now have named Service ports and multi-port Services.\r\n3) SkyDNS supports arbitrary backend implementations, eliminating the need to run an etcd instance just for DNS.\r\n4) Several feature requests and PRs that touch on DNS.\r\n\r\n# Goal\r\n\r\nTo evaluate options for designs and decide on one path to implement.\r\n\r\n# Headless services\r\n\r\nThe obvious thing to serve for a headless service is the set of pod IPs that implement the service - a plain old DNS round-robin.  The reason we haven't just done that for \"classic\" services is that many DNS client libraries do not respect DNS TTLs, or applications assume (intentionally or not) that they can resolve a name once and then remember that result forever.  But for headless services, we don't have much choice - we really have to do SOMETHING.\r\n\r\nThe simplest solution, then, is just to resolve service names into multiple A records, pointing to pod IPs, where we would normally resolve to a single portal IP.\r\n\r\n## SkyDNS schema\r\n\r\nHeadless services will need one config record per service-pod.\r\n\r\n```\r\n/skydns/local/kubernetes/myns/mysvc/$POD_NAME =  '{\"host\":\"$POD_IP\", \"port\":$SERVICE_PORT}'\r\n```\r\n\r\nSkyDNS will aggregate the pods under `mysvc` and report them all as A records for `mysvc.myns.kubernetes.local`.  This will also have the side-effect of creating A records for each `$POD_NAME.mysvc.myns.kubernetes.local`.\r\n\r\nThere's a wrinkle, though.  Where do we get pod names and IPs?  The DNS code will have to do more than just watch Services (which is all it does today).  It will have to also watch Endpoints.  When it observes a headless Service, it will look at the Endpoints for that Service and extract the endpoint IPs.\r\n\r\nThe last question here is what to use as the pod name in the above schema?  I see two options.\r\n\r\n1) Use the `EndpointAddress.TargetRef` field.  For example, a TargetRef that points to `{Kind: \"Pod\", Name: \"p1\"}` could become \"pod-p1\" or even just \"p1\".  This has the nice effect of producing A records that match kubernetes objects.  We have to be careful about name length (DNS says 63 characters per segment) and we have to handle dots (we allow names with dots, but they have special meaning in DNS) and we have to possibly handle collisions (if we just use Name instead of Kind-Name) and we still have to handle EndpointAdress objects without a TargetRef (it is optional).\r\n\r\n2) Generate random names.  Instead of `mypod.mysvc.myns.kubernetes.local` we would get `ag54d.mysvc.myns.kubernetes.local`, but avoid all of the concerns above.  They don't even have to be very random - just unique within the service.\r\n\r\n## Consistency\r\n\r\nThe followup to this idea is that we should probably offer the same expansion for Endpoints under a classic Service, to enable things like DNS-SD for Prometheus.  But where?  Unfortunately, we did not leave any room in the DNS space to extend.  For any value of `*.mysvc.myns.kubernetes.local`, SkyDNS will do the wrong thing and expose multiple A records for `mysvc.myns.kubernetes.local`.  Where this behavior helps us with headless services it hurts with regular services.\r\n\r\nOptions:\r\n\r\n1a) Inject a level in the hierarchy.  What was `mysvc.myns.kubernetes.local` becomes `mysvc.service.myns.kubernetes.local` or `mysvc.portal.myns.kubernetes.local`.  Now we can expose the endpoints directly as `mysvc.endpoints.myns.kubernetes.local`.  We can probably hide the transition by making the DNS search path be `service.myns.kubernetes.local myns.kubernetes.local kubernetes.local`.  This allows namespace local services to be resolved with a single token. It makes cross-namespace names a bit uglier: `kubernetes.default` now becomes `kubernetes.service.default`.\r\n\r\n1b) Inject a level in the hierarchy.  What was `mysvc.myns.kubernetes.local` becomes `mysvc.myns.service.kubernetes.local` or `mysvc.myns.portal.kubernetes.local`.  Now we can expose the endpoints directly as `mysvc.myns.endpoints.kubernetes.local`.  We can probably hide the transition by making the DNS search path be `myns.service.kubernetes.local service.kubernetes.local kubernetes.local`.  This allows namespace local services to be resolved with a single token and makes cross-namespace names the same as now: `kubernetes.default` now resolves to  `kubernetes.default.service.kubernetes.local`, while allowing `kubernetes.default.endpoints`.\r\n\r\n2) Insert a level of name.  What was `mysvc.myns.kubernetes.local` becomes `portal.mysvc.myns.kubernetes.local`.  The leaves `endpoints.mysvc.myns.kubernetes.local` available for the endpoints A records.  Accessing the master service becomes `portal.kubernetes.default`.\r\n\r\n3) Lobby SkyDNS for some feature to do what we need.  Other users have asked for similar things. (https://github.com/skynetservices/skydns/issues/147)\r\n\r\n4) Lobby SkyDNS to support multiple first-level domains and distinguish pods and services at that scope.  `mysvc.myns.kubernetes.local` remains while `mysvc.myns.kube-endpoints.local` is available for the endpoints A records. (https://github.com/skynetservices/skydns/issues/47)\r\n\r\n4) Fork SkyDNS and build our own DNS server.  The skydns codebase could largely be reused.\r\n\r\n# Multiport services and named ports\r\n\r\nGiving ports names pretty much begs for proper DNS SRV records (services).  we can support this for both headless and \"classic\" services.  We should follow the RFC for SRV records, for maximum impact.  SRV records will only be generated when a port has an explicit name.\r\n\r\n## SkyDNS schema\r\n\r\nWe will need one config record per port.\r\n\r\n```\r\n/skydns/local/kubernetes/myns/mysvc/_$PORT_PROTOCOL/_$PORT_NAME/$POD_NAME '{\"host\":\"$POD_NAME.svc.ns.kubernetes.local\", \"port\":$PORT}'\r\n```\r\n\r\nThis will create an SRV record for `_$PORT_NAME._$PORT_PROTOCOL.svc.ns.kubernetes.local` on `$PORT`.  For example: `_http._tcp.svc.ns.kubernetes.local`.  This is RFC compliant.\r\n\r\nWe will get the port name, protocol, and number from the Endpoints object in the case of headless Services and from the Service object in case of \"classic\" services.  We have the same issue with pod name as above and will use the same solution for headless Services, while classic Services will just use the literal \"portal\".\r\n\r\n# SkyDNS backends\r\n\r\nThe groundwork has been laid to implement SkyDNS on top of kubernetes directly, without etcd as an intermediary.  I'd like to make this change, but only AFTER the schema changes are implemented.  Once that is done, we can consider putting the DNS serving logic into each kube-proxy, since it is already caching all the information we need to serve DNS.  This does have the downside that kube-proxy is not an HA component, whereas a DNS pod could be.\r\n\r\n# Other open issues\r\n\r\nThis does not address nominal services.  I will update this as I process the current state of that issue.  I think this schema will address the DNS-SD (service discovery) needs of systems like Prometheus in the headless case.  In the classic Services case, we need to get raw endpoints.\r\n\r\n# Appendix\r\n\r\nThese are two scripts I worked up to manual test the schema against running SkyDNS.  They assumes you are running on the node that is running the DNS pod.\r\n\r\n```\r\n#!/bin/bash\r\n\r\nfunction d {\r\ndocker exec -ti $(docker ps | grep etcd | cut -f1 -d' ') \"$@\"\r\n}\r\n\r\necho multiport headless service with DNSRR\r\nd /etcdctl rm --recursive /skydns/local/kubernetes/ns\r\n\r\n# Given a headless Service 'svc', with 3 Endpoints, only 2 with Podrefs, each with 2 named ports and 1 unnamed port\r\n\r\n# Create records for the hosts, based on port names.  SkyDNS will infer a base name and produce A and SRV records.\r\n# Generate a random name for the unnamed port\r\nUNKPORT=unknown-port-$RANDOM\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod1/$UNKPORT '{\"host\":\"1.1.1.1\", \"port\":9376}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod1/http '{\"host\":\"1.1.1.1\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod1/https '{\"host\":\"1.1.1.1\", \"port\":443}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod2/$UNKPORT '{\"host\":\"2.2.2.2\", \"port\":9376}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod2/http '{\"host\":\"2.2.2.2\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/pod2/https '{\"host\":\"2.2.2.2\", \"port\":443}'\r\n# Generate a random name for the un-referenced pod\r\nUNKPOD=unknown-pod-$RANDOM\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/$UNKPOD/$UNKPORT '{\"host\":\"3.3.3.3\", \"port\":9376}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/$UNKPOD/http '{\"host\":\"3.3.3.3\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/$UNKPOD/https '{\"host\":\"3.3.3.3\", \"port\":443}'\r\n\r\n# Create explicit SRV respords for the RFC-style service names.  SkyDNS does not recurse through records with leading underscores.\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_http/pod1 '{\"host\":\"pod1.svc.ns.kubernetes.local\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_https/pod1 '{\"host\":\"pod1.svc.ns.kubernetes.local\", \"port\":443}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_http/pod2 '{\"host\":\"pod2.svc.ns.kubernetes.local\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_https/pod2 '{\"host\":\"pod2.svc.ns.kubernetes.local\", \"port\":443}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_http/$UNKPOD \"{\\\"host\\\":\\\"$UNKPOD.svc.ns.kubernetes.local\\\", \\\"port\\\":80}\"\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_https/$UNKPOD \"{\\\"host\\\":\\\"$UNKPOD.svc.ns.kubernetes.local\\\", \\\"port\\\":443}\"\r\n\r\n# Expect multiple A records - one for each IP above\r\ndig @10.0.0.10 +noall +answer +additional +ignore svc.ns.kubernetes.local A\r\nread X\r\n\r\n# Expect one A record for each pod\r\ndig @10.0.0.10 +noall +answer +additional +ignore pod1.svc.ns.kubernetes.local A\r\ndig @10.0.0.10 +noall +answer +additional +ignore pod2.svc.ns.kubernetes.local A\r\nread X\r\n\r\n# Expect one A record for each port.pod\r\ndig @10.0.0.10 +noall +answer +additional +ignore http.pod1.svc.ns.kubernetes.local A\r\ndig @10.0.0.10 +noall +answer +additional +ignore https.pod1.svc.ns.kubernetes.local A\r\ndig @10.0.0.10 +noall +answer +additional +ignore http.pod2.svc.ns.kubernetes.local A\r\ndig @10.0.0.10 +noall +answer +additional +ignore https.pod2.svc.ns.kubernetes.local A\r\nread X\r\n\r\n# Expect one SRV record for each pod for each port\r\ndig @10.0.0.10 +noall +answer +additional +ignore svc.ns.kubernetes.local SRV\r\nread X\r\n\r\n# Expect one SRV record for each pod for each port\r\ndig @10.0.0.10 +noall +answer +additional +ignore _http._tcp.svc.ns.kubernetes.local SRV\r\ndig @10.0.0.10 +noall +answer +additional +ignore _https._tcp.svc.ns.kubernetes.local SRV\r\nread X\r\n```\r\n\r\n```\r\n#!/bin/bash\r\n\r\nfunction d {\r\ndocker exec -ti $(docker ps | grep etcd | cut -f1 -d' ') \"$@\"\r\n}\r\n\r\necho multiport headless service with DNSRR\r\nd /etcdctl rm --recursive /skydns/local/kubernetes/ns\r\n\r\n# Given a Service 'svc' with 2 named ports and 1 unnamed port\r\n\r\n# Create records for the hosts, based on port names.  SkyDNS will infer a base name and produce A and SRV records.\r\n# Generate a random name for the unnamed port\r\nUNKPORT=unknown-port-$RANDOM\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/portal/$UNKPORT '{\"host\":\"1.1.1.1\", \"port\":9376}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/portal/http '{\"host\":\"1.1.1.1\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/portal/https '{\"host\":\"1.1.1.1\", \"port\":443}'\r\n\r\n# Create explicit SRV respords for the RFC-style service names.  SkyDNS does not recurse through records with leading underscores.\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_http/portal '{\"host\":\"svc.ns.kubernetes.local\", \"port\":80}'\r\nd /etcdctl set /skydns/local/kubernetes/ns/svc/_tcp/_https/portal '{\"host\":\"svc.ns.kubernetes.local\", \"port\":443}'\r\n\r\n# Expect one A record\r\ndig @10.0.0.10 +noall +answer +additional +ignore svc.ns.kubernetes.local A\r\nread X\r\n\r\n# Expect one SRV record for each port\r\ndig @10.0.0.10 +noall +answer +additional +ignore svc.ns.kubernetes.local SRV\r\nread X\r\n\r\n# Expect one SRV record for each port\r\ndig @10.0.0.10 +noall +answer +additional +ignore _http._tcp.svc.ns.kubernetes.local SRV\r\ndig @10.0.0.10 +noall +answer +additional +ignore _https._tcp.svc.ns.kubernetes.local SRV\r\nread X\r\n```"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/101099513","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6666#issuecomment-101099513","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6666","id":101099513,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2015-05-12T02:11:17Z","updated_at":"2015-05-12T02:11:17Z","body":"On the topic of default suffix, I think \"*.cluster.local\" is strictly superior to kubernetes.local.  It properly identifies scope (a cluster), is generally descriptive (the local clusters DNS scope). We may not make it required, but it's a nice generic domain, and it has the advantage of separating remote DNS entries injected into the local environment in the future (which one is my cluster?  Oh, the one called cluster.  Which one is remote?)\r\n\r\nOpinions"}}