{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3579","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3579/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3579/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3579/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3579","id":54630368,"number":3579,"title":"Proposal for Deferred Creation of Addon Objects","user":{"login":"zmerlynn","id":4942464,"avatar_url":"https://avatars.githubusercontent.com/u/4942464?v=3","gravatar_id":"","url":"https://api.github.com/users/zmerlynn","html_url":"https://github.com/zmerlynn","followers_url":"https://api.github.com/users/zmerlynn/followers","following_url":"https://api.github.com/users/zmerlynn/following{/other_user}","gists_url":"https://api.github.com/users/zmerlynn/gists{/gist_id}","starred_url":"https://api.github.com/users/zmerlynn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zmerlynn/subscriptions","organizations_url":"https://api.github.com/users/zmerlynn/orgs","repos_url":"https://api.github.com/users/zmerlynn/repos","events_url":"https://api.github.com/users/zmerlynn/events{/privacy}","received_events_url":"https://api.github.com/users/zmerlynn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-01-16T22:00:57Z","updated_at":"2015-01-20T18:41:52Z","closed_at":null,"body":"# Deferred Creation of \"Addon Objects\"\r\n\r\nIn PRs #2224 and #3292, we added a considerable amount of initialization of state in `kube-up.sh` after the actual kube-up call. This pattern really doesn't work very well for GKE: In a GKE, kube-up is essentially a single API call that can be made from a variety of clients that aren't `kube-up.sh`.\r\n\r\nThis bug proposes a way to pieces of the initialization in `kube-up.sh` onto the master, so that, aspirationally, `kube-up.sh` results in a single call to a provider-specific `kube-up`. It is split into phases, though, because some things, such as firewall rules, won't be handled from the master for some time.\r\n\r\n## Phase 1: Cheesy `init.d` script on the master to create objects\r\n\r\nIn this phase, we move (some) initialization to an `init.d` script on the master. The actual YAML files for the created objects are distributed in salt tarball. We add a tiny `/etc/init.d` script via salt to the master that essentially does the exact same logic that that `kube-up.sh` was doing, with the guiding environment variables plumbed through appropriately to salt on the master (e.g. `ENABLE_CLUSTER_DNS`, `ENABLE_CLUSTER_MONITORING`, etc.). The only complexity for this script is that it needs to wait for the apiserver to be ready before it starts creating objects.\r\n\r\nNon-goal: Right now, monitoring and logging both have to use firewall rules. In the fullness of time, this rule could also be maintained by the cluster rather than being an extra-cluster resource that the\r\nclient needs to maintain. In phase 1 in GCE, we still require the `monitoring-heapster` rule for monitoring and the `fluentd-elasticsearch-logging` rule for logging. These are smells, though, and should be maintained by the master, which already has appropriate scopes (even in GCE). After this phase, though, SkyDNS would work after cluster creation on GKE without any special work, and monitoring and logging could both be enabled with a minor tweak to firewalls.\r\n\r\n## Phase 2: Move deferred addon objects to a container\r\n\r\nRather than having an `init.d` script handle this, load the deferred installation into a container. This has the added complexity of needing a `kubectl` that can auth to the master (so possibly cheat and use `hostDir` to steal the kubelet's creds), but it makes this piece robust. Turn this from a silly script into a bit more of a daemon that actually enforces these \"system containers\" are present at any given time.\r\n\r\nThe main benefits of a container are:\r\n* Resilience - The script may be finnicky to apiserver instability.\r\n* Handling configuration change - The init.d script could handle this, but especially given previous point, it's probably better as a real daemon.\r\n* Disconnect from salt - We can just pass env variables in to the pod. Initially these would be from salt, but they don't have to be. \r\n\r\nThe complexity is primarily high around the build, hence phase 1.\r\n\r\n## Phase 3: Incorporate firewall rules, etc.\r\n\r\nThis is the all singing, all dancing pony. Move everything off `kube-up.sh` and have the master control firewall rules. Handwavy at this point.\r\n\r\n<hr>\r\nThis proposal is meant to address #3305 and follow on issues from pr #3292. I hope to have the phase 1 work complete soon, so comments appreciated.\r\n\r\ncc @thockin @satnam6502 @vishh @brendanburns \r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/70707833","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3579#issuecomment-70707833","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3579","id":70707833,"user":{"login":"erictune","id":8105590,"avatar_url":"https://avatars.githubusercontent.com/u/8105590?v=3","gravatar_id":"","url":"https://api.github.com/users/erictune","html_url":"https://github.com/erictune","followers_url":"https://api.github.com/users/erictune/followers","following_url":"https://api.github.com/users/erictune/following{/other_user}","gists_url":"https://api.github.com/users/erictune/gists{/gist_id}","starred_url":"https://api.github.com/users/erictune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erictune/subscriptions","organizations_url":"https://api.github.com/users/erictune/orgs","repos_url":"https://api.github.com/users/erictune/repos","events_url":"https://api.github.com/users/erictune/events{/privacy}","received_events_url":"https://api.github.com/users/erictune/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T18:41:52Z","updated_at":"2015-01-20T18:41:52Z","body":"@zmerlynn is there an issue about services creating firewalls, and if not\r\nwould you feel up to filing one?\r\n\r\nOn Tue, Jan 20, 2015 at 10:40 AM, Vish Kannan <notifications@github.com>\r\nwrote:\r\n\r\n> @erictune <https://github.com/erictune>: A service for 'heapster' itself\r\n> will be added soon which will most likely not need a firewall rule. There\r\n> are two other services for monitoring that require firewall rules. We can\r\n> get rid of these firewall rules once the proxy on the apiserver works for\r\n> Grafana.\r\n> The firewall rules are required to enable access to hostPort's of a Pod.\r\n> If we were to infer them, we would have to do that from the Pod Spec and\r\n> not the service spec.\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/GoogleCloudPlatform/kubernetes/issues/3579#issuecomment-70707460>\r\n> .\r\n>"}}