{"action":"created","comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/34180949","id":34180949,"diff_hunk":"@@ -0,0 +1,107 @@\n+# Kubernetes Large Cluster\n+\n+## Support\n+At v1.0, Kubernetes supports clusters up to 100 nodes with 30-50 pods per node and 1-2 container per pod (as defined in the [1.0 roadmap](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/roadmap.md#reliability-and-performance)).\n+\n+## Setup\n+\n+Normally the number of nodes in a cluster is controlled by the the value `NUM_MINIONS` in the platform-specific `config-default.sh` file (for example, see [GCE's `config-default.sh`](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/gce/config-default.sh)).\n+\n+Simply changing that value to something very large, however, may cause the setup script to fail for many cloud providers. A GCE deployment, for example, will run in to quota issues and fail to bring the cluster up.\n+\n+When setting up a large Kubernetes cluster, the following must be taken into consideration.\n+\n+### Quota Issues\n+\n+To avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:\n+* Increase the quota for things like CPU, IPs, etc.\n+  * In [GCE, for example,](https://cloud.google.com/compute/docs/resource-quotas) you'll want to increase the quota for:\n+    * CPUs\n+    * VM instances\n+    * Total persistent disk reserved\n+    * In-use IP addresses\n+    * Firewall Rules\n+    * Forwarding rules\n+    * Routes\n+    * Target pools\n+* Gating the setup script so that it brings up new node VMs in smaller batches with waits in between, because some cloud providers limit the number of VMs you can create during a given period.\n+\n+### Addon Resources\n+To prevent memory leaks or other resource issues in [cluster addons](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/) from consuming all the resources available on a node, Kubernetes sets resource limits on addon containers to limit the CPU and Memory resources they can consume (See PR [#10653](https://github.com/GoogleCloudPlatform/kubernetes/pull/10653/files) and [#10778](https://github.com/GoogleCloudPlatform/kubernetes/pull/10778/files)).\n+\n+For example:\n+```YAML\n+      containers:\n+        - image: gcr.io/google_containers/heapster:v0.15.0\n+          name: heapster\n+          resources:\n+            limits:\n+              cpu: 100m\n+              memory: 200Mi\n+```\n+\n+These limits, however, are based on data collected from addons running on 4-node clusters (see [#10335](https://github.com/GoogleCloudPlatform/kubernetes/issues/10335#issuecomment-117861225)). The addons consume a lot more resources when running on large deployment clusters (see [#5880](https://github.com/GoogleCloudPlatform/kubernetes/issues/5880#issuecomment-113984085)). So, if a large cluster is deployed without adjusting these values, the addons may continuously get killed because they keep hitting the limits.\n+\n+To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:\n+* Scale memory and CPU limits for each of the following addons, if used, along with the size of cluster (there is one replica of each handling the entire cluster so memory and CPU usage tends to grow proportionally with size/load on cluster):\n+  * Heapster ([GCM/GCL backed](cluster/addons/cluster-monitoring/google/heapster-controller.yaml), [InfluxDB backed](cluster/addons/cluster-monitoring/influxdb/heapster-controller.yaml), [GCM/InfluxDB](cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml), [standalone](cluster/addons/cluster-monitoring/standalone/heapster-controller.yaml))\n+  * [InfluxDB and Grafana](cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml)\n+  * [skydns, kube2sky, and dns etcd](cluster/addons/dns/skydns-rc.yaml.in)\n+  * [Kibana](cluster/addons/fluentd-elasticsearch/kibana-controller.yaml)\n+* Scale number of replicas for the following addons, if used, along with the size of cluster (there are multiple replicas of each so increasing replicas should help handle increased load, but, since load per replica also increases slightly, also consider increasing CPU/memory limits):\n+  * [elasticsearch](cluster/addons/fluentd-elasticsearch/es-controller.yaml)\n+* Increase memory and CPU limits sligthly for each of the following addons, if used, along with the size of cluster (there is one replica per node but CPU/memory usage increases slightly along with cluster load/size as well):\n+  * [FluentD with ElasticSearch Plugin](cluster/saltbase/salt/fluentd-es/fluentd-es.yaml)\n+  * [FluentD with GCP Plugin](cluster/saltbase/salt/fluentd-gcp/fluentd-gcp.yaml)\n+\n+\n+#### Detecting Resource Starved Containers","path":"docs/cluster-large.md","position":58,"original_position":58,"commit_id":"0c104cd1a74c8088aca3366504281952f6f6433a","original_commit_id":"0c104cd1a74c8088aca3366504281952f6f6433a","user":{"login":"erictune","id":8105590,"avatar_url":"https://avatars.githubusercontent.com/u/8105590?v=3","gravatar_id":"","url":"https://api.github.com/users/erictune","html_url":"https://github.com/erictune","followers_url":"https://api.github.com/users/erictune/followers","following_url":"https://api.github.com/users/erictune/following{/other_user}","gists_url":"https://api.github.com/users/erictune/gists{/gist_id}","starred_url":"https://api.github.com/users/erictune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erictune/subscriptions","organizations_url":"https://api.github.com/users/erictune/orgs","repos_url":"https://api.github.com/users/erictune/repos","events_url":"https://api.github.com/users/erictune/events{/privacy}","received_events_url":"https://api.github.com/users/erictune/received_events","type":"User","site_admin":false},"body":"Please move this section to its own document, and link from here, and also link from docs/compute_resources.md.  ","created_at":"2015-07-08T18:22:22Z","updated_at":"2015-07-08T18:22:22Z","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804#discussion_r34180949","pull_request_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804","_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments/34180949"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804#discussion_r34180949"},"pull_request":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804"}}},"pull_request":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804","id":39330291,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804","diff_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804.diff","patch_url":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804.patch","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10804","number":10804,"state":"open","locked":false,"title":"Add documentation for creating large kubernetes clusters","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"body":"Closes #10779 (adds guidance for addon container scaling)\r\n\r\nCC @dchen1107 @satnam6502 ","created_at":"2015-07-07T02:42:30Z","updated_at":"2015-07-08T18:22:22Z","closed_at":null,"merged_at":null,"merge_commit_sha":"1cbc5465b228e7d3950a89c38bca021180b8449d","assignee":{"login":"satnam6502","id":6164339,"avatar_url":"https://avatars.githubusercontent.com/u/6164339?v=3","gravatar_id":"","url":"https://api.github.com/users/satnam6502","html_url":"https://github.com/satnam6502","followers_url":"https://api.github.com/users/satnam6502/followers","following_url":"https://api.github.com/users/satnam6502/following{/other_user}","gists_url":"https://api.github.com/users/satnam6502/gists{/gist_id}","starred_url":"https://api.github.com/users/satnam6502/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satnam6502/subscriptions","organizations_url":"https://api.github.com/users/satnam6502/orgs","repos_url":"https://api.github.com/users/satnam6502/repos","events_url":"https://api.github.com/users/satnam6502/events{/privacy}","received_events_url":"https://api.github.com/users/satnam6502/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804/commits","review_comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804/comments","review_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10804/comments","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/0c104cd1a74c8088aca3366504281952f6f6433a","head":{"label":"saad-ali:clusterScaleDoc","ref":"clusterScaleDoc","sha":"0c104cd1a74c8088aca3366504281952f6f6433a","user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"repo":{"id":27614790,"name":"kubernetes","full_name":"saad-ali/kubernetes","owner":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/saad-ali/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/saad-ali/kubernetes","forks_url":"https://api.github.com/repos/saad-ali/kubernetes/forks","keys_url":"https://api.github.com/repos/saad-ali/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/saad-ali/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/saad-ali/kubernetes/teams","hooks_url":"https://api.github.com/repos/saad-ali/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/saad-ali/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/saad-ali/kubernetes/events","assignees_url":"https://api.github.com/repos/saad-ali/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/saad-ali/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/saad-ali/kubernetes/tags","blobs_url":"https://api.github.com/repos/saad-ali/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/saad-ali/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/saad-ali/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/saad-ali/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/saad-ali/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/saad-ali/kubernetes/languages","stargazers_url":"https://api.github.com/repos/saad-ali/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/saad-ali/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/saad-ali/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/saad-ali/kubernetes/subscription","commits_url":"https://api.github.com/repos/saad-ali/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/saad-ali/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/saad-ali/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/saad-ali/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/saad-ali/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/saad-ali/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/saad-ali/kubernetes/merges","archive_url":"https://api.github.com/repos/saad-ali/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/saad-ali/kubernetes/downloads","issues_url":"https://api.github.com/repos/saad-ali/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/saad-ali/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/saad-ali/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/saad-ali/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/saad-ali/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/saad-ali/kubernetes/releases{/id}","created_at":"2014-12-05T23:31:31Z","updated_at":"2015-05-07T05:23:04Z","pushed_at":"2015-07-08T00:35:40Z","git_url":"git://github.com/saad-ali/kubernetes.git","ssh_url":"git@github.com:saad-ali/kubernetes.git","clone_url":"https://github.com/saad-ali/kubernetes.git","svn_url":"https://github.com/saad-ali/kubernetes","homepage":null,"size":87222,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"GoogleCloudPlatform:master","ref":"master","sha":"af906eedd94c36d97762577221d1b33d5326e561","user":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"GoogleCloudPlatform/kubernetes","owner":{"login":"GoogleCloudPlatform","id":2810941,"avatar_url":"https://avatars.githubusercontent.com/u/2810941?v=3","gravatar_id":"","url":"https://api.github.com/users/GoogleCloudPlatform","html_url":"https://github.com/GoogleCloudPlatform","followers_url":"https://api.github.com/users/GoogleCloudPlatform/followers","following_url":"https://api.github.com/users/GoogleCloudPlatform/following{/other_user}","gists_url":"https://api.github.com/users/GoogleCloudPlatform/gists{/gist_id}","starred_url":"https://api.github.com/users/GoogleCloudPlatform/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoogleCloudPlatform/subscriptions","organizations_url":"https://api.github.com/users/GoogleCloudPlatform/orgs","repos_url":"https://api.github.com/users/GoogleCloudPlatform/repos","events_url":"https://api.github.com/users/GoogleCloudPlatform/events{/privacy}","received_events_url":"https://api.github.com/users/GoogleCloudPlatform/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoogleCloudPlatform/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes","forks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/forks","keys_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/teams","hooks_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/events","assignees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/tags","blobs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/languages","stargazers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/subscription","commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/merges","archive_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/downloads","issues_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-07-08T16:59:22Z","pushed_at":"2015-07-08T18:17:45Z","git_url":"git://github.com/GoogleCloudPlatform/kubernetes.git","ssh_url":"git@github.com:GoogleCloudPlatform/kubernetes.git","clone_url":"https://github.com/GoogleCloudPlatform/kubernetes.git","svn_url":"https://github.com/GoogleCloudPlatform/kubernetes","homepage":"http://kubernetes.io","size":307267,"stargazers_count":8774,"watchers_count":8774,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":1965,"mirror_url":null,"open_issues_count":1606,"forks":1965,"open_issues":1606,"watchers":8774,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804"},"html":{"href":"https://github.com/GoogleCloudPlatform/kubernetes/pull/10804"},"issue":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10804"},"comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/10804/comments"},"review_comments":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804/comments"},"review_comment":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/pulls/10804/commits"},"statuses":{"href":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/statuses/0c104cd1a74c8088aca3366504281952f6f6433a"}}}}