{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3604","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3604/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3604/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3604/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3604","id":54729148,"number":3604,"title":"Kubernetes architecture in AWS VPC","user":{"login":"rgolg","id":7472046,"avatar_url":"https://avatars.githubusercontent.com/u/7472046?v=3","gravatar_id":"","url":"https://api.github.com/users/rgolg","html_url":"https://github.com/rgolg","followers_url":"https://api.github.com/users/rgolg/followers","following_url":"https://api.github.com/users/rgolg/following{/other_user}","gists_url":"https://api.github.com/users/rgolg/gists{/gist_id}","starred_url":"https://api.github.com/users/rgolg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rgolg/subscriptions","organizations_url":"https://api.github.com/users/rgolg/orgs","repos_url":"https://api.github.com/users/rgolg/repos","events_url":"https://api.github.com/users/rgolg/events{/privacy}","received_events_url":"https://api.github.com/users/rgolg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-01-19T06:02:22Z","updated_at":"2015-01-21T21:00:23Z","closed_at":null,"body":"Hello Kubernetes community,\r\n\r\nHappy to have found my way here. I'm very interested in making Kubernetes work in an AWS VPC and have looked at the changes @silasdavis has made to the original CloudFormation template for setting up Kubernetes in AWS running on CoreOS (https://github.com/GoogleCloudPlatform/kubernetes/pull/2649) (Thanks @silasdavis!). However, I'm struggling with choosing the best architectural approach.\r\n\r\nLet me preface all this by saying that I feel that I am bringing in baggage from my experience with designing and running immutable infrastructures using AMI instances and other AWS resources in AWS VPCs. Maybe the controls provided within a Kubernetes cluster, along with the controls on the nodes, pods, and containers make the need for VPCs and network partitioning obsolete. So *please* let me know if I should be looking at this differently and there's a better Kubernetes/Container-native approach. \r\n\r\nAlso, if someday AWS decides to support Kubernetes directly with their Elastic Container Service (when it is real...still waiting to get my preview access!) all of this will quickly become obsolete. Frankly, the easiest thing would be if AWS allowed the option for creating a Kubernetes master when deploying a subnet, similar to how an ECS agent can be included in an AMI (based on what I've seen and read about ECS).\r\n\r\nProbably stating the obvious here but my most important objective for this infrastructure architecture is balancing security, performance, scalability, and flexibility.\r\n\r\nThe options I'm considering are:\r\n\r\n* One cluster per subnet in a VPC, with one master autoscaled with min/max/desired number of instances of one, and a set of minions in the same subnet behind an autoscaling group that can possibly be scaled by external monitors triggering CloudWatch alarms for scale up and scale down events. With this setup, at least theoretically, the proxies for the same pods spread across different subnets in different availability zones can be registered to ELBs allowing for load-balancing applications/services in pods across availability zones.\r\n * Pros:\r\n    * Load-balancing applications/services in pods across multiple availability zones.\r\n    * The Kubernetes master communicates with the minions in the same subnet avoiding any networking complexities between the replication controller on the master and the nodes and Kubelets on each pod.\r\n * Cons:\r\n    * Seems like an overkill to run a separate master on each subnet.\r\n* One cluster per security zone: \r\n  \r\n  *Definition:\r\n      * public subnets route to internet through the VPC's internet gateway. Resources in public subnets can be accessed publicly if assigned EIPs.\r\n      * private subnets route to internet through NAT instances in the public subnets. Resources in private subnets cannot be accessed publicly.\r\n      * management subnets are private subnets that can communicate with resources in both public and private subnets. Resources in management subnets cannot be accessed publicly.\r\n      * All resources in public/private/management subnets can communicate with each other. More granular security constraints can be applied with network ACLs and security groups around resources in each of the subnets.\r\n      * public/private/management subnets are deployed in pairs, with one subnet of each pair in one availability zone and the other of the pair in another availability zone.*\r\n  1. Deploy a Kubernetes cluster for the public subnets, with the Kubernetes master deployed to the management subnet and its minions on the public subnets.\r\n  2. Deploy a Kubernetes cluster for the private subnets, with the Kubernetes master deployed to the management subnet and its minions on the private subnets.\r\n\r\n * Pros:\r\n   * Load-balancing applications/services in pods across multiple availability zones.\r\n * Cons:\r\n   * Kubernetes master nodes running in the management subnets may have issues communicating with the minions in the private and public subnets. I'm not sure if there will be any issues here but I have read in a couple of places that its better to keep all nodes of the Kubernetes cluster on the same network. Not sure if that means keeping them on the same subnet as well.\r\n\r\nI'm curious to find out how others are thinking about deploying Kubernetes in AWS and would love to contribute anything I can and work within the community to make it happen.\r\n\r\nThanks,\r\nReza\r\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/70920526","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3604#issuecomment-70920526","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3604","id":70920526,"user":{"login":"satnam6502","id":6164339,"avatar_url":"https://avatars.githubusercontent.com/u/6164339?v=3","gravatar_id":"","url":"https://api.github.com/users/satnam6502","html_url":"https://github.com/satnam6502","followers_url":"https://api.github.com/users/satnam6502/followers","following_url":"https://api.github.com/users/satnam6502/following{/other_user}","gists_url":"https://api.github.com/users/satnam6502/gists{/gist_id}","starred_url":"https://api.github.com/users/satnam6502/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satnam6502/subscriptions","organizations_url":"https://api.github.com/users/satnam6502/orgs","repos_url":"https://api.github.com/users/satnam6502/repos","events_url":"https://api.github.com/users/satnam6502/events{/privacy}","received_events_url":"https://api.github.com/users/satnam6502/received_events","type":"User","site_admin":false},"created_at":"2015-01-21T21:00:23Z","updated_at":"2015-01-21T21:00:23Z","body":"Yes, I did some scalability work up to the end of last year and I hope to pick it up again next week. Basic issues:\r\n* Generating > 255 IP addresses. I used a bash function to allow the generation of 1,000 IP addresses. This was written by @jbeda and I have not checked it in because I am not sure if this is really how we want to do it -- it is in a Gist which I can't find anymore.\r\n* I had to do a lot of work to make cluster bring up and tear down more robust otherwise -- at least for GCE -- there was no change of successfully creating a large cluster. This process now seems pretty solid for GCE but I imagine similar things would need to be done for other providers. For example, any GCE operation can fail for unknown reasons -- so I added a lot of re-try logic.\r\n* We still need to work on basic Kubernetes things for scalability e.g. detecting / listing large numbers of minions and listing large numbers of pods with appropriate caching. This is the work I hope to turn to next week.\r\n* At this early stage the workloads I have been using have just been lots of pause containers created independently (rather than with a replication controller). Next week I hope to use the kubernetes/serve_hostname:1.0 image with a replication controller and hostPort to ensure even balancing across all nodes. I would then instantiate several of these (with different host ports). The objective is to have 1,000 nodes working with 10,000 containers.\r\n\r\n"}}