{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5761","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5761/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5761/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5761/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/5761","id":63457831,"number":5761,"title":"Scalability and Soak Test Experiments: serve_hostname","user":{"login":"satnam6502","id":6164339,"avatar_url":"https://avatars.githubusercontent.com/u/6164339?v=3","gravatar_id":"","url":"https://api.github.com/users/satnam6502","html_url":"https://github.com/satnam6502","followers_url":"https://api.github.com/users/satnam6502/followers","following_url":"https://api.github.com/users/satnam6502/following{/other_user}","gists_url":"https://api.github.com/users/satnam6502/gists{/gist_id}","starred_url":"https://api.github.com/users/satnam6502/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/satnam6502/subscriptions","organizations_url":"https://api.github.com/users/satnam6502/orgs","repos_url":"https://api.github.com/users/satnam6502/repos","events_url":"https://api.github.com/users/satnam6502/events{/privacy}","received_events_url":"https://api.github.com/users/satnam6502/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/performance","name":"area/performance","color":"fef2c0"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/reliability","name":"area/reliability","color":"a10d01"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/misc/perf-1.0","name":"misc/perf-1.0","color":"006b75"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P2","name":"priority/P2","color":"fbca04"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/testing","name":"team/testing","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-03-21T21:49:13Z","updated_at":"2015-03-24T07:31:35Z","closed_at":null,"body":"I am creating this issue so we can track the results of the soak and scalability experiments I am doing with the soak test that I wrote in PR #5750.\r\n\r\nFor this batch of resulted I created a 50 node GCE cluster.\r\nFirst I run serve_hostnames with 1 pod per node and 1,000 queries. This worked fine with each pod (on each node) returning a decent number of queries (i.e. well balanced).\r\n\r\nThen when I ran the experiment again with 30 pods per node and 10,000 queries I noticed that for the 17th node an attempt to create the 25th pod failed. You can see the complete log at this Gist: https://gist.github.com/satnam6502/ec4f5afb3b5df6819fa1 and also the output of the working first case.\r\n\r\n@davidopp @roberthbailey @piosz @wojtek-t \r\n\r\n```\r\n$ ./serve_hostnames --per_node=30 --queries=10000 --up_to=100 2>&1 | tee hostnames-1.log\r\n...\r\nI0321 21:37:22.915744   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-5 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:24.392571   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-6 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:24.832058   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-7 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:25.653823   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-8 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:27.863089   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-9 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:30.712703   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-10 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:31.109360   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-11 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:32.092721   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-12 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:33.221556   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-13 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:35.612905   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-14 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:38.408803   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-15 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:39.837127   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-16 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:41.694377   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-17 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:42.221357   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-18 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:44.901134   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-19 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:48.700996   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-20 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:51.066495   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-21 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:53.249759   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-22 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:54.612782   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-23 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nI0321 21:37:57.354708   22897 serve_hostnames.go:123] Creating pod serve-hostnames-3537/serve-hostname-16-24 on node kubernetes-minion-95vc.c.kubernetes-scale.internal\r\nW0321 21:37:58.509195   22897 serve_hostnames.go:143] Failed to create pod serve-hostname-16-24\r\nI0321 21:37:58.509235   22897 serve_hostnames.go:111] Cleaning up service\r\n```"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/85375565","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/5761#issuecomment-85375565","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/5761","id":85375565,"user":{"login":"gmarek","id":1703842,"avatar_url":"https://avatars.githubusercontent.com/u/1703842?v=3","gravatar_id":"","url":"https://api.github.com/users/gmarek","html_url":"https://github.com/gmarek","followers_url":"https://api.github.com/users/gmarek/followers","following_url":"https://api.github.com/users/gmarek/following{/other_user}","gists_url":"https://api.github.com/users/gmarek/gists{/gist_id}","starred_url":"https://api.github.com/users/gmarek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gmarek/subscriptions","organizations_url":"https://api.github.com/users/gmarek/orgs","repos_url":"https://api.github.com/users/gmarek/repos","events_url":"https://api.github.com/users/gmarek/events{/privacy}","received_events_url":"https://api.github.com/users/gmarek/received_events","type":"User","site_admin":false},"created_at":"2015-03-24T07:31:35Z","updated_at":"2015-03-24T07:31:35Z","body":"That I agree, but you're running 50 nodes right now, which is noticeable amount. I guess 4 thread master, which is 1 master thread per 12 nodes should be acceptable. Reason why I'm suggesting this, is 1.0 milestone, that 99%-ile of API calls should return in <1 sec (#4521), which I think would be really hard for cluster composed of 50 minions and 1 single threaded master (and we're far from it right now).\r\n\r\n@bgrant0607 @thockin @brendanburns - what do you think of assuming assuming single threaded minions (1 thread per 10 nodes)?"}}