{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18266","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18266/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18266/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18266/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18266","id":120683976,"number":18266,"title":"when increase kube-api-qps param in kube-controller-mananger, etcd2.2.2 out of memory","user":{"login":"magicwang-cn","id":7587313,"avatar_url":"https://avatars.githubusercontent.com/u/7587313?v=3","gravatar_id":"","url":"https://api.github.com/users/magicwang-cn","html_url":"https://github.com/magicwang-cn","followers_url":"https://api.github.com/users/magicwang-cn/followers","following_url":"https://api.github.com/users/magicwang-cn/following{/other_user}","gists_url":"https://api.github.com/users/magicwang-cn/gists{/gist_id}","starred_url":"https://api.github.com/users/magicwang-cn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/magicwang-cn/subscriptions","organizations_url":"https://api.github.com/users/magicwang-cn/orgs","repos_url":"https://api.github.com/users/magicwang-cn/repos","events_url":"https://api.github.com/users/magicwang-cn/events{/privacy}","received_events_url":"https://api.github.com/users/magicwang-cn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-12-07T02:10:37Z","updated_at":"2015-12-07T09:41:16Z","closed_at":null,"body":"@wojtek-t \r\ni have created a k8s cluster with 400 nodes, 16000pods[40 pods per node] and use a 5 nodes etcd cluster[with 8G mem, 4 cpus]. and i tested for e2e density, found an etcd problem[OOM], this is the details:\r\n\r\nfirst, definitely, this issue is cased by a param of kube-controller-manager(--kube-api-qps, from 20 to 100, and this can increase the rate of pods add/delete operation)\r\nsecond, also my doubt, batch creating pods[create a rc with 16000 replicas] will not occur this problem, but batch deleting pods[scale the rc with 0 replicas] does(they both use etcd watch method).\r\nfinal, it  might be a misuse of etcd?\r\n\r\ndetails:\r\nin the process of deploying 16000 pods will not occur this problem, and the etcd mem use just about 800~900M(total 8G), the etcd snapshort is just about 46M\r\nwhen i batch deleting pods , the etcd mem use increasing straight up, then OOM.\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/162464846","html_url":"https://github.com/kubernetes/kubernetes/issues/18266#issuecomment-162464846","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18266","id":162464846,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2015-12-07T09:41:15Z","updated_at":"2015-12-07T09:41:15Z","body":"I don't think it's related to watch, because there is no difference between create & delete \"event\" from the watch perspective..."}}