{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14702","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14702/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14702/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14702/events","html_url":"https://github.com/kubernetes/kubernetes/issues/14702","id":108779107,"number":14702,"title":"Audit and evaluate apiserver client usage in kubelet","user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-29T01:43:23Z","updated_at":"2015-09-29T15:00:03Z","closed_at":null,"body":"The current apiserver client QPS for kubelet is the default 5.0, which is not high for kubelet. When batch creating pods, the startup latency is completely dominated by the QPS (e.g. see https://github.com/kubernetes/kubernetes/issues/14554#issuecomment-143829191, events consumed most of the bandwidth). Moreover, the *perceived* kubelet performance may be affected by slight change in kubelet's behavior, due to the QPS limitation (e.g., https://github.com/kubernetes/kubernetes/issues/14554#issuecomment-143898216).\r\n\r\nWe should audit our apiserver client usage and re-evaluate whether to bump the QPS, or to clearly surface any increase in the client usage.\r\n\r\nMajor consumers for the apiserver client includes:\r\n * events\r\n * pod status\r\n * node status (relatively low)\r\n\r\nThe breakdown of starting a pod of N containers (when there is no failure):\r\n * (N+1) * 3 events  (for each container, including the infra container, there are pulled, created, and started events)\r\n * 3 pod status update (the initial pending, running, and finally the running+ready status). This may increase once we let user control per-container probing period.\r\n\r\nFor starting 30 single-container pods on kubelet, the pods along would generate ((1+1)\\*3+3)\\*30= 9\\*30=270 requests.\r\nThis would take kubelet 270/5 qps = 54 seconds to send out.\r\n\r\n/cc @kubernetes/goog-node "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/144086763","html_url":"https://github.com/kubernetes/kubernetes/issues/14702#issuecomment-144086763","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14702","id":144086763,"user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"created_at":"2015-09-29T15:00:03Z","updated_at":"2015-09-29T15:00:03Z","body":"@yujuhong - yes - I'm looking into it - I have an extremely hacky prototype here:\r\nhttps://github.com/kubernetes/kubernetes/compare/master...wojtek-t:separate_events_etcd\r\nthat I used to verify performance impact of it. It seems to be pretty significant in large cluster, so I will be working on making it reviewable and mergable this week."}}