{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17959","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17959/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17959/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17959/events","html_url":"https://github.com/kubernetes/kubernetes/issues/17959","id":119538133,"number":17959,"title":"Kubelet using expensive /containers/json?all=1 call","user":{"login":"huang195","id":8008677,"avatar_url":"https://avatars.githubusercontent.com/u/8008677?v=3","gravatar_id":"","url":"https://api.github.com/users/huang195","html_url":"https://github.com/huang195","followers_url":"https://api.github.com/users/huang195/followers","following_url":"https://api.github.com/users/huang195/following{/other_user}","gists_url":"https://api.github.com/users/huang195/gists{/gist_id}","starred_url":"https://api.github.com/users/huang195/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/huang195/subscriptions","organizations_url":"https://api.github.com/users/huang195/orgs","repos_url":"https://api.github.com/users/huang195/repos","events_url":"https://api.github.com/users/huang195/events{/privacy}","received_events_url":"https://api.github.com/users/huang195/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/performance","name":"area/performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-11-30T17:18:24Z","updated_at":"2015-12-01T22:37:00Z","closed_at":"2015-11-30T19:24:44Z","body":"We've been trying to squeeze a large number of pods onto a single node well beyond the recommended 30 pods per node. We first ran into this issue (#16296), where cadvisor was taking a huge chunk of cpu time. After decreasing the cadvisor run frequency and @vishh fix on cadvisor's networking parsing, cadvisor's cpu usage went down significantly to allow us to squeeze more containers onto a node. \r\n\r\nAfter more experiments, we found the next bottleneck seems to be Docker engine as it was getting hit very hard from several K8s components. More profiling indicates that things converge to this function:\r\n```\r\nfunc (dm *DockerManager) GetPodStatus(pod *api.Pod) (*api.PodStatus, error)\r\n```\r\nWithin this function, the main culprit is:\r\n```\r\ncontainers, err := dm.client.ListContainers(docker.ListContainersOptions{All: true})\r\n```\r\nbecause when you have a large number of containers, it takes a lot of time for Docker to encode all the information to json, and then for k8s to decode it, i.e., a lot of `/containers/json?all=1` calls. As k8s already uses docker labels (specifically \"io.kubernetes.pod.name\") to annotate which containers belong to which pod, it would be less expensive by calling `/containers/json?filter=...` instead, so there's less encoding/decoding needed. I modified this code to:\r\n```\r\n    namespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name}\r\n    filter := fmt.Sprint(kubernetesNameLabel, \"=\", namespacedName.String())\r\n    labels := map[string][]string{\r\n        \"label\": {filter},\r\n    }\r\n    containers, err := dm.client.ListContainers(docker.ListContainersOptions{All: false, Filters: labels})\r\n```\r\n\r\nIs there any downside of doing this? Or corner cases where it won't work? \r\n\r\ncc @yujuhong since you were looking at k8s scalability issues"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/161118930","html_url":"https://github.com/kubernetes/kubernetes/issues/17959#issuecomment-161118930","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17959","id":161118930,"user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T22:37:00Z","updated_at":"2015-12-01T22:37:00Z","body":"> @vishh Just tried it after disabling cadvisor, and it looks like it is scaling better than before. It started to take a minute or more to create more pods when there are around 150 pods on the node, whereas in release-1.1, it was around 120 pods per node before it started to take more than a minute to create pods.\r\n\r\n@huang195, did you create the pod after the 150 pods have reached steady state? How did you measure the latency?"}}