{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/12691","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/12691/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/12691/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/12691/events","html_url":"https://github.com/kubernetes/kubernetes/issues/12691","id":100883787,"number":12691,"title":"kubelet should only trigger relevant pod workers for pod spec changes","user":{"login":"yujuhong","id":10524058,"avatar_url":"https://avatars.githubusercontent.com/u/10524058?v=3","gravatar_id":"","url":"https://api.github.com/users/yujuhong","html_url":"https://github.com/yujuhong","followers_url":"https://api.github.com/users/yujuhong/followers","following_url":"https://api.github.com/users/yujuhong/following{/other_user}","gists_url":"https://api.github.com/users/yujuhong/gists{/gist_id}","starred_url":"https://api.github.com/users/yujuhong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yujuhong/subscriptions","organizations_url":"https://api.github.com/users/yujuhong/orgs","repos_url":"https://api.github.com/users/yujuhong/repos","events_url":"https://api.github.com/users/yujuhong/events{/privacy}","received_events_url":"https://api.github.com/users/yujuhong/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/component/kubelet","name":"component/kubelet","color":"0052cc"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-08-13T22:32:14Z","updated_at":"2015-08-13T22:32:14Z","closed_at":null,"body":"Currently, kubelet receive a set of desired pods through the update channel, and blindly triggers all pod workers to sync as well as performing global cleanup. This is inefficient because a single pod change would engage N workers (goroutines), assuming there are N desired pods. Each worker performs a series inspection which swamps Docker, slowing down the reaction to the actual pod change. I did a simple experiment which showed that there was at least ~30% less SyncPod calls when only the relevant pod workers were called.\r\n\r\nKubelet should react to different type of pod changes according and dispatch the work to relevant pod worker:\r\n\r\n```\r\nADD -->  pod admission -->  pod worker dispatcher\r\nUPDATE --> update validation  -->  pod worker dispatcher\r\nREMOVE --> kill pod worker/cleanup\r\n```\r\nThe cleanup work includes removing orphaned pod directories/volumes etc.\r\n\r\nIn today's kubelet, all the work is done in SyncPods (pod admission/validation/dispatcher and global cleanup). I recently broke down the code in `SyncPods` into functions to isolate the different utilities (#12560) as the precursor for eliminating `SyncPods`. If we separate the each component, there is no need for the `SyncPods` function to exist.\r\n\r\nIn addition, kubelet needs to know what kind of spec change it receives (ADD/UPDATE/REMOVE). In fact, the [kubelet/config] module (which watches various sources) supports all different update types, however, for reason unknown to me, kubelet is restricted to only receive the SET operation with all pods included in one update. Due to the lost of information, kubelet is forced to [derive the sync type](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/kubelet/pod_manager.go#L125) later in the code (reverse-engineering!). One way to solve this is to restore the operation types and use them properly.\r\n\r\nKnown caveats: \r\n - Handle kubelet restarts\r\n - Handle mirror pods\r\n\r\nMarking this p1 because this is a relative low-hanging fruit with potential to improve kubelet's performance significantly."}}