{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18126","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18126/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18126/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18126/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18126","id":120075385,"number":18126,"title":"scheduler: using inefficient math library causes significant slowdown","user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-03T01:39:14Z","updated_at":"2015-12-03T01:59:55Z","closed_at":null,"body":"We did a few benchmark and profiling tests for the scheduler for 1000 nodes cluster.\r\n\r\nWe found the scheduler component is the bottleneck of starting pods. And it spents most of the time in an external pkg (https://github.com/kubernetes/kubernetes/blob/master/pkg/api/resource/quantity.go#L27) that handling math operations.\r\n\r\nProfiling result for schedule 1000 pods over 1000 nodes:\r\nhttps://storage.googleapis.com/profiling/org.svg\r\n\r\nBy giving the library a closer look, I found it does too many unnecessary allocations and causes GC pressure.\r\n\r\nBy mocking an efficient version of rounding function (do not handle overflow), we reduced nearly 50% of the latency for secluding 1000 pods(23s vs 53s). As we scheduled more pods, the effect getting more significant since it generates more objects per schedule.\r\n\r\nNew profiling result for schedule 1000 pods over 1000 nodes:\r\nhttps://storage.googleapis.com/profiling/fix.svg\r\n\r\nAny suggestion on how we should improve this?\r\n\r\n/cc @hongchaodeng @lavalamp @philips "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/161493112","html_url":"https://github.com/kubernetes/kubernetes/issues/18126#issuecomment-161493112","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18126","id":161493112,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T01:59:55Z","updated_at":"2015-12-03T01:59:55Z","body":"> Alternatively, writing a special-purpose Value/MilliValue function probably could solve this.\r\n\r\nThat is what I did for the mock up. This seems not to be very hard. Or we can fix the upstream library if we want. \r\n\r\n@timstclair Any idea?"}}