{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18091","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18091/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18091/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18091/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18091","id":120007809,"number":18091,"title":"Scheduler Intergration Testing Proposal","user":{"login":"hongchaodeng","id":920884,"avatar_url":"https://avatars.githubusercontent.com/u/920884?v=3","gravatar_id":"","url":"https://api.github.com/users/hongchaodeng","html_url":"https://github.com/hongchaodeng","followers_url":"https://api.github.com/users/hongchaodeng/followers","following_url":"https://api.github.com/users/hongchaodeng/following{/other_user}","gists_url":"https://api.github.com/users/hongchaodeng/gists{/gist_id}","starred_url":"https://api.github.com/users/hongchaodeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hongchaodeng/subscriptions","organizations_url":"https://api.github.com/users/hongchaodeng/orgs","repos_url":"https://api.github.com/users/hongchaodeng/repos","events_url":"https://api.github.com/users/hongchaodeng/events{/privacy}","received_events_url":"https://api.github.com/users/hongchaodeng/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-12-02T18:36:36Z","updated_at":"2015-12-02T20:13:13Z","closed_at":null,"body":"We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation.\r\n\r\nWe have the following goals:\r\n- Goal 1: Save time on testing. More accurate scheduling latency.\r\n  - We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components.\r\n- Goal 2: Profiling runtime metrics and find out performance bottleneck.\r\n  - Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics.\r\n- Goal 3: Reproduce test result easily.\r\n  - We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. \r\n\r\nWe have the following plans for writing scheduler tests:\r\n- density test (by adding a new Go test)\r\n  - schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes\r\n  - print out scheduling rate every second\r\n  - let you learn the rate changes vs number of scheduled pods\r\n- benchmark (by modifying the existing benchmark in integration or simply add a new one)\r\n  - make use of `go test -bench` and report nanosecond/op.\r\n  - schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10.\r\n\r\nWe are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? \r\n\r\nAny suggestions are welcomed!"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/161420157","html_url":"https://github.com/kubernetes/kubernetes/issues/18091#issuecomment-161420157","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18091","id":161420157,"user":{"login":"timothysc","id":169553,"avatar_url":"https://avatars.githubusercontent.com/u/169553?v=3","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"created_at":"2015-12-02T20:13:13Z","updated_at":"2015-12-02T20:13:13Z","body":"> schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes\r\n> print out scheduling rate every second\r\n> let you learn the rate changes vs number of scheduled pods\r\n\r\nSo density already does this... the one option it looks like you may want is custom control #18099 real or fake is irrelevant as the e2e's just need to point to an api-server. \r\n\r\nre: benchmarks, seems like we could simply add to the unit-tests. \r\n\r\n\r\n\r\n"}}