{"action":"closed","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3734","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3734/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3734/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3734/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3734","id":55210021,"number":3734,"title":"kube-proxy appears to have some bad long-tail latency issues","user":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/networking","name":"area/networking","color":"006b75"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/performance","name":"area/performance","color":"fef2c0"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P3","name":"priority/P3","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-01-22T21:17:16Z","updated_at":"2015-01-28T00:23:14Z","closed_at":"2015-01-28T00:23:14Z","body":"using redis-benchmark as a test:\r\n\r\nwith the proxy:\r\n====== INCR ======\r\n  10000 requests completed in 0.67 seconds\r\n  50 parallel clients\r\n  3 bytes payload\r\n  keep alive: 1\r\n\r\n1.16% <= 1 milliseconds\r\n8.60% <= 2 milliseconds\r\n72.41% <= 3 milliseconds\r\n95.61% <= 4 milliseconds\r\n98.83% <= 5 milliseconds\r\n99.02% <= 6 milliseconds\r\n99.25% <= 7 milliseconds\r\n99.28% <= 8 milliseconds\r\n99.35% <= 9 milliseconds\r\n99.54% <= 10 milliseconds\r\n99.58% <= 11 milliseconds\r\n99.59% <= 13 milliseconds\r\n99.60% <= 20 milliseconds\r\n99.61% <= 21 milliseconds\r\n99.62% <= 23 milliseconds\r\n99.63% <= 25 milliseconds\r\n99.64% <= 27 milliseconds\r\n99.65% <= 29 milliseconds\r\n99.66% <= 31 milliseconds\r\n99.67% <= 32 milliseconds\r\n99.68% <= 34 milliseconds\r\n99.69% <= 35 milliseconds\r\n99.70% <= 39 milliseconds\r\n99.71% <= 40 milliseconds\r\n99.72% <= 42 milliseconds\r\n99.73% <= 43 milliseconds\r\n99.74% <= 49 milliseconds\r\n99.75% <= 52 milliseconds\r\n99.76% <= 53 milliseconds\r\n99.77% <= 55 milliseconds\r\n99.78% <= 57 milliseconds\r\n99.79% <= 60 milliseconds\r\n99.80% <= 61 milliseconds\r\n99.81% <= 64 milliseconds\r\n99.82% <= 66 milliseconds\r\n99.83% <= 68 milliseconds\r\n99.84% <= 70 milliseconds\r\n99.85% <= 73 milliseconds\r\n99.86% <= 74 milliseconds\r\n99.87% <= 77 milliseconds\r\n99.88% <= 80 milliseconds\r\n99.89% <= 82 milliseconds\r\n99.90% <= 85 milliseconds\r\n99.91% <= 89 milliseconds\r\n99.92% <= 92 milliseconds\r\n99.93% <= 95 milliseconds\r\n99.94% <= 98 milliseconds\r\n99.95% <= 100 milliseconds\r\n99.96% <= 103 milliseconds\r\n99.97% <= 107 milliseconds\r\n99.98% <= 110 milliseconds\r\n99.99% <= 113 milliseconds\r\n100.00% <= 116 milliseconds\r\n14880.95 requests per second\r\n\r\nwithout:\r\n====== INCR ======\r\n  10000 requests completed in 0.43 seconds\r\n  50 parallel clients\r\n  3 bytes payload\r\n  keep alive: 1\r\n\r\n0.20% <= 1 milliseconds\r\n67.35% <= 2 milliseconds\r\n94.11% <= 3 milliseconds\r\n97.21% <= 4 milliseconds\r\n98.23% <= 5 milliseconds\r\n99.04% <= 6 milliseconds\r\n99.51% <= 14 milliseconds\r\n99.82% <= 15 milliseconds\r\n100.00% <= 15 milliseconds\r\n23255.81 requests per second\r\n\r\nWe're basically at the same point for the 99%, but past that the proxy falls off a cliff.  Would be interesting to understand why."}}