{"action":"created","issue":{"url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/29","labels_url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/29/comments","events_url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/29/events","html_url":"https://github.com/ptaoussanis/clojure-web-server-benchmarks/issues/29","id":64842696,"number":29,"title":"wrk2 request rates (and the resulting latencies) seem arbitrary and not very realistic","user":{"login":"ztellman","id":31295,"avatar_url":"https://avatars.githubusercontent.com/u/31295?v=3","gravatar_id":"","url":"https://api.github.com/users/ztellman","html_url":"https://github.com/ztellman","followers_url":"https://api.github.com/users/ztellman/followers","following_url":"https://api.github.com/users/ztellman/following{/other_user}","gists_url":"https://api.github.com/users/ztellman/gists{/gist_id}","starred_url":"https://api.github.com/users/ztellman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ztellman/subscriptions","organizations_url":"https://api.github.com/users/ztellman/orgs","repos_url":"https://api.github.com/users/ztellman/repos","events_url":"https://api.github.com/users/ztellman/events{/privacy}","received_events_url":"https://api.github.com/users/ztellman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-27T20:12:23Z","updated_at":"2015-03-29T22:05:05Z","closed_at":null,"body":"I'm mostly looking for a response from @xfeep and @ptaoussanis, but anyone with an opinion should chime in.\r\n\r\nIt struck me as a bit weird that the latencies were all measured in seconds, but didn't have the time to really look into it before the benchmarks were published.  Apparently our benchmark is simulating an open queueing network with a sustained throughput of 400k qps.  This means that any throughput that falls short of 400k is treated as if it's sitting on a queue, which is getting progressively longer with each passing second.\r\n\r\nAs far as I know, no one in the Clojure landscape has a system with the above properties.  Some systems at Factual have sustained request rates of 200-250k qps across 4-8 machines, but if the request volume exceeds what it can handle, it returns a 503 status code.  I'd characterize this as a \"typical\" setup.  Almost any large-scale system I'm aware of uses backpressure or load-shedding, not some infinitely growing queue.\r\n\r\nPeak throughput, obviously, can't be measured without choosing some arbitrarily large number (though it looks like the choice of 30k qps for the non-keepalive benchmark may be at least somewhat less than the peak possible throughput, since pretty much everyone achieved that).  But if I'm building a system that's concerned with latency, I won't push the max possible volume through a server, I'll limit it to no more than 80% or so.\r\n\r\nA more useful set of benchmarks would be mean and 99.99% latency values for 50k qps, 100k qps, etc.  I know this creates even more of a permutation explosion, but I think the current set of latency values are useless at best, and greatly misleading at worst.  I'd like to seem them replaced with something else as soon as we can agree on what that should be."},"comment":{"url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/comments/87487018","html_url":"https://github.com/ptaoussanis/clojure-web-server-benchmarks/issues/29#issuecomment-87487018","issue_url":"https://api.github.com/repos/ptaoussanis/clojure-web-server-benchmarks/issues/29","id":87487018,"user":{"login":"ztellman","id":31295,"avatar_url":"https://avatars.githubusercontent.com/u/31295?v=3","gravatar_id":"","url":"https://api.github.com/users/ztellman","html_url":"https://github.com/ztellman","followers_url":"https://api.github.com/users/ztellman/followers","following_url":"https://api.github.com/users/ztellman/following{/other_user}","gists_url":"https://api.github.com/users/ztellman/gists{/gist_id}","starred_url":"https://api.github.com/users/ztellman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ztellman/subscriptions","organizations_url":"https://api.github.com/users/ztellman/orgs","repos_url":"https://api.github.com/users/ztellman/repos","events_url":"https://api.github.com/users/ztellman/events{/privacy}","received_events_url":"https://api.github.com/users/ztellman/received_events","type":"User","site_admin":false},"created_at":"2015-03-29T22:05:05Z","updated_at":"2015-03-29T22:05:05Z","body":"I know there are configurations that can be used to increase the range of ports, but I'm not sure how important that is to benchmark.  I'm not sure, offhand, what the difference between the 1k and 60k benchmarks are if they're all closing the connections after the first response.  Isn't a single client opening 30k connections/sec and 30k clients opening 1 connection/sec functionally equivalent?  And if pretty much every server can handle the resulting 30k qps, what is the benchmark telling us?  It may capture the cost of a TCP handshake over localhost, but again, that doesn't seem very relevant to any real-world use case.\r\n\r\nAs far as I can tell, the work being discussed in your link is what single scalar value should be used to represent the \"speed\" of a server.  What's being suggested there seems fine, if a little vague, but my concern is mostly w.r.t. the latency values.  I've run a very simple set of benchmarks on my local machine using Aleph:\r\n\r\n```\r\nfor x in {5000,10000,15000,20000,25000,30000,35000,40000}; do wrk2 -t2 -c100 -d60s --latency -R $x http://127.0.0.1:8080; done\r\n```\r\n\r\n![screen shot 2015-03-29 at 2 50 05 pm](https://cloud.githubusercontent.com/assets/31295/6887977/eb3846ce-d623-11e4-91b9-bd06b2f12bf6.png)\r\n\r\nMy understanding of the linked conversation is that we would effectively draw a vertical line wherever any of these go over the specified thresholds.  This would be nice as a way to order the servers, but I think it would also be useful to just have a graph like the above for each of the servers, since that will give the best indication of the relative performance of each of the servers, no matter what a person's latency requirements are.\r\n\r\nAnd even if we do decide to wait for the TechEmpower guys to come up with a better performance measurement, I think we should immediately remove the latency graphs, and at least discuss what the relative value of the 1k and 60k non-keepalive benchmarks are. "}}