{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/9456","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/9456/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/9456/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/9456/events","html_url":"https://github.com/JuliaLang/julia/issues/9456","id":52835668,"number":9456,"title":"Benchmarking harness for Julia","user":{"login":"eschnett","id":213514,"avatar_url":"https://avatars.githubusercontent.com/u/213514?v=3","gravatar_id":"","url":"https://api.github.com/users/eschnett","html_url":"https://github.com/eschnett","followers_url":"https://api.github.com/users/eschnett/followers","following_url":"https://api.github.com/users/eschnett/following{/other_user}","gists_url":"https://api.github.com/users/eschnett/gists{/gist_id}","starred_url":"https://api.github.com/users/eschnett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eschnett/subscriptions","organizations_url":"https://api.github.com/users/eschnett/orgs","repos_url":"https://api.github.com/users/eschnett/repos","events_url":"https://api.github.com/users/eschnett/events{/privacy}","received_events_url":"https://api.github.com/users/eschnett/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2014-12-24T20:29:08Z","updated_at":"2015-04-29T02:12:51Z","closed_at":null,"body":"Julia as a language is not only about correctness, it is also about performance. Consequently, there should be a benchmarking harness in `Base`, equivalent to the testing harness `Base.Test`. This would allow several interesting things:\r\n- Performance regressions are noticed earlier\r\n- People can look at the benchmarks, and learn efficient coding styles\r\n- The benefit of certain changes (`@inbounds`, `@fastmath`) is immediately clear\r\n- Dangerous changes \"for the sake of performance\" (e.g. using `@inbounds` in `Base`) can be vetted; they would only be allowed if they actually show a performance benefit\r\n\r\nSince performance varies by architecture, it is probably necessary to set up a few dedicated testing machines where the benchmarks can be run regularly. These machines would need to keep a history of benchmark results for comparison. I've seen <http://speed.julialang.org/> which looks interesting -- I wonder if it could be set up to look at potentially thousands of small benchmark results.\r\n\r\nI imagine that many Julia packages will in the future contain `@bench` statements in addition to `@test` statements."},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/97284008","html_url":"https://github.com/JuliaLang/julia/issues/9456#issuecomment-97284008","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/9456","id":97284008,"user":{"login":"ScottPJones","id":11798751,"avatar_url":"https://avatars.githubusercontent.com/u/11798751?v=3","gravatar_id":"","url":"https://api.github.com/users/ScottPJones","html_url":"https://github.com/ScottPJones","followers_url":"https://api.github.com/users/ScottPJones/followers","following_url":"https://api.github.com/users/ScottPJones/following{/other_user}","gists_url":"https://api.github.com/users/ScottPJones/gists{/gist_id}","starred_url":"https://api.github.com/users/ScottPJones/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ScottPJones/subscriptions","organizations_url":"https://api.github.com/users/ScottPJones/orgs","repos_url":"https://api.github.com/users/ScottPJones/repos","events_url":"https://api.github.com/users/ScottPJones/events{/privacy}","received_events_url":"https://api.github.com/users/ScottPJones/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T02:12:51Z","updated_at":"2015-04-29T02:12:51Z","body":"@vtjnash, yes, great link... I had to deal with that a lot on early Alpha chips (direct mapped cache instead of n-way associative cache... link order could make huge difference in performance... until we figured out what it was, we were going crazy trying to get consistent benchmark results..."}}