{"action":"created","issue":{"url":"https://api.github.com/repos/odlgroup/odl/issues/128","labels_url":"https://api.github.com/repos/odlgroup/odl/issues/128/labels{/name}","comments_url":"https://api.github.com/repos/odlgroup/odl/issues/128/comments","events_url":"https://api.github.com/repos/odlgroup/odl/issues/128/events","html_url":"https://github.com/odlgroup/odl/issues/128","id":120870738,"number":128,"title":"Performance tests","user":{"login":"adler-j","id":2202312,"avatar_url":"https://avatars.githubusercontent.com/u/2202312?v=3","gravatar_id":"","url":"https://api.github.com/users/adler-j","html_url":"https://github.com/adler-j","followers_url":"https://api.github.com/users/adler-j/followers","following_url":"https://api.github.com/users/adler-j/following{/other_user}","gists_url":"https://api.github.com/users/adler-j/gists{/gist_id}","starred_url":"https://api.github.com/users/adler-j/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adler-j/subscriptions","organizations_url":"https://api.github.com/users/adler-j/orgs","repos_url":"https://api.github.com/users/adler-j/repos","events_url":"https://api.github.com/users/adler-j/events{/privacy}","received_events_url":"https://api.github.com/users/adler-j/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/odlgroup/odl/labels/accepted","name":"accepted","color":"009800"},{"url":"https://api.github.com/repos/odlgroup/odl/labels/continuous%20integration","name":"continuous integration","color":"bfdadc"}],"state":"open","locked":false,"assignee":{"login":"kohr-h","id":5030250,"avatar_url":"https://avatars.githubusercontent.com/u/5030250?v=3","gravatar_id":"","url":"https://api.github.com/users/kohr-h","html_url":"https://github.com/kohr-h","followers_url":"https://api.github.com/users/kohr-h/followers","following_url":"https://api.github.com/users/kohr-h/following{/other_user}","gists_url":"https://api.github.com/users/kohr-h/gists{/gist_id}","starred_url":"https://api.github.com/users/kohr-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kohr-h/subscriptions","organizations_url":"https://api.github.com/users/kohr-h/orgs","repos_url":"https://api.github.com/users/kohr-h/repos","events_url":"https://api.github.com/users/kohr-h/events{/privacy}","received_events_url":"https://api.github.com/users/kohr-h/received_events","type":"User","site_admin":false},"milestone":null,"comments":23,"created_at":"2015-12-07T21:22:06Z","updated_at":"2015-12-10T09:05:39Z","closed_at":null,"body":"Since we now have a CI server, I think we should add more complicated tests that do not fall under the \"unit test\" category. There are a few reasons for this:\r\n\r\n* We do not want ODL to become slower over time\r\n* We need to see that ODL scales properly to large data sets\r\n* We need to test algorithms with no quick example (reconstructions, large scale optimization)\r\n* These can also serve as more complicated examples of solving \"real\" problems for our users.\r\n\r\nI therefore propose that we add a new folder somewhere, and populate it with these tests. There are a few things to decide on first:\r\n\r\nScope\r\n-----\r\nTo me, there are two purposes here, performance and solution of large scale problems. While they may coincide, they don't necessarily do so. In general the large scale problems are more of examples but with realistic-ish data, while the performance tests could be written in a more simple way but simply with larger parameters. This suggests that this is a two part endeavor.\r\n\r\nUse of supporting library\r\n-------------------------\r\nWith a short Google search, I've found the following:\r\n\r\n* [AirSpeed Velocity](http://asv.readthedocs.org/en/latest/index.html) (asv): Used by numpy, seems to provide decent looking graphs. Syntax is quite pytest like.\r\n* [benchmark](https://pypi.python.org/pypi/benchmark/0.1.5): On pip but seems abandoned.\r\n* [codespeed](https://github.com/tobami/codespeed/): Used by PyPy. Also good looking. See http://speed.pypy.org/ for examples. Somewhat more complicated syntax.\r\n\r\nOf these, to me AirSpeed Velocity looks the best.\r\n\r\nLocation\r\n--------\r\nPossible locations for these include:\r\n\r\n* `benchmarks`: Numpy does this, and `asv` seems to prefer it.\r\n* `test/benchmarks`: A bit more hidden away.\r\n* `test/slow`: A more wide definition.\r\n\r\nI propose that we do the first and the last one, we then add [custom command line options](https://pytest.org/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option) to py.test so that users can activate the slow tests by `py.test --runslow` (or similar). Finally we run the benchmarks by `asv run` if you want.\r\n\r\nDoes this sound like a good idea, do @kohr-h think it would be hard to add?"},"comment":{"url":"https://api.github.com/repos/odlgroup/odl/issues/comments/163538830","html_url":"https://github.com/odlgroup/odl/issues/128#issuecomment-163538830","issue_url":"https://api.github.com/repos/odlgroup/odl/issues/128","id":163538830,"user":{"login":"kohr-h","id":5030250,"avatar_url":"https://avatars.githubusercontent.com/u/5030250?v=3","gravatar_id":"","url":"https://api.github.com/users/kohr-h","html_url":"https://github.com/kohr-h","followers_url":"https://api.github.com/users/kohr-h/followers","following_url":"https://api.github.com/users/kohr-h/following{/other_user}","gists_url":"https://api.github.com/users/kohr-h/gists{/gist_id}","starred_url":"https://api.github.com/users/kohr-h/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kohr-h/subscriptions","organizations_url":"https://api.github.com/users/kohr-h/orgs","repos_url":"https://api.github.com/users/kohr-h/repos","events_url":"https://api.github.com/users/kohr-h/events{/privacy}","received_events_url":"https://api.github.com/users/kohr-h/received_events","type":"User","site_admin":false},"created_at":"2015-12-10T09:05:39Z","updated_at":"2015-12-10T09:05:39Z","body":"Should be sufficient if we do it outside the test function, Jenkins produces timing per test function. We should not use fixtures though, they cause the HTML output to be named by index, not by test function. I don't quite know why."}}