{"action":"created","issue":{"url":"https://api.github.com/repos/nengo/nengo/issues/678","labels_url":"https://api.github.com/repos/nengo/nengo/issues/678/labels{/name}","comments_url":"https://api.github.com/repos/nengo/nengo/issues/678/comments","events_url":"https://api.github.com/repos/nengo/nengo/issues/678/events","html_url":"https://github.com/nengo/nengo/pull/678","id":60418686,"number":678,"title":"Comparison tests for analytics data.","user":{"login":"jgosmann","id":850157,"avatar_url":"https://avatars.githubusercontent.com/u/850157?v=3","gravatar_id":"","url":"https://api.github.com/users/jgosmann","html_url":"https://github.com/jgosmann","followers_url":"https://api.github.com/users/jgosmann/followers","following_url":"https://api.github.com/users/jgosmann/following{/other_user}","gists_url":"https://api.github.com/users/jgosmann/gists{/gist_id}","starred_url":"https://api.github.com/users/jgosmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgosmann/subscriptions","organizations_url":"https://api.github.com/users/jgosmann/orgs","repos_url":"https://api.github.com/users/jgosmann/repos","events_url":"https://api.github.com/users/jgosmann/events{/privacy}","received_events_url":"https://api.github.com/users/jgosmann/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/nengo/nengo/labels/needs%20review","name":"needs review","color":"eb6420"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/nengo/nengo/milestones/6","html_url":"https://github.com/nengo/nengo/milestones/2.1.0%20release","labels_url":"https://api.github.com/repos/nengo/nengo/milestones/6/labels","id":747893,"number":6,"title":"2.1.0 release","description":"For changes that will affect model code.","creator":{"login":"tbekolay","id":203709,"avatar_url":"https://avatars.githubusercontent.com/u/203709?v=3","gravatar_id":"","url":"https://api.github.com/users/tbekolay","html_url":"https://github.com/tbekolay","followers_url":"https://api.github.com/users/tbekolay/followers","following_url":"https://api.github.com/users/tbekolay/following{/other_user}","gists_url":"https://api.github.com/users/tbekolay/gists{/gist_id}","starred_url":"https://api.github.com/users/tbekolay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbekolay/subscriptions","organizations_url":"https://api.github.com/users/tbekolay/orgs","repos_url":"https://api.github.com/users/tbekolay/repos","events_url":"https://api.github.com/users/tbekolay/events{/privacy}","received_events_url":"https://api.github.com/users/tbekolay/received_events","type":"User","site_admin":false},"open_issues":43,"closed_issues":16,"state":"open","created_at":"2014-08-08T17:42:35Z","updated_at":"2015-04-13T20:21:27Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2015-03-09T22:08:20Z","updated_at":"2015-04-13T21:05:17Z","closed_at":"2015-04-13T20:21:27Z","pull_request":{"url":"https://api.github.com/repos/nengo/nengo/pulls/678","html_url":"https://github.com/nengo/nengo/pull/678","diff_url":"https://github.com/nengo/nengo/pull/678.diff","patch_url":"https://github.com/nengo/nengo/pull/678.patch"},"body":"Github does not allow to change the branch a PR is made against. Had to close #657 and reopen a new PR. Copied the description, but not the comments.\r\n\r\n--------------------------------------\r\n\r\nThis is a follow-up PR to #647.\r\n\r\nIt allows to mark test functions with `pytest.mark.compare` and indicates that the function will compare the data from two test runs producing analytics data. A test marked with `compare` will not run per default. Actually, it will be pruned from the collected tests and does not influence the success/skipped/failure statistics.\r\n\r\nTo run the comparison tests one has to pass `--compare [old-data] [new-data]` to `py.test`. This will prune all normal tests and only collect the comparison tests.\r\n\r\n## How to write a comparison test\r\n\r\n1. Mark it with `pytest.mark.compare`.\r\n2. Name it `test_compare_<testname>` given that `test_<testname>` produced the relevant data. This naming scheme is necessary to automatically load the relevant analytics data. It would be possible to add the mark based on the test name, but maybe that's too much magic. It would also be possible to get rid of the `test_` prefix (but again it would be a bit more magic).\r\n3. Add the `analytics_data` fixture as function argument. It will be a two element sequence with the same order as the arguments to `--compare`. By conventions the first one should be the older data.\r\n4. Print some useful information (e.g., \"X got Y% more accurate.\").\r\n5. Do an `assert` whether the metric improved. That way a failure will be reported when something got worse.\r\n\r\nAll output to stdout and stderr will also be shown for successful tests as there might be useful information (like by how much something got better).\r\n\r\nI'll prepare another PR demonstrating such a comparison test."},"comment":{"url":"https://api.github.com/repos/nengo/nengo/issues/comments/92498460","html_url":"https://github.com/nengo/nengo/pull/678#issuecomment-92498460","issue_url":"https://api.github.com/repos/nengo/nengo/issues/678","id":92498460,"user":{"login":"tbekolay","id":203709,"avatar_url":"https://avatars.githubusercontent.com/u/203709?v=3","gravatar_id":"","url":"https://api.github.com/users/tbekolay","html_url":"https://github.com/tbekolay","followers_url":"https://api.github.com/users/tbekolay/followers","following_url":"https://api.github.com/users/tbekolay/following{/other_user}","gists_url":"https://api.github.com/users/tbekolay/gists{/gist_id}","starred_url":"https://api.github.com/users/tbekolay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbekolay/subscriptions","organizations_url":"https://api.github.com/users/tbekolay/orgs","repos_url":"https://api.github.com/users/tbekolay/repos","events_url":"https://api.github.com/users/tbekolay/events{/privacy}","received_events_url":"https://api.github.com/users/tbekolay/received_events","type":"User","site_admin":false},"created_at":"2015-04-13T21:05:17Z","updated_at":"2015-04-13T21:05:17Z","body":"Whoops, sorry about that. For the record, :+1: "}}