{"action":"created","issue":{"url":"https://api.github.com/repos/drothlis/stb-tester/issues/250","labels_url":"https://api.github.com/repos/drothlis/stb-tester/issues/250/labels{/name}","comments_url":"https://api.github.com/repos/drothlis/stb-tester/issues/250/comments","events_url":"https://api.github.com/repos/drothlis/stb-tester/issues/250/events","html_url":"https://github.com/drothlis/stb-tester/pull/250","id":56318591,"number":250,"title":"stbt run: Treat AssertionErrors as test failures, not test errors","user":{"login":"drothlis","id":12424,"avatar_url":"https://avatars.githubusercontent.com/u/12424?v=3","gravatar_id":"","url":"https://api.github.com/users/drothlis","html_url":"https://github.com/drothlis","followers_url":"https://api.github.com/users/drothlis/followers","following_url":"https://api.github.com/users/drothlis/following{/other_user}","gists_url":"https://api.github.com/users/drothlis/gists{/gist_id}","starred_url":"https://api.github.com/users/drothlis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drothlis/subscriptions","organizations_url":"https://api.github.com/users/drothlis/orgs","repos_url":"https://api.github.com/users/drothlis/repos","events_url":"https://api.github.com/users/drothlis/events{/privacy}","received_events_url":"https://api.github.com/users/drothlis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/drothlis/stb-tester/milestones/6","labels_url":"https://api.github.com/repos/drothlis/stb-tester/milestones/6/labels","id":962518,"number":6,"title":"0.22","description":null,"creator":{"login":"drothlis","id":12424,"avatar_url":"https://avatars.githubusercontent.com/u/12424?v=3","gravatar_id":"","url":"https://api.github.com/users/drothlis","html_url":"https://github.com/drothlis","followers_url":"https://api.github.com/users/drothlis/followers","following_url":"https://api.github.com/users/drothlis/following{/other_user}","gists_url":"https://api.github.com/users/drothlis/gists{/gist_id}","starred_url":"https://api.github.com/users/drothlis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drothlis/subscriptions","organizations_url":"https://api.github.com/users/drothlis/orgs","repos_url":"https://api.github.com/users/drothlis/repos","events_url":"https://api.github.com/users/drothlis/events{/privacy}","received_events_url":"https://api.github.com/users/drothlis/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":0,"state":"open","created_at":"2015-02-02T22:21:36Z","updated_at":"2015-02-03T01:02:44Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-02-03T01:02:28Z","updated_at":"2015-02-03T10:19:32Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/drothlis/stb-tester/pulls/250","html_url":"https://github.com/drothlis/stb-tester/pull/250","diff_url":"https://github.com/drothlis/stb-tester/pull/250.diff","patch_url":"https://github.com/drothlis/stb-tester/pull/250.patch"},"body":"This brings stb-tester in line with traditional testing frameworks like\r\nnose and pytest. Along with the introduction of `match()` in 33e8e356\r\nand (in a future commit) `wait_until()`, this is part of a drive to\r\nprovide a more composible API. You can say things like:\r\n\r\n* `assert match(\"xyz.png\")`\r\n* `assert not match(\"xyz.png\")`\r\n* `assert match(\"a.png\") and match(\"b.png\") and not match(\"c.png\")`\r\n* `assert wait_until(lambda: not match(\"xyz.png\"))`\r\n* `assert match_text(\"Main menu\")`\r\n* etc.\r\n\r\nThere are drawbacks to using `assert` instead of `wait_for_match`:\r\n\r\n* The exception message won't contain the reason why the match\r\n  failed (unless you specify it as a second parameter to `assert`,\r\n  which is tedious and we don't expect you to do it), and\r\n* The exception won't have the offending video-frame attached (so the\r\n  screenshot that `stbt batch run` saves alongside the failing test\r\n  logs will be a few frames later than the frame that actually caused\r\n  the test to fail).\r\n\r\nWe hope to solve both problems at some point in the future: the first\r\none by using pytest and its assertion introspection[1] (which rewrites\r\npython modules on import to provide the additional information on\r\nassertion failures), and the second one by extending pytest's\r\nmodule-rewriting to attach the video frame to the AssertionError.\r\n\r\n[1] http://pytest.org/latest/assert.html#advanced-assertion-introspection"},"comment":{"url":"https://api.github.com/repos/drothlis/stb-tester/issues/comments/72626368","html_url":"https://github.com/drothlis/stb-tester/pull/250#issuecomment-72626368","issue_url":"https://api.github.com/repos/drothlis/stb-tester/issues/250","id":72626368,"user":{"login":"LewisHaley","id":3766135,"avatar_url":"https://avatars.githubusercontent.com/u/3766135?v=3","gravatar_id":"","url":"https://api.github.com/users/LewisHaley","html_url":"https://github.com/LewisHaley","followers_url":"https://api.github.com/users/LewisHaley/followers","following_url":"https://api.github.com/users/LewisHaley/following{/other_user}","gists_url":"https://api.github.com/users/LewisHaley/gists{/gist_id}","starred_url":"https://api.github.com/users/LewisHaley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LewisHaley/subscriptions","organizations_url":"https://api.github.com/users/LewisHaley/orgs","repos_url":"https://api.github.com/users/LewisHaley/repos","events_url":"https://api.github.com/users/LewisHaley/events{/privacy}","received_events_url":"https://api.github.com/users/LewisHaley/received_events","type":"User","site_admin":false},"created_at":"2015-02-03T10:19:32Z","updated_at":"2015-02-03T10:19:32Z","body":"> The exception won't have the offending video-frame attached (so the screenshot that stbt batch run saves alongside the failing test logs will be a few frames later than the frame that actually caused the test to fail).\r\n\r\nWorth noting that due to #111, rather and \"a few frames\", when using a VidiU this is more like 20-30 seconds."}}