{"action":"created","issue":{"url":"https://api.github.com/repos/drothlis/stb-tester/issues/191","labels_url":"https://api.github.com/repos/drothlis/stb-tester/issues/191/labels{/name}","comments_url":"https://api.github.com/repos/drothlis/stb-tester/issues/191/comments","events_url":"https://api.github.com/repos/drothlis/stb-tester/issues/191/events","html_url":"https://github.com/drothlis/stb-tester/pull/191","id":36595358,"number":191,"title":"stbt batch: New hooks for custom test report and summary generators","user":{"login":"szemate","id":3908828,"avatar_url":"https://avatars.githubusercontent.com/u/3908828?v=3","gravatar_id":"","url":"https://api.github.com/users/szemate","html_url":"https://github.com/szemate","followers_url":"https://api.github.com/users/szemate/followers","following_url":"https://api.github.com/users/szemate/following{/other_user}","gists_url":"https://api.github.com/users/szemate/gists{/gist_id}","starred_url":"https://api.github.com/users/szemate/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/szemate/subscriptions","organizations_url":"https://api.github.com/users/szemate/orgs","repos_url":"https://api.github.com/users/szemate/repos","events_url":"https://api.github.com/users/szemate/events{/privacy}","received_events_url":"https://api.github.com/users/szemate/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2014-06-26T17:10:56Z","updated_at":"2015-01-12T13:32:51Z","closed_at":"2015-01-12T13:32:51Z","pull_request":{"url":"https://api.github.com/repos/drothlis/stb-tester/pulls/191","html_url":"https://github.com/drothlis/stb-tester/pull/191","diff_url":"https://github.com/drothlis/stb-tester/pull/191.diff","patch_url":"https://github.com/drothlis/stb-tester/pull/191.patch"},"body":"`report.py` has certain limitations: generating static pages and not\r\nusing a database back-end makes it somewhat unsuitable for reporting\r\nlarge volumes of test results; it doesn't allow sending stb-tester\r\nresults to an external reporting system.\r\n\r\nTo allow `stbt batch` users to overcome the above issues, add two new\r\nuser hooks, `batch.report` and `batch.summary` that may replace some or\r\nall of `report.py` functionality with user programs.\r\n\r\nThe idea to replace `report.py` is the result of the following\r\nconsiderations:\r\n- Forking and building stb-tester with a customised `stbt batch` would\r\n  involve the maintenance burden of continuously backporting changes\r\n  from upstream. We'd prefer to use official releases.\r\n- Implementing a custom test runner instead of using `stbt batch` is\r\n  certainly viable, but replacing just the report generator saves the\r\n  hassle of reimplementing a very similar test runner.\r\n- Replacing `stbt batch report` would be ambiguous, because this is the\r\n  script that and invokes the `classify` hook. Also, it generates\r\n  `failure-reason`, which is actually very useful.\r\n\r\n`report.py` has two different functionalities:\r\n- `report.py <testrun-dir>` generates a test report of a single test\r\n  run,\r\n- `report.py index.html` generates a summary report of all test runs.\r\nThe former may be replaced with `batch.report` and the latter with\r\n`batch.summary` user programs, using single-word hook names for\r\nsimplicity. Specifying `batch.report` or `batch.summary` with no\r\nvalue skips test report or summary generation respectively.\r\n\r\nA possible use case is generating dynamic report pages. The `report`\r\nhook fills a database using the contents of files provided by `stbt\r\nbatch run`. The `summary` hook is skipped, or it may e.g. send e-mail\r\nnotifications if the ratio of failed tests exceeds a threshold.\r\n\r\nSee the additions to README.rst for further details."},"comment":{"url":"https://api.github.com/repos/drothlis/stb-tester/issues/comments/69569765","html_url":"https://github.com/drothlis/stb-tester/pull/191#issuecomment-69569765","issue_url":"https://api.github.com/repos/drothlis/stb-tester/issues/191","id":69569765,"user":{"login":"drothlis","id":12424,"avatar_url":"https://avatars.githubusercontent.com/u/12424?v=3","gravatar_id":"","url":"https://api.github.com/users/drothlis","html_url":"https://github.com/drothlis","followers_url":"https://api.github.com/users/drothlis/followers","following_url":"https://api.github.com/users/drothlis/following{/other_user}","gists_url":"https://api.github.com/users/drothlis/gists{/gist_id}","starred_url":"https://api.github.com/users/drothlis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drothlis/subscriptions","organizations_url":"https://api.github.com/users/drothlis/orgs","repos_url":"https://api.github.com/users/drothlis/repos","events_url":"https://api.github.com/users/drothlis/events{/privacy}","received_events_url":"https://api.github.com/users/drothlis/received_events","type":"User","site_admin":false},"created_at":"2015-01-12T13:32:51Z","updated_at":"2015-01-12T13:32:51Z","body":"@szemate sure. Closing."}}