{"action":"created","issue":{"url":"https://api.github.com/repos/sstephenson/bats/issues/110","labels_url":"https://api.github.com/repos/sstephenson/bats/issues/110/labels{/name}","comments_url":"https://api.github.com/repos/sstephenson/bats/issues/110/comments","events_url":"https://api.github.com/repos/sstephenson/bats/issues/110/events","html_url":"https://github.com/sstephenson/bats/pull/110","id":95956748,"number":110,"title":"Standard Library of Test Helpers","user":{"login":"ztombol","id":5340257,"avatar_url":"https://avatars.githubusercontent.com/u/5340257?v=3","gravatar_id":"","url":"https://api.github.com/users/ztombol","html_url":"https://github.com/ztombol","followers_url":"https://api.github.com/users/ztombol/followers","following_url":"https://api.github.com/users/ztombol/following{/other_user}","gists_url":"https://api.github.com/users/ztombol/gists{/gist_id}","starred_url":"https://api.github.com/users/ztombol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ztombol/subscriptions","organizations_url":"https://api.github.com/users/ztombol/orgs","repos_url":"https://api.github.com/users/ztombol/repos","events_url":"https://api.github.com/users/ztombol/events{/privacy}","received_events_url":"https://api.github.com/users/ztombol/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-07-19T23:04:27Z","updated_at":"2015-08-02T22:01:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/sstephenson/bats/pulls/110","html_url":"https://github.com/sstephenson/bats/pull/110","diff_url":"https://github.com/sstephenson/bats/pull/110.diff","patch_url":"https://github.com/sstephenson/bats/pull/110.patch"},"body":"# Overview\r\n\r\nThis pull request introduces a Standard Library of Test Helpers that provides better feedback on failed tests to help debugging. It also argues that including this library in Bats greatly simplifies its usage.\r\n\r\n\r\n# Problem\r\n\r\n**When a test fails** Bats displays a lot of useful information, e.g. the failed expression and its line number. However, in many cases **additional information would be helpful**. Providing relevant information to speed up debugging is an actively pursued feature. It has been requested and proposed multiple times without success. Implementations generally fall into one of the following two categories.\r\n\r\n1. Modifying Bats to **automatically print** a static set of information, e.g. `$status` and `$output`, when a test fails.\r\n\r\n   **Advantages:**\r\n   - *Just works* - There is no need to modify tests to take advantage of this feature.\r\n\r\n   **Disadvantages:**\r\n   - *Invasive* - Prints regardless the user wants it or not (consequence of being automatic).\r\n   - *Inflexible* - May print irrelevant information making debugging harder.\r\n\r\n   Due to its disadvantages this approach has been rejected multiple times in the past (see #8, #101, #105).\r\n\r\n2. Using **test helpers** to print only information relevant to the failure.\r\n   \r\n   **Advantages:**\r\n   - *Concise* - Only displays the information relevant to the failure.\r\n   - *Opt-in* - Produces output only if the developer wants it.\r\n\r\n   **Disadvantages:**\r\n   - *Tests need a rewrite* - Tests have to be rewritten to take advantage of this feature (consequence of being opt-in).\r\n\r\n   This is the recommended approach (see #5).\r\n\r\nSolution 2 is clearly the superior, and in fact the [recommended][bats-5] approach. Two years have passed since that recommendation but **a general purpose library has never emerged.**\r\n\r\n***Note:*** *To be fair, @juanibiapina shared his library of assertions in #5. However, as of 2015-07-19 the link is dead and library cannot be found on his Github account.*\r\n\r\n\r\n# Standard Library of Test Helpers\r\n\r\nAt the moment **projects have to develop their own library** by adapting another project's test helpers if the licences permit it or writing one from scratch. Six projects on the list of [projects using Bats][bats-wiki-proj] use a test helper library. All of which are a variation of the same library.\r\n\r\n***Note:*** *I know that the list mentioned above is not up to date, but users of Bats may take the projects on that list as references when building their own test suites.*\r\n\r\n- [rbenv][th-rbenv]\r\n- [ruby-build][th-ruby-build]\r\n- [nodenv][th-nodenv]\r\n- [shy][th-shy]\r\n- [envirius][th-envirius]\r\n- [github-markdown-toc][th-gh-md-toc]\r\n\r\nThe current situation has the following problems:\r\n\r\n- **Reinventing the wheel** - Every project has to invest into developing their own, very similar, library.\r\n- **Licensing** - If the licence of the source and your project are not compatible, you are back to writing it from scratch.\r\n- **No testing** - Ironically none of the six projects mentioned above actually test their test helpers. Hence, bugs may propagate from one project to another.\r\n- **Non-standard API** - It is harder for developers new to a project to contribute because they need to read the source of the test helpers on the off chance that these behave a little different than in another project where they encountered a similar library.\r\n- **No documentation** - Total lack of documentation of test helpers in any of the mentioned projects further raises the barrier to entry.\r\n- **Isolated improvements** - The lack of an upstream version and concentrated development efforts mean that improvements to the library of one project does not directly benefit another, and sharing improvements is difficult.\r\n\r\nA properly licensed *Standard Library of Test Helpers* could mitigate these problems and provide a better experience for Bats users. Despite the advantages of a general purpose library, one has never emerged. This pull request **introduces the Standard Library containing the most commonly sought functions**.\r\n\r\n\r\n# Including the Standard Library in Bats\r\n\r\nAlthough, distributing a library of test helpers with Bats has been [rejected in the early days][bats-5], I believe its time to re-evaluate the question in light of the current situation.\r\n\r\nThe dependency chain of the test suit for a typical project using the included Standard library would look like the following.\r\n\r\n```\r\n+-------+    +------------------+    +----------+    +------+\r\n| Tests | -> | Project specific | -> | Standard | -> | Bats |\r\n|       |    |  Test Libraries  |    | Library  |    |      |\r\n+-------+    +------------------+    +----------+    +------+\r\n\r\n\\_______________________________/    \\______________________/\r\n               |                                |\r\n            Project                           Bats\r\n```\r\n\r\nIncluding the Standard Library in Bats would greatly simplify its usage and would put less burden on individual projects.\r\n\r\n- **In sync with Bats** - The Standard Library would always be kept compatible with Bats. This would eliminate the delay between a compatibility breaking update of Bats and the update of the Standard Library. Project specific test helper libraries would still need to be updated of course.\r\n- **No vendoring** - The Standard Library would be available wherever Bats is installed without the project needing to vendor and periodically update it.\r\n\r\nAdditionally, having an official, universally available, easy to use Standard Library would encourage contribution.\r\n\r\n\r\n# Licensing\r\n\r\nRegardless whether a general purpose library, like the one proposed in this pull request, is included in Bats, [licensing][gpl-faq-interpreter] remains an important question.\r\n\r\nTest files are just input data to Bats. The licence of Bats does not restrict the licence of the input files or the licence of the software being tested.\r\n\r\nHowever, when using test helper libraries, the licence of the libraries and the licence of the test files using them must be compatible.\r\n\r\nFurthermore, if a test file sources a file of the project (for example to do unit tests on a library) the licences of the sourced file, the test helper library, and the test file must all be compatible.\r\n\r\nThis means that **the licence of a general purpose test helper library can limit what projects may use it**. A permissive licence or even the Public Domain may be necessary to allow many projects to use the library.\r\n\r\n\r\n# Summary\r\n\r\nA Standard Library of Test Helpers helps projects to write better test suits faster. However, no such library has emerged during the past two years. This pull request introduces a library containing the most common test helpers.\r\n\r\nIncluding this library in Bats, rather than hosting it in a separate repository, improves its usability, simplifies maintenance and encourages contribution.\r\n\r\nThe licence of the Standard Library has a great impact on its usability and should be carefully considered.\r\n\r\n\r\n\r\n<!-- References -->\r\n\r\n[bats-5]: https://github.com/sstephenson/bats/issues/5\r\n[bats-wiki-proj]: https://github.com/sstephenson/bats/wiki/Projects-Using-Bats\r\n[th-rbenv]: https://github.com/sstephenson/rbenv/blob/master/test/test_helper.bash\r\n[th-ruby-build]: https://github.com/sstephenson/ruby-build/blob/master/test/test_helper.bash\r\n[th-nodenv]: https://github.com/OiNutter/nodenv/blob/master/test/test_helper.bash\r\n[th-shy]: https://github.com/aaronroyer/shy/blob/master/test/test_helper.bash\r\n[th-envirius]: https://github.com/ekalinin/envirius/blob/master/tests/test_helper.bash\r\n[th-gh-md-toc]: https://github.com/ekalinin/github-markdown-toc/blob/master/tests/test_helper.bash\r\n[gpl-faq-interpreter]: http://www.gnu.org/licenses/gpl-faq.en.html#IfInterpreterIsGPL"},"comment":{"url":"https://api.github.com/repos/sstephenson/bats/issues/comments/127075586","html_url":"https://github.com/sstephenson/bats/pull/110#issuecomment-127075586","issue_url":"https://api.github.com/repos/sstephenson/bats/issues/110","id":127075586,"user":{"login":"ztombol","id":5340257,"avatar_url":"https://avatars.githubusercontent.com/u/5340257?v=3","gravatar_id":"","url":"https://api.github.com/users/ztombol","html_url":"https://github.com/ztombol","followers_url":"https://api.github.com/users/ztombol/followers","following_url":"https://api.github.com/users/ztombol/following{/other_user}","gists_url":"https://api.github.com/users/ztombol/gists{/gist_id}","starred_url":"https://api.github.com/users/ztombol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ztombol/subscriptions","organizations_url":"https://api.github.com/users/ztombol/orgs","repos_url":"https://api.github.com/users/ztombol/repos","events_url":"https://api.github.com/users/ztombol/events{/privacy}","received_events_url":"https://api.github.com/users/ztombol/received_events","type":"User","site_admin":false},"created_at":"2015-08-02T22:01:47Z","updated_at":"2015-08-02T22:01:47Z","body":"@sstephenson thank you for the detailed feedback!\r\n\r\n> - No need to split this up into multiple modules—one standard library file is sufficient. In other words, we don't need the separate \"core\" distinction.\r\n> - Why shouldn't we automatically include these helpers for every test case? We can source the stdlib before the test file, so any existing functions with the same name just override the defaults.\r\n\r\nI agree with both. Simplifies usage quite a bit.\r\n\r\n> - If we decide we can't automatically include the helpers, we'll need a better name than load_std. I'd rather not load an STD into my tests ;-)\r\n\r\nThis cracked me up! Good one! :smile:\r\n\r\n> - Let's move the documentation right into the main readme. These helpers should be front and center.\r\n> - We ought to publish a separate man page for the helpers, too, as with bats(7). Giving the standard library a name like batslib might make this easier.\r\n\r\nAgreed. I'm also thinking of rewriting the documentation to be more concise (e.g. man page style), we'll see how it works out.\r\n\r\nMost of these are already done. Once I finish the man page I'll update the pull request. Hopefully by Wednesday.\r\n\r\n*P.S.: Sorry for the delay! Life happened. I'm back on working on this pull request. Replies should be quicker now. :)*"}}