{"action":"created","comment":{"url":"https://api.github.com/repos/Clever/dev-handbook/pulls/comments/35555971","id":35555971,"diff_hunk":"@@ -34,6 +34,82 @@ A piece of code is not done until it's tested.\n \n So just write tests. Always.\n \n+## Testing Principles\n+\n+Things you should keep in mind while writing tests for a codebase.\n+\n+### Write Complete Testing Suites\n+\n+Testing suites should attempt to be exhaustive in verifying the code does what is expected. None of\n+these tests should require any manual work, other than executing the initial command to run the\n+tests.\n+\n+An exhaustive test suite implies that you should be confident of merging and deploying your changes\n+once all your tests pass in your continuous integration environment. Any manual steps that are needed\n+to verify code changes should be encoded as automated tests that run in your continuous integration\n+environment.  As an example, if to test your code, you bring up an instance of your service and\n+manually send it an example request, then this flow should be encoded as an integration test.\n+\n+### Test Deterministically\n+\n+Tests should be deterministic. That means that if you run a test multiple times without changing the\n+code, it should produce the same results. A test that sometimes fails and sometimes passes is not\n+consistent, and therefore can't be relied on to give you confidence about the correctness of a\n+program.\n+\n+Concretely, this means you should avoid relying on factors like randomness, time, or external\n+services in the structure of your tests. For instance, if you are testing a button that changes a\n+piece of UI, you shouldn't wait for an arbitrary number of seconds to check that the UI refreshes,\n+since the time that it takes to refresh is not deterministic. Instead, you can emit some sort of\n+event when the UI is finished refreshing, and make the test listen for this event.\n+\n+Testing non-deterministic code can also be very difficult. If you can, avoid non-deterministic\n+behavior in your code. If you must have it, try to isolate it so it can be deterministically mocked\n+in tests.\n+\n+One type of non-determinism that *is* useful for testing is generating random inputs.\n+[Generative testing](http://blog.8thlight.com/connor-mendenhall/2013/10/31/check-your-work.html), in\n+which a program generates different test inputs each time the tests are run, can help describe\n+properties of your program that should hold for all inputs. They can make a stronger assertion about\n+correctness, since, if they pass, it means your program works for many random inputs, not just a few\n+representative ones. Generative tests can also come up with edge cases that you may not have thought\n+of.\n+\n+### Prefer Readability and Ease of Code\n+\n+Tests should be written in a manner that prioritizes readability and ease of writing new tests and\n+modifying existing ones. In particular, these should be prioritized over other typical concerns like\n+abstraction and removing code duplication. If abstractions are needed, they should be created in a\n+manner that preserves the logic in the test functions code without requiring many hops. Another way\n+of saying this is that it should be apparent what the test is doing by just looking at the test\n+code.\n+\n+As an example, we had a testing library called which populated a bunch of fake data in the database\n+specified by the test. A lot of our tests used with a single statement of the form\n+`db_test_helpers.set_up_test_data`. This single function populated a bunch of state within the\n+database including creating students, teachers, districts, apps. However, it also hid away all the\n+details of the state being created. This meant that it became very hard to reason about tests or\n+modify your tests if the current state was not appropriate for them.\n+\n+Instead, a better approach is to provide easy abstractions for creating individual pieces of data\n+like `add_student`, `add_district` and use them as needed in different tests.\n+\n+### Ensure Tests are Independent\n+\n+Each test should be written in a manner that makes it independent of other tests. If there is any\n+state needed for a test, the test should be responsible for getting to the state before running it\n+and resetting it after execution. That is, the state before and after the test should be identical\n+whether the test passes or fails. If your tests are independent, then running the tests in any order\n+should not affect the outcome of the tests.\n+\n+A stronger version of independence could mean that there is no shared state between different tests.\n+As an example, if the test involves talking to a database, then different tests should talk to\n+different logical databases. Another example, if you are testing the behavior of a particular class,","path":"testing.md","position":74,"original_position":74,"commit_id":"5272b08e7622479be23c367a91da518dfd4eed41","original_commit_id":"5272b08e7622479be23c367a91da518dfd4eed41","user":{"login":"jonahkagan","id":530106,"avatar_url":"https://avatars.githubusercontent.com/u/530106?v=3","gravatar_id":"","url":"https://api.github.com/users/jonahkagan","html_url":"https://github.com/jonahkagan","followers_url":"https://api.github.com/users/jonahkagan/followers","following_url":"https://api.github.com/users/jonahkagan/following{/other_user}","gists_url":"https://api.github.com/users/jonahkagan/gists{/gist_id}","starred_url":"https://api.github.com/users/jonahkagan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonahkagan/subscriptions","organizations_url":"https://api.github.com/users/jonahkagan/orgs","repos_url":"https://api.github.com/users/jonahkagan/repos","events_url":"https://api.github.com/users/jonahkagan/events{/privacy}","received_events_url":"https://api.github.com/users/jonahkagan/received_events","type":"User","site_admin":false},"body":"setting up a different db for each test case seems costly in terms of time. that's the only real advantage to shared state that i can think of. how do you balance that concern?","created_at":"2015-07-27T16:28:26Z","updated_at":"2015-07-27T16:28:26Z","html_url":"https://github.com/Clever/dev-handbook/pull/48#discussion_r35555971","pull_request_url":"https://api.github.com/repos/Clever/dev-handbook/pulls/48","_links":{"self":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/comments/35555971"},"html":{"href":"https://github.com/Clever/dev-handbook/pull/48#discussion_r35555971"},"pull_request":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/48"}}},"pull_request":{"url":"https://api.github.com/repos/Clever/dev-handbook/pulls/48","id":40720349,"html_url":"https://github.com/Clever/dev-handbook/pull/48","diff_url":"https://github.com/Clever/dev-handbook/pull/48.diff","patch_url":"https://github.com/Clever/dev-handbook/pull/48.patch","issue_url":"https://api.github.com/repos/Clever/dev-handbook/issues/48","number":48,"state":"open","locked":false,"title":"Add a testing principles section to dev handbook","user":{"login":"agclever","id":10036433,"avatar_url":"https://avatars.githubusercontent.com/u/10036433?v=3","gravatar_id":"","url":"https://api.github.com/users/agclever","html_url":"https://github.com/agclever","followers_url":"https://api.github.com/users/agclever/followers","following_url":"https://api.github.com/users/agclever/following{/other_user}","gists_url":"https://api.github.com/users/agclever/gists{/gist_id}","starred_url":"https://api.github.com/users/agclever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agclever/subscriptions","organizations_url":"https://api.github.com/users/agclever/orgs","repos_url":"https://api.github.com/users/agclever/repos","events_url":"https://api.github.com/users/agclever/events{/privacy}","received_events_url":"https://api.github.com/users/agclever/received_events","type":"User","site_admin":false},"body":"This is essentially moving some of the ideas I added in https://docs.google.com/document/d/1_TQrSeTocq-sROzI0fBKPZ-MRiMmoY1F0UsS9jt_a64/edit# to the dev-handbook.\r\n\r\nI added a testing principles section and moved the section on testing deterministically from the `how` section. The rest of the stuff in the `how` section are ways of achieving the principles effectively as opposed to being more fundamental principles in my opinion.\r\n\r\nThe idea for this PR is two fold:\r\n* Document these principles in a place that are easily accessible to existing and new clever engineers.\r\n* Use this section as a template for a future post on the clever engineering blog.","created_at":"2015-07-23T21:26:46Z","updated_at":"2015-07-27T16:28:26Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8d00a2566604de99f75b53509c49f3b1df85fa4a","assignee":{"login":"schimmy","id":603357,"avatar_url":"https://avatars.githubusercontent.com/u/603357?v=3","gravatar_id":"","url":"https://api.github.com/users/schimmy","html_url":"https://github.com/schimmy","followers_url":"https://api.github.com/users/schimmy/followers","following_url":"https://api.github.com/users/schimmy/following{/other_user}","gists_url":"https://api.github.com/users/schimmy/gists{/gist_id}","starred_url":"https://api.github.com/users/schimmy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schimmy/subscriptions","organizations_url":"https://api.github.com/users/schimmy/orgs","repos_url":"https://api.github.com/users/schimmy/repos","events_url":"https://api.github.com/users/schimmy/events{/privacy}","received_events_url":"https://api.github.com/users/schimmy/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/Clever/dev-handbook/pulls/48/commits","review_comments_url":"https://api.github.com/repos/Clever/dev-handbook/pulls/48/comments","review_comment_url":"https://api.github.com/repos/Clever/dev-handbook/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Clever/dev-handbook/issues/48/comments","statuses_url":"https://api.github.com/repos/Clever/dev-handbook/statuses/5272b08e7622479be23c367a91da518dfd4eed41","head":{"label":"Clever:testing-principles","ref":"testing-principles","sha":"5272b08e7622479be23c367a91da518dfd4eed41","user":{"login":"Clever","id":1885509,"avatar_url":"https://avatars.githubusercontent.com/u/1885509?v=3","gravatar_id":"","url":"https://api.github.com/users/Clever","html_url":"https://github.com/Clever","followers_url":"https://api.github.com/users/Clever/followers","following_url":"https://api.github.com/users/Clever/following{/other_user}","gists_url":"https://api.github.com/users/Clever/gists{/gist_id}","starred_url":"https://api.github.com/users/Clever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Clever/subscriptions","organizations_url":"https://api.github.com/users/Clever/orgs","repos_url":"https://api.github.com/users/Clever/repos","events_url":"https://api.github.com/users/Clever/events{/privacy}","received_events_url":"https://api.github.com/users/Clever/received_events","type":"Organization","site_admin":false},"repo":{"id":16847837,"name":"dev-handbook","full_name":"Clever/dev-handbook","owner":{"login":"Clever","id":1885509,"avatar_url":"https://avatars.githubusercontent.com/u/1885509?v=3","gravatar_id":"","url":"https://api.github.com/users/Clever","html_url":"https://github.com/Clever","followers_url":"https://api.github.com/users/Clever/followers","following_url":"https://api.github.com/users/Clever/following{/other_user}","gists_url":"https://api.github.com/users/Clever/gists{/gist_id}","starred_url":"https://api.github.com/users/Clever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Clever/subscriptions","organizations_url":"https://api.github.com/users/Clever/orgs","repos_url":"https://api.github.com/users/Clever/repos","events_url":"https://api.github.com/users/Clever/events{/privacy}","received_events_url":"https://api.github.com/users/Clever/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Clever/dev-handbook","description":"A guide to the processes, conventions, and philosophies of the Clever dev team.","fork":false,"url":"https://api.github.com/repos/Clever/dev-handbook","forks_url":"https://api.github.com/repos/Clever/dev-handbook/forks","keys_url":"https://api.github.com/repos/Clever/dev-handbook/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Clever/dev-handbook/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Clever/dev-handbook/teams","hooks_url":"https://api.github.com/repos/Clever/dev-handbook/hooks","issue_events_url":"https://api.github.com/repos/Clever/dev-handbook/issues/events{/number}","events_url":"https://api.github.com/repos/Clever/dev-handbook/events","assignees_url":"https://api.github.com/repos/Clever/dev-handbook/assignees{/user}","branches_url":"https://api.github.com/repos/Clever/dev-handbook/branches{/branch}","tags_url":"https://api.github.com/repos/Clever/dev-handbook/tags","blobs_url":"https://api.github.com/repos/Clever/dev-handbook/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Clever/dev-handbook/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Clever/dev-handbook/git/refs{/sha}","trees_url":"https://api.github.com/repos/Clever/dev-handbook/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Clever/dev-handbook/statuses/{sha}","languages_url":"https://api.github.com/repos/Clever/dev-handbook/languages","stargazers_url":"https://api.github.com/repos/Clever/dev-handbook/stargazers","contributors_url":"https://api.github.com/repos/Clever/dev-handbook/contributors","subscribers_url":"https://api.github.com/repos/Clever/dev-handbook/subscribers","subscription_url":"https://api.github.com/repos/Clever/dev-handbook/subscription","commits_url":"https://api.github.com/repos/Clever/dev-handbook/commits{/sha}","git_commits_url":"https://api.github.com/repos/Clever/dev-handbook/git/commits{/sha}","comments_url":"https://api.github.com/repos/Clever/dev-handbook/comments{/number}","issue_comment_url":"https://api.github.com/repos/Clever/dev-handbook/issues/comments{/number}","contents_url":"https://api.github.com/repos/Clever/dev-handbook/contents/{+path}","compare_url":"https://api.github.com/repos/Clever/dev-handbook/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Clever/dev-handbook/merges","archive_url":"https://api.github.com/repos/Clever/dev-handbook/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Clever/dev-handbook/downloads","issues_url":"https://api.github.com/repos/Clever/dev-handbook/issues{/number}","pulls_url":"https://api.github.com/repos/Clever/dev-handbook/pulls{/number}","milestones_url":"https://api.github.com/repos/Clever/dev-handbook/milestones{/number}","notifications_url":"https://api.github.com/repos/Clever/dev-handbook/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Clever/dev-handbook/labels{/name}","releases_url":"https://api.github.com/repos/Clever/dev-handbook/releases{/id}","created_at":"2014-02-14T19:58:08Z","updated_at":"2015-07-16T09:25:34Z","pushed_at":"2015-07-24T22:11:32Z","git_url":"git://github.com/Clever/dev-handbook.git","ssh_url":"git@github.com:Clever/dev-handbook.git","clone_url":"https://github.com/Clever/dev-handbook.git","svn_url":"https://github.com/Clever/dev-handbook","homepage":"","size":600,"stargazers_count":22,"watchers_count":22,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":13,"forks":4,"open_issues":13,"watchers":22,"default_branch":"master"}},"base":{"label":"Clever:master","ref":"master","sha":"8ef6bfd4591c5a3b37c089bb8c769417f805b0b4","user":{"login":"Clever","id":1885509,"avatar_url":"https://avatars.githubusercontent.com/u/1885509?v=3","gravatar_id":"","url":"https://api.github.com/users/Clever","html_url":"https://github.com/Clever","followers_url":"https://api.github.com/users/Clever/followers","following_url":"https://api.github.com/users/Clever/following{/other_user}","gists_url":"https://api.github.com/users/Clever/gists{/gist_id}","starred_url":"https://api.github.com/users/Clever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Clever/subscriptions","organizations_url":"https://api.github.com/users/Clever/orgs","repos_url":"https://api.github.com/users/Clever/repos","events_url":"https://api.github.com/users/Clever/events{/privacy}","received_events_url":"https://api.github.com/users/Clever/received_events","type":"Organization","site_admin":false},"repo":{"id":16847837,"name":"dev-handbook","full_name":"Clever/dev-handbook","owner":{"login":"Clever","id":1885509,"avatar_url":"https://avatars.githubusercontent.com/u/1885509?v=3","gravatar_id":"","url":"https://api.github.com/users/Clever","html_url":"https://github.com/Clever","followers_url":"https://api.github.com/users/Clever/followers","following_url":"https://api.github.com/users/Clever/following{/other_user}","gists_url":"https://api.github.com/users/Clever/gists{/gist_id}","starred_url":"https://api.github.com/users/Clever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Clever/subscriptions","organizations_url":"https://api.github.com/users/Clever/orgs","repos_url":"https://api.github.com/users/Clever/repos","events_url":"https://api.github.com/users/Clever/events{/privacy}","received_events_url":"https://api.github.com/users/Clever/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Clever/dev-handbook","description":"A guide to the processes, conventions, and philosophies of the Clever dev team.","fork":false,"url":"https://api.github.com/repos/Clever/dev-handbook","forks_url":"https://api.github.com/repos/Clever/dev-handbook/forks","keys_url":"https://api.github.com/repos/Clever/dev-handbook/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Clever/dev-handbook/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Clever/dev-handbook/teams","hooks_url":"https://api.github.com/repos/Clever/dev-handbook/hooks","issue_events_url":"https://api.github.com/repos/Clever/dev-handbook/issues/events{/number}","events_url":"https://api.github.com/repos/Clever/dev-handbook/events","assignees_url":"https://api.github.com/repos/Clever/dev-handbook/assignees{/user}","branches_url":"https://api.github.com/repos/Clever/dev-handbook/branches{/branch}","tags_url":"https://api.github.com/repos/Clever/dev-handbook/tags","blobs_url":"https://api.github.com/repos/Clever/dev-handbook/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Clever/dev-handbook/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Clever/dev-handbook/git/refs{/sha}","trees_url":"https://api.github.com/repos/Clever/dev-handbook/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Clever/dev-handbook/statuses/{sha}","languages_url":"https://api.github.com/repos/Clever/dev-handbook/languages","stargazers_url":"https://api.github.com/repos/Clever/dev-handbook/stargazers","contributors_url":"https://api.github.com/repos/Clever/dev-handbook/contributors","subscribers_url":"https://api.github.com/repos/Clever/dev-handbook/subscribers","subscription_url":"https://api.github.com/repos/Clever/dev-handbook/subscription","commits_url":"https://api.github.com/repos/Clever/dev-handbook/commits{/sha}","git_commits_url":"https://api.github.com/repos/Clever/dev-handbook/git/commits{/sha}","comments_url":"https://api.github.com/repos/Clever/dev-handbook/comments{/number}","issue_comment_url":"https://api.github.com/repos/Clever/dev-handbook/issues/comments{/number}","contents_url":"https://api.github.com/repos/Clever/dev-handbook/contents/{+path}","compare_url":"https://api.github.com/repos/Clever/dev-handbook/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Clever/dev-handbook/merges","archive_url":"https://api.github.com/repos/Clever/dev-handbook/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Clever/dev-handbook/downloads","issues_url":"https://api.github.com/repos/Clever/dev-handbook/issues{/number}","pulls_url":"https://api.github.com/repos/Clever/dev-handbook/pulls{/number}","milestones_url":"https://api.github.com/repos/Clever/dev-handbook/milestones{/number}","notifications_url":"https://api.github.com/repos/Clever/dev-handbook/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Clever/dev-handbook/labels{/name}","releases_url":"https://api.github.com/repos/Clever/dev-handbook/releases{/id}","created_at":"2014-02-14T19:58:08Z","updated_at":"2015-07-16T09:25:34Z","pushed_at":"2015-07-24T22:11:32Z","git_url":"git://github.com/Clever/dev-handbook.git","ssh_url":"git@github.com:Clever/dev-handbook.git","clone_url":"https://github.com/Clever/dev-handbook.git","svn_url":"https://github.com/Clever/dev-handbook","homepage":"","size":600,"stargazers_count":22,"watchers_count":22,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":13,"forks":4,"open_issues":13,"watchers":22,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/48"},"html":{"href":"https://github.com/Clever/dev-handbook/pull/48"},"issue":{"href":"https://api.github.com/repos/Clever/dev-handbook/issues/48"},"comments":{"href":"https://api.github.com/repos/Clever/dev-handbook/issues/48/comments"},"review_comments":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/48/comments"},"review_comment":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Clever/dev-handbook/pulls/48/commits"},"statuses":{"href":"https://api.github.com/repos/Clever/dev-handbook/statuses/5272b08e7622479be23c367a91da518dfd4eed41"}}}}