{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/server/issues/208","labels_url":"https://api.github.com/repos/ga4gh/server/issues/208/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/208/comments","events_url":"https://api.github.com/repos/ga4gh/server/issues/208/events","html_url":"https://github.com/ga4gh/server/pull/208","id":59804001,"number":208,"title":"Framework for data driven tests.","user":{"login":"jeromekelleher","id":2664569,"avatar_url":"https://avatars.githubusercontent.com/u/2664569?v=3","gravatar_id":"","url":"https://api.github.com/users/jeromekelleher","html_url":"https://github.com/jeromekelleher","followers_url":"https://api.github.com/users/jeromekelleher/followers","following_url":"https://api.github.com/users/jeromekelleher/following{/other_user}","gists_url":"https://api.github.com/users/jeromekelleher/gists{/gist_id}","starred_url":"https://api.github.com/users/jeromekelleher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeromekelleher/subscriptions","organizations_url":"https://api.github.com/users/jeromekelleher/orgs","repos_url":"https://api.github.com/users/jeromekelleher/repos","events_url":"https://api.github.com/users/jeromekelleher/events{/privacy}","received_events_url":"https://api.github.com/users/jeromekelleher/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-03-04T14:23:56Z","updated_at":"2015-03-04T19:26:49Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ga4gh/server/pulls/208","html_url":"https://github.com/ga4gh/server/pull/208","diff_url":"https://github.com/ga4gh/server/pull/208.diff","patch_url":"https://github.com/ga4gh/server/pull/208.patch"},"body":"This PR adds the initial frameworks required for data driven tests, and some examples for variants and references. These tests work as follows:\r\n\r\n1. We have a number of modules within the `tests/datadriven` module. Each module has a top level function that is a generator over the test cases that are run. This uses nose's ability to use generators that yield individual test cases (see http://nose.readthedocs.org/en/latest/writing_tests.html). Each test that we run is a effectively a pair (testMethod, dataDir).\r\n2. In each module, we have one or more TestClasses, and we allocate an instance of this class for each data directory. The test class builds a model of the data in this directory and also allocates the corresponding object from `ga4gh.datamodel`. We then compare these two models, and ensure that they are consistent.\r\n3. Each TestClass has a set of test methods, and each of these is called exactly once. Note that this does violate the idea that unit tests should be entirely independent of each other. However, with a little programming discipline I think we can make these tests effectively independent. The cost of building the alternative model of the data in question is not trivial, and it seems pointless to mandate that this is done before every single test.\r\n\r\nHere is an example of what a run looks like:\r\n```\r\n$ nosetests -vs tests/datadriven/\r\ntests.datadriven.test_references.ReferenceSetTest.testNumReferences:GRCh38 ... ok\r\ntests.datadriven.ReferenceSetTest.testProtocolElementValid:GRCh38 ... ok\r\ntests.datadriven.test_references.ReferenceSetTest.testNumReferences:GRCh38_copy ... ok\r\ntests.datadriven.ReferenceSetTest.testProtocolElementValid:GRCh38_copy ... ok\r\ntests.datadriven.test_references.ReferenceSetTest.testNumReferences:example_1 ... ok\r\ntests.datadriven.ReferenceSetTest.testProtocolElementValid:example_1 ... ok\r\ntests.datadriven.VariantSetTest.testProtocolElementValid:example_1 ... ok\r\ntests.datadriven.test_variants.VariantSetTest.testSearchAllVariants:example_1 ... ok\r\ntests.datadriven.VariantSetTest.testProtocolElementValid:example_2 ... ok\r\ntests.datadriven.test_variants.VariantSetTest.testSearchAllVariants:example_2 ... ok\r\ntests.datadriven.VariantSetTest.testProtocolElementValid:example_3 ... ok\r\ntests.datadriven.test_variants.VariantSetTest.testSearchAllVariants:example_3 ... ok\r\ntests.datadriven.VariantSetTest.testProtocolElementValid:example_4 ... ok\r\ntests.datadriven.test_variants.VariantSetTest.testSearchAllVariants:example_4 ... ok\r\n\r\n----------------------------------------------------------------------\r\nRan 14 tests in 0.144s\r\n```\r\n\r\nWhy not use standard unit tests? Unit tests are very valuable and we should continue to make lots of them. However, the principle thing that our server does is interact with data that it is provided with and serve this out to users. We will get data in all sorts of weird and wonderful formats with all sorts of quirks. We need to be able to deal with this, and to build in test cases to ensure we handle these quirks properly. When we find a data file that breaks our code, we add an example data file to our tests data, and the data driven tests automatically pick it up and run it.\r\n\r\nThe main disadvantage of this approach is that we can't extent `unittest.TestCase`, and so we lose our nice `self.assertX` methods. This is unfortunate, and is because of the way nose test generators work. We can possibly work around this with some nasty hackery, but I'm not sure if it's a good idea.\r\n\r\nI'd hope to make this a fairly fundamental piece of infrastructure for how we move forward in terms of building out reads, variants and references support, so please review this PR and let me know what you think.\r\n"},"comment":{"url":"https://api.github.com/repos/ga4gh/server/issues/comments/77227586","html_url":"https://github.com/ga4gh/server/pull/208#issuecomment-77227586","issue_url":"https://api.github.com/repos/ga4gh/server/issues/208","id":77227586,"user":{"login":"diekhans","id":110053,"avatar_url":"https://avatars.githubusercontent.com/u/110053?v=3","gravatar_id":"","url":"https://api.github.com/users/diekhans","html_url":"https://github.com/diekhans","followers_url":"https://api.github.com/users/diekhans/followers","following_url":"https://api.github.com/users/diekhans/following{/other_user}","gists_url":"https://api.github.com/users/diekhans/gists{/gist_id}","starred_url":"https://api.github.com/users/diekhans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diekhans/subscriptions","organizations_url":"https://api.github.com/users/diekhans/orgs","repos_url":"https://api.github.com/users/diekhans/repos","events_url":"https://api.github.com/users/diekhans/events{/privacy}","received_events_url":"https://api.github.com/users/diekhans/received_events","type":"User","site_admin":false},"created_at":"2015-03-04T19:26:49Z","updated_at":"2015-03-04T19:26:49Z","body":"\nI would argue for a more human-editable format than the INSERTS\nand then a program to load the SQLite database.\n\nMaciek Smuga-Otto <notifications@github.com> writes:\n> @diekhans I'm thinking a little bit ahead to how we may be storing graph data\n> for testing v0.6.x of the API (which I believe is essential for the development\n> effort). A SQLite based dataset could always be represented by its creation\n> script (CREATE TABLEs followed by INSERTs), and an AVRO based one can be\n> expressed in JSON, so I think we'd be able to adhere to an ASCII-only policy if\n> that's what ends up being mandated.\n> \n> —\n> Reply to this email directly or view it on GitHub.*\n> \n"}}