{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/9493","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/9493/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/9493/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/9493/events","html_url":"https://github.com/JuliaLang/julia/issues/9493","id":53087620,"number":9493,"title":"More comprehensive test coverage","user":{"login":"timholy","id":1525481,"avatar_url":"https://avatars.githubusercontent.com/u/1525481?v=3","gravatar_id":"","url":"https://api.github.com/users/timholy","html_url":"https://github.com/timholy","followers_url":"https://api.github.com/users/timholy/followers","following_url":"https://api.github.com/users/timholy/following{/other_user}","gists_url":"https://api.github.com/users/timholy/gists{/gist_id}","starred_url":"https://api.github.com/users/timholy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timholy/subscriptions","organizations_url":"https://api.github.com/users/timholy/orgs","repos_url":"https://api.github.com/users/timholy/repos","events_url":"https://api.github.com/users/timholy/events{/privacy}","received_events_url":"https://api.github.com/users/timholy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/JuliaLang/julia/labels/test","name":"test","color":"e102d8"},{"url":"https://api.github.com/repos/JuliaLang/julia/labels/up%20for%20grabs","name":"up for grabs","color":"0b02e1"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":157,"created_at":"2014-12-30T02:04:58Z","updated_at":"2015-05-08T02:40:45Z","closed_at":null,"body":"Updated Feb 23, 2015.\r\n\r\nThis would make a whole bunch of good \"first contribution\" projects, and does not require deep insider knowledge of the language.\r\n\r\nThe basic idea is to expand the test suite to make sure that julia's base code works as promised. Here is one recommended way to contribute toward this goal:\r\n\r\n## The easy way\r\n\r\n1. Go visit https://coveralls.io/r/JuliaLang/julia. (At the time of this writing, coverage numbers are in the 70s%. If you see numbers that are much lower than this, don't trust the reports; there is likely a problem in measuring coverage.)\r\n\r\n2. Browse through the source files and find some untested functionality (highlighted in red) that you think you might be able to write a test for. \r\n\r\n3. Write a test that exercises this functionality---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in `test/runtests.jl`. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see [CONTRIBUTING.md](https://github.com/JuliaLang/julia/blob/master/CONTRIBUTING.md)).\r\n\r\n\r\n## The manual method\r\n\r\n1. Make sure you have an up-to-date git checkout and are on the `master` branch. Build julia with `make`\r\n\r\n2. Copy/paste the following script into an editor window and save it. Here I'll use the filename `/tmp/coverage_tests.jl`:\r\n```julia\r\nusing Base.Test\r\ntestnames = [\"core\", \"keywordargs\", \"numbers\", \"strings\", \"dates\",\r\n    \"hashing\", \"remote\", \"iobuffer\", \"staged\", \"arrayops\",\r\n    \"subarray\", \"reduce\", \"reducedim\", \"random\", \"intfuncs\",\r\n    \"simdloop\", \"blas\", \"fft\", \"dsp\", \"sparse\", \"bitarray\", \"copy\", \"math\",\r\n    \"functional\", \"bigint\", \"sorting\", \"statistics\", \"spawn\",\r\n    \"backtrace\", \"priorityqueue\", \"arpack\", \"file\", \"suitesparse\", \"version\",\r\n    \"pollfd\", \"mpfr\", \"broadcast\", \"complex\", \"socket\",\r\n    \"floatapprox\", \"readdlm\", \"reflection\", \"regex\", \"float16\", \"combinatorics\",\r\n    \"sysinfo\", \"rounding\", \"ranges\", \"mod2pi\", \"euler\", \"show\",\r\n    \"lineedit\", \"replcompletions\", \"repl\", \"test\", \"goto\",\r\n    \"llvmcall\", \"grisu\", \"nullable\", \"meta\", \"profile\",\r\n    \"libgit2\", \"docs\", \"base64\", \"pkg\", \"linalg1\", \"linalg2\",\r\n\t\"linalg3\", \"linalg4\", \"linalg/lapack\", \"linalg/triangular\", \"linalg/tridiag\", \r\n\t\"linalg/pinv\", \"linalg/cholmod\", \"linalg/umfpack\", \"linalg/givens\"\r\n]\r\nfor tst in testnames\r\n    println(tst)\r\n    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,\"julia\",\"test\",\"$tst.jl\"))\r\nend\r\n```\r\n(This is the list of tests currently in `test/runtests.jl`, with 3 omissions (`resolve`, `reflection`, and `meta`) and one addition (`pkg`). The omitted tests explicitly test inlining, which we're going to disable, or are problematic when inlining is disabled.)\r\n\r\n3. From the top-level directory, type `rm usr/lib/julia/sys.so`. Deleting `sys.so` will prevent julia from using any pre-compiled functions, increasing the accuracy of the results. (This also makes startup a bit slower, but that's fine for this test---and once you're done, simply typing `make` will cause this file to be rebuilt).\r\n\r\n4. Via the command prompt, navigate to julia's `test/` directory.\r\n\r\n5. Start julia with `julia --code-coverage=all --inline=no`. This turns on code-coverage and prevents inlining (inlining makes it difficult to accurately assess whether a function has been tested).\r\n\r\n6. At the julia prompt, type `include(\"/tmp/coverage_tests.jl\")`.\r\n\r\n7. Once the tests complete successfully, quit julia\r\n\r\n8. Navigate to the `base/` directory.\r\n\r\n9. Browse through the *.jl.cov files, and look for functions (or branches within functions) that either have `0` in front of them (indicating that they were run 0 times), or have `-` in front of them (indicating that they were never compiled).\r\n\r\n10. Write a test that exercises some function that is not adequately covered---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in `test/runtests.jl`. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see [CONTRIBUTING.md](https://github.com/JuliaLang/julia/blob/master/CONTRIBUTING.md)).\r\n"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/100078217","html_url":"https://github.com/JuliaLang/julia/issues/9493#issuecomment-100078217","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/9493","id":100078217,"user":{"login":"kshyatt","id":828643,"avatar_url":"https://avatars.githubusercontent.com/u/828643?v=3","gravatar_id":"","url":"https://api.github.com/users/kshyatt","html_url":"https://github.com/kshyatt","followers_url":"https://api.github.com/users/kshyatt/followers","following_url":"https://api.github.com/users/kshyatt/following{/other_user}","gists_url":"https://api.github.com/users/kshyatt/gists{/gist_id}","starred_url":"https://api.github.com/users/kshyatt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kshyatt/subscriptions","organizations_url":"https://api.github.com/users/kshyatt/orgs","repos_url":"https://api.github.com/users/kshyatt/repos","events_url":"https://api.github.com/users/kshyatt/events{/privacy}","received_events_url":"https://api.github.com/users/kshyatt/received_events","type":"User","site_admin":false},"created_at":"2015-05-08T02:40:45Z","updated_at":"2015-05-08T02:40:45Z","body":"Awesome! Sorry I was such a :beetle: about this."}}