{"action":"created","comment":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/41554480","id":41554480,"diff_hunk":"@@ -239,6 +239,26 @@ end\n abstract AbstractTestSet\n \n \"\"\"\n+    record(ts::AbstractTestSet, res::Result)\n+\n+Record a result to a testset. This function is called by the `@testset`\n+infrastructure each time a contained `@test` macro completes, and is given the\n+test result (which could be an `Error`). This will also be called with an `Error`\n+if an exception is thrown inside the test block but outside of a `@test` context.\n+\"\"\"\n+function record end","path":"base/test.jl","position":11,"original_position":11,"commit_id":"2acdb837de026fd205f6ef6cc5f52e55b9cdeae0","original_commit_id":"2acdb837de026fd205f6ef6cc5f52e55b9cdeae0","user":{"login":"IainNZ","id":692635,"avatar_url":"https://avatars.githubusercontent.com/u/692635?v=3","gravatar_id":"","url":"https://api.github.com/users/IainNZ","html_url":"https://github.com/IainNZ","followers_url":"https://api.github.com/users/IainNZ/followers","following_url":"https://api.github.com/users/IainNZ/following{/other_user}","gists_url":"https://api.github.com/users/IainNZ/gists{/gist_id}","starred_url":"https://api.github.com/users/IainNZ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/IainNZ/subscriptions","organizations_url":"https://api.github.com/users/IainNZ/orgs","repos_url":"https://api.github.com/users/IainNZ/repos","events_url":"https://api.github.com/users/IainNZ/events{/privacy}","received_events_url":"https://api.github.com/users/IainNZ/received_events","type":"User","site_admin":false},"body":"Oh wow, I did not know that this was how it worked! Interesting...","created_at":"2015-10-08T19:05:49Z","updated_at":"2015-10-08T19:05:49Z","html_url":"https://github.com/JuliaLang/julia/pull/13470#discussion_r41554480","pull_request_url":"https://api.github.com/repos/JuliaLang/julia/pulls/13470","_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments/41554480"},"html":{"href":"https://github.com/JuliaLang/julia/pull/13470#discussion_r41554480"},"pull_request":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/13470"}}},"pull_request":{"url":"https://api.github.com/repos/JuliaLang/julia/pulls/13470","id":46950186,"html_url":"https://github.com/JuliaLang/julia/pull/13470","diff_url":"https://github.com/JuliaLang/julia/pull/13470.diff","patch_url":"https://github.com/JuliaLang/julia/pull/13470.patch","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/13470","number":13470,"state":"open","locked":false,"title":"Adds custom testset type API","user":{"login":"ssfrr","id":734386,"avatar_url":"https://avatars.githubusercontent.com/u/734386?v=3","gravatar_id":"","url":"https://api.github.com/users/ssfrr","html_url":"https://github.com/ssfrr","followers_url":"https://api.github.com/users/ssfrr/followers","following_url":"https://api.github.com/users/ssfrr/following{/other_user}","gists_url":"https://api.github.com/users/ssfrr/gists{/gist_id}","starred_url":"https://api.github.com/users/ssfrr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssfrr/subscriptions","organizations_url":"https://api.github.com/users/ssfrr/orgs","repos_url":"https://api.github.com/users/ssfrr/repos","events_url":"https://api.github.com/users/ssfrr/events{/privacy}","received_events_url":"https://api.github.com/users/ssfrr/received_events","type":"User","site_admin":false},"body":"This PR creates an API for packages to create their own `AbstractTestSet`\r\nsubtypes. `Base.Test` takes responsibility for maintaining the stack of\r\nnested testsets as they are executed, but any result accumulation is the\r\nresponsibility of the `AbstractTestSet` subtype. `Base.Test` also makes\r\nsure that nested `@testset` invocations use the same `AbstractTestSet`\r\nsubtype as their parent unless it is set explicitly, but it does not\r\npropogate any properties of the testset. If a package wants to inherit\r\nproperties from a parent it is responsible for implementing that (which\r\nshould be pretty easy using the stack infrastructure that `Base.Test`\r\nprovides). Propagating the options would have required more complexity,\r\nand maintaining an option stack, and it's not clear that it's the right\r\nbehavior for all the subtypes.\r\n\r\nDefining a basic `AbstractTestSet` subtype looks like:\r\n\r\n```julia\r\nimport Base.Test: record, finish, get_testset_depth, get_testset\r\nimport Base.Test: AbstractTestSet, Result, Pass, Fail, Error\r\nimmutable CustomTestSet <: Base.Test.AbstractTestSet\r\n    description::AbstractString\r\n    foo::Int\r\n    results::Vector\r\n    # constructor takes a description string and options keyword arguments\r\n    CustomTestSet(desc; foo=1) = new(desc, foo, [])\r\nend\r\n\r\nrecord(ts::CustomTestSet, child::AbstractTestSet) = push!(ts.results, child)\r\nrecord(ts::CustomTestSet, res::Result) = push!(ts.results, res)\r\nfunction finish(ts::CustomTestSet)\r\n    # just record if we're not the top-level parent\r\n    if get_testset_depth() > 0\r\n        record(get_testset(), ts)\r\n    end\r\n    ts\r\nend\r\n```\r\n\r\nAnd using that testset looks like\r\n\r\n```julia\r\n@testset CustomTestSet foo=4 \"custom testset inner 2\" begin\r\n    # this testset should inherit the type, but not the argument.\r\n    @testset \"custom testset inner\" begin\r\n        @test true\r\n    end\r\nend\r\n```\r\n\r\nIt also includes some other small changes:\r\n\r\n* Makes `finish(::DefaultTestType)` return the testset object so it gets\r\n  returned from the `@testset` macro\r\n* now `@testloop` returns an Array of testsets (or whatever is returned from\r\n  the `finish` method of each invocation in the loop)\r\n\r\nStill TODO:\r\n\r\n- [x] Add docs on how to create your own testset\r\n- [x] Update docstrings for `@testset` and `@testloop` to document return values\r\n- [x] Decide on what should be exported. Currently the methods you want to\r\n    import to create your own `AbstractTestSet` are not exported\r\n\r\n@IainNZ","created_at":"2015-10-06T20:00:31Z","updated_at":"2015-10-08T19:05:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ad356519f033200155e7535176ac844ca8952e62","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/JuliaLang/julia/pulls/13470/commits","review_comments_url":"https://api.github.com/repos/JuliaLang/julia/pulls/13470/comments","review_comment_url":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/13470/comments","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/2acdb837de026fd205f6ef6cc5f52e55b9cdeae0","head":{"label":"ssfrr:custom-testset","ref":"custom-testset","sha":"2acdb837de026fd205f6ef6cc5f52e55b9cdeae0","user":{"login":"ssfrr","id":734386,"avatar_url":"https://avatars.githubusercontent.com/u/734386?v=3","gravatar_id":"","url":"https://api.github.com/users/ssfrr","html_url":"https://github.com/ssfrr","followers_url":"https://api.github.com/users/ssfrr/followers","following_url":"https://api.github.com/users/ssfrr/following{/other_user}","gists_url":"https://api.github.com/users/ssfrr/gists{/gist_id}","starred_url":"https://api.github.com/users/ssfrr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssfrr/subscriptions","organizations_url":"https://api.github.com/users/ssfrr/orgs","repos_url":"https://api.github.com/users/ssfrr/repos","events_url":"https://api.github.com/users/ssfrr/events{/privacy}","received_events_url":"https://api.github.com/users/ssfrr/received_events","type":"User","site_admin":false},"repo":{"id":15365228,"name":"julia","full_name":"ssfrr/julia","owner":{"login":"ssfrr","id":734386,"avatar_url":"https://avatars.githubusercontent.com/u/734386?v=3","gravatar_id":"","url":"https://api.github.com/users/ssfrr","html_url":"https://github.com/ssfrr","followers_url":"https://api.github.com/users/ssfrr/followers","following_url":"https://api.github.com/users/ssfrr/following{/other_user}","gists_url":"https://api.github.com/users/ssfrr/gists{/gist_id}","starred_url":"https://api.github.com/users/ssfrr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssfrr/subscriptions","organizations_url":"https://api.github.com/users/ssfrr/orgs","repos_url":"https://api.github.com/users/ssfrr/repos","events_url":"https://api.github.com/users/ssfrr/events{/privacy}","received_events_url":"https://api.github.com/users/ssfrr/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ssfrr/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":true,"url":"https://api.github.com/repos/ssfrr/julia","forks_url":"https://api.github.com/repos/ssfrr/julia/forks","keys_url":"https://api.github.com/repos/ssfrr/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ssfrr/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ssfrr/julia/teams","hooks_url":"https://api.github.com/repos/ssfrr/julia/hooks","issue_events_url":"https://api.github.com/repos/ssfrr/julia/issues/events{/number}","events_url":"https://api.github.com/repos/ssfrr/julia/events","assignees_url":"https://api.github.com/repos/ssfrr/julia/assignees{/user}","branches_url":"https://api.github.com/repos/ssfrr/julia/branches{/branch}","tags_url":"https://api.github.com/repos/ssfrr/julia/tags","blobs_url":"https://api.github.com/repos/ssfrr/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ssfrr/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ssfrr/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/ssfrr/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ssfrr/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/ssfrr/julia/languages","stargazers_url":"https://api.github.com/repos/ssfrr/julia/stargazers","contributors_url":"https://api.github.com/repos/ssfrr/julia/contributors","subscribers_url":"https://api.github.com/repos/ssfrr/julia/subscribers","subscription_url":"https://api.github.com/repos/ssfrr/julia/subscription","commits_url":"https://api.github.com/repos/ssfrr/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/ssfrr/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/ssfrr/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/ssfrr/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/ssfrr/julia/contents/{+path}","compare_url":"https://api.github.com/repos/ssfrr/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ssfrr/julia/merges","archive_url":"https://api.github.com/repos/ssfrr/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ssfrr/julia/downloads","issues_url":"https://api.github.com/repos/ssfrr/julia/issues{/number}","pulls_url":"https://api.github.com/repos/ssfrr/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/ssfrr/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/ssfrr/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ssfrr/julia/labels{/name}","releases_url":"https://api.github.com/repos/ssfrr/julia/releases{/id}","created_at":"2013-12-21T20:31:36Z","updated_at":"2014-07-29T03:39:06Z","pushed_at":"2015-10-08T17:50:39Z","git_url":"git://github.com/ssfrr/julia.git","ssh_url":"git@github.com:ssfrr/julia.git","clone_url":"https://github.com/ssfrr/julia.git","svn_url":"https://github.com/ssfrr/julia","homepage":"http://julialang.org/","size":84561,"stargazers_count":0,"watchers_count":0,"language":"Julia","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"JuliaLang:master","ref":"master","sha":"128f8c6d009338a240673a3cbc6d2279e517a8d3","user":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"repo":{"id":1644196,"name":"julia","full_name":"JuliaLang/julia","owner":{"login":"JuliaLang","id":743164,"avatar_url":"https://avatars.githubusercontent.com/u/743164?v=3","gravatar_id":"","url":"https://api.github.com/users/JuliaLang","html_url":"https://github.com/JuliaLang","followers_url":"https://api.github.com/users/JuliaLang/followers","following_url":"https://api.github.com/users/JuliaLang/following{/other_user}","gists_url":"https://api.github.com/users/JuliaLang/gists{/gist_id}","starred_url":"https://api.github.com/users/JuliaLang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JuliaLang/subscriptions","organizations_url":"https://api.github.com/users/JuliaLang/orgs","repos_url":"https://api.github.com/users/JuliaLang/repos","events_url":"https://api.github.com/users/JuliaLang/events{/privacy}","received_events_url":"https://api.github.com/users/JuliaLang/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/JuliaLang/julia","description":"The Julia Language: A fresh approach to technical computing.","fork":false,"url":"https://api.github.com/repos/JuliaLang/julia","forks_url":"https://api.github.com/repos/JuliaLang/julia/forks","keys_url":"https://api.github.com/repos/JuliaLang/julia/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JuliaLang/julia/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JuliaLang/julia/teams","hooks_url":"https://api.github.com/repos/JuliaLang/julia/hooks","issue_events_url":"https://api.github.com/repos/JuliaLang/julia/issues/events{/number}","events_url":"https://api.github.com/repos/JuliaLang/julia/events","assignees_url":"https://api.github.com/repos/JuliaLang/julia/assignees{/user}","branches_url":"https://api.github.com/repos/JuliaLang/julia/branches{/branch}","tags_url":"https://api.github.com/repos/JuliaLang/julia/tags","blobs_url":"https://api.github.com/repos/JuliaLang/julia/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JuliaLang/julia/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JuliaLang/julia/git/refs{/sha}","trees_url":"https://api.github.com/repos/JuliaLang/julia/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JuliaLang/julia/statuses/{sha}","languages_url":"https://api.github.com/repos/JuliaLang/julia/languages","stargazers_url":"https://api.github.com/repos/JuliaLang/julia/stargazers","contributors_url":"https://api.github.com/repos/JuliaLang/julia/contributors","subscribers_url":"https://api.github.com/repos/JuliaLang/julia/subscribers","subscription_url":"https://api.github.com/repos/JuliaLang/julia/subscription","commits_url":"https://api.github.com/repos/JuliaLang/julia/commits{/sha}","git_commits_url":"https://api.github.com/repos/JuliaLang/julia/git/commits{/sha}","comments_url":"https://api.github.com/repos/JuliaLang/julia/comments{/number}","issue_comment_url":"https://api.github.com/repos/JuliaLang/julia/issues/comments{/number}","contents_url":"https://api.github.com/repos/JuliaLang/julia/contents/{+path}","compare_url":"https://api.github.com/repos/JuliaLang/julia/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JuliaLang/julia/merges","archive_url":"https://api.github.com/repos/JuliaLang/julia/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JuliaLang/julia/downloads","issues_url":"https://api.github.com/repos/JuliaLang/julia/issues{/number}","pulls_url":"https://api.github.com/repos/JuliaLang/julia/pulls{/number}","milestones_url":"https://api.github.com/repos/JuliaLang/julia/milestones{/number}","notifications_url":"https://api.github.com/repos/JuliaLang/julia/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JuliaLang/julia/labels{/name}","releases_url":"https://api.github.com/repos/JuliaLang/julia/releases{/id}","created_at":"2011-04-21T07:01:50Z","updated_at":"2015-10-08T16:41:59Z","pushed_at":"2015-10-08T18:51:25Z","git_url":"git://github.com/JuliaLang/julia.git","ssh_url":"git@github.com:JuliaLang/julia.git","clone_url":"https://github.com/JuliaLang/julia.git","svn_url":"https://github.com/JuliaLang/julia","homepage":"http://julialang.org/","size":232454,"stargazers_count":6096,"watchers_count":6096,"language":"Julia","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1337,"mirror_url":null,"open_issues_count":1443,"forks":1337,"open_issues":1443,"watchers":6096,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/13470"},"html":{"href":"https://github.com/JuliaLang/julia/pull/13470"},"issue":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/13470"},"comments":{"href":"https://api.github.com/repos/JuliaLang/julia/issues/13470/comments"},"review_comments":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/13470/comments"},"review_comment":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/JuliaLang/julia/pulls/13470/commits"},"statuses":{"href":"https://api.github.com/repos/JuliaLang/julia/statuses/2acdb837de026fd205f6ef6cc5f52e55b9cdeae0"}}}}