{"action":"created","issue":{"url":"https://api.github.com/repos/Snaipe/Criterion/issues/40","labels_url":"https://api.github.com/repos/Snaipe/Criterion/issues/40/labels{/name}","comments_url":"https://api.github.com/repos/Snaipe/Criterion/issues/40/comments","events_url":"https://api.github.com/repos/Snaipe/Criterion/issues/40/events","html_url":"https://github.com/Snaipe/Criterion/issues/40","id":106302381,"number":40,"title":"Feature Request - As a theorist, I need the TheoryDataPoint context in ASSERT report hook","user":{"login":"am11","id":3840695,"avatar_url":"https://avatars.githubusercontent.com/u/3840695?v=3","gravatar_id":"","url":"https://api.github.com/users/am11","html_url":"https://github.com/am11","followers_url":"https://api.github.com/users/am11/followers","following_url":"https://api.github.com/users/am11/following{/other_user}","gists_url":"https://api.github.com/users/am11/gists{/gist_id}","starred_url":"https://api.github.com/users/am11/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/am11/subscriptions","organizations_url":"https://api.github.com/users/am11/orgs","repos_url":"https://api.github.com/users/am11/repos","events_url":"https://api.github.com/users/am11/events{/privacy}","received_events_url":"https://api.github.com/users/am11/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Snaipe/Criterion/labels/Feature","name":"Feature","color":"2d9e11"},{"url":"https://api.github.com/repos/Snaipe/Criterion/labels/Minor","name":"Minor","color":"fffdd6"}],"state":"open","locked":false,"assignee":{"login":"Snaipe","id":977973,"avatar_url":"https://avatars.githubusercontent.com/u/977973?v=3","gravatar_id":"","url":"https://api.github.com/users/Snaipe","html_url":"https://github.com/Snaipe","followers_url":"https://api.github.com/users/Snaipe/followers","following_url":"https://api.github.com/users/Snaipe/following{/other_user}","gists_url":"https://api.github.com/users/Snaipe/gists{/gist_id}","starred_url":"https://api.github.com/users/Snaipe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Snaipe/subscriptions","organizations_url":"https://api.github.com/users/Snaipe/orgs","repos_url":"https://api.github.com/users/Snaipe/repos","events_url":"https://api.github.com/users/Snaipe/events{/privacy}","received_events_url":"https://api.github.com/users/Snaipe/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2015-09-14T09:18:21Z","updated_at":"2015-09-16T02:57:47Z","closed_at":null,"body":"Hello,\r\n\r\nIn our current effort of switching from Ruby-based test runner to Criterion, I have encountered a scenario where we are alerting the observer about a TODO test, which unexpectedly passes:\r\nhttps://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L26 &\r\nhttps://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L37\r\n\r\nIn order to replicate that effect with Criterion, we have experimented [with ASSERT report hook](https://github.com/am11/libsass/blob/2435ace5e/test/spec_theories.c#L142) which although emits the status of assertion `test->passed`, but does not provide context for Theory's arguments or expected/actual result. Analogous to `struct criterion_theory_stats` for theory, it would be of great help if the existing `struct criterion_assert_stats` or a new `struct criterion_theory_assert_stats` let us exercise:\r\n\r\n```c\r\n// Supposedly, it requires a separate `THEORY_ASSERT`\r\nReportHook(THEORY_ASSERT)(struct criterion_theory_assert_stats *test) {\r\n  if(test->passed && should_test_todo && strstr(test->formatted_args, \"-todo-\")) {\r\n    // alert unexpectedness \r\n  }\r\n}\r\n```\r\n---\r\nNitpick:\r\nOn a related note, it seems like `test->message` always contains the error string, even when `test->passed` is `true`."},"comment":{"url":"https://api.github.com/repos/Snaipe/Criterion/issues/comments/140610613","html_url":"https://github.com/Snaipe/Criterion/issues/40#issuecomment-140610613","issue_url":"https://api.github.com/repos/Snaipe/Criterion/issues/40","id":140610613,"user":{"login":"am11","id":3840695,"avatar_url":"https://avatars.githubusercontent.com/u/3840695?v=3","gravatar_id":"","url":"https://api.github.com/users/am11","html_url":"https://github.com/am11","followers_url":"https://api.github.com/users/am11/followers","following_url":"https://api.github.com/users/am11/following{/other_user}","gists_url":"https://api.github.com/users/am11/gists{/gist_id}","starred_url":"https://api.github.com/users/am11/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/am11/subscriptions","organizations_url":"https://api.github.com/users/am11/orgs","repos_url":"https://api.github.com/users/am11/repos","events_url":"https://api.github.com/users/am11/events{/privacy}","received_events_url":"https://api.github.com/users/am11/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T02:57:47Z","updated_at":"2015-09-16T02:57:47Z","body":"> The ideal solution here would be to implement parameterized tests, that would iterate over a user-generated dataset rather than the cartesian product of many datasets.\r\n\r\nYup this will eventually benefit us in getting correct total count in test results. Other than that, your [placeholder workaround](https://github.com/am11/libsass/blob/64267fce0df339c876cb876d17cf58e9c65d429e/test/spec_theories.c#L144) for theories is doing a brilliant job. :heavy_check_mark: "}}