{"action":"created","issue":{"url":"https://api.github.com/repos/sass/sass-spec/issues/136","labels_url":"https://api.github.com/repos/sass/sass-spec/issues/136/labels{/name}","comments_url":"https://api.github.com/repos/sass/sass-spec/issues/136/comments","events_url":"https://api.github.com/repos/sass/sass-spec/issues/136/events","html_url":"https://github.com/sass/sass-spec/issues/136","id":49020782,"number":136,"title":"Cannot test error reporting","user":{"login":"xzyfer","id":579928,"avatar_url":"https://avatars.githubusercontent.com/u/579928?v=3","gravatar_id":"","url":"https://api.github.com/users/xzyfer","html_url":"https://github.com/xzyfer","followers_url":"https://api.github.com/users/xzyfer/followers","following_url":"https://api.github.com/users/xzyfer/following{/other_user}","gists_url":"https://api.github.com/users/xzyfer/gists{/gist_id}","starred_url":"https://api.github.com/users/xzyfer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xzyfer/subscriptions","organizations_url":"https://api.github.com/users/xzyfer/orgs","repos_url":"https://api.github.com/users/xzyfer/repos","events_url":"https://api.github.com/users/xzyfer/events{/privacy}","received_events_url":"https://api.github.com/users/xzyfer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2014-11-16T23:44:04Z","updated_at":"2015-09-18T23:12:40Z","closed_at":null,"body":"As it stands it's not possible to assert an error condition. i.e `error on duplicate keys in maps `  https://github.com/sass/libsass/issues/628\r\n\r\nThere are cases where a spec like the following should be possible https://github.com/sass/sass-spec/pull/133/files.\r\n\r\nMaybe we can redirect error output in the assertion some how. Thoughts? \r\n\r\n/cc @lunelson @hcatlin "},"comment":{"url":"https://api.github.com/repos/sass/sass-spec/issues/comments/141591153","html_url":"https://github.com/sass/sass-spec/issues/136#issuecomment-141591153","issue_url":"https://api.github.com/repos/sass/sass-spec/issues/136","id":141591153,"user":{"login":"saper","id":248374,"avatar_url":"https://avatars.githubusercontent.com/u/248374?v=3","gravatar_id":"","url":"https://api.github.com/users/saper","html_url":"https://github.com/saper","followers_url":"https://api.github.com/users/saper/followers","following_url":"https://api.github.com/users/saper/following{/other_user}","gists_url":"https://api.github.com/users/saper/gists{/gist_id}","starred_url":"https://api.github.com/users/saper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saper/subscriptions","organizations_url":"https://api.github.com/users/saper/orgs","repos_url":"https://api.github.com/users/saper/repos","events_url":"https://api.github.com/users/saper/events{/privacy}","received_events_url":"https://api.github.com/users/saper/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T23:12:40Z","updated_at":"2015-09-18T23:12:40Z","body":"Before we rewrite a whole thing, I just had a look at the ruby code with my basic understanding of the language and found the following:\r\n\r\n* a [test case](https://github.com/sass/sass-spec/blob/6d38353b04e22a044eaab6f6e9bebaeeb5c5f7e4/lib/sass_spec/test_case.rb#L45) reports standard output, cleaned standard output, standard error and compile error and the status code resulting from the compilation\r\n* a [runner](https://github.com/sass/sass-spec/blob/6d38353b04e22a044eaab6f6e9bebaeeb5c5f7e4/lib/sass_spec/runner.rb#L44-L45) just loads few files that describe the expected values:\r\n  * \"expected standard output\"\r\n  *\"expected clean standard output\" (optional)\r\n  * applicable output style\r\n   and feeds them to the test case\r\n* a [comparator](https://github.com/sass/sass-spec/blob/6d38353b04e22a044eaab6f6e9bebaeeb5c5f7e4/lib/sass_spec/runner.rb#L44-L45) takes standard output, standard error, status, \"clean\" output and applies some very simple rules to determine if the test failed or succeeded. It currently does not use standard error, but it's there.\r\n\r\nIt seems to be trivial to add testing against std error, std output or both. You could even have something like \"I want this in standard output and not this in standard error\". The logic can be pretty arbitrary.\r\n\r\nThe only real issue is to specify **how** to express those rules. Something like:\r\n\r\n1. If there is a `*testcase*.*outputstyle*.skip` file, please skip the test\r\n1. If there is a `*testcase*.*outputstyle*.status` file, please match it against the status code returned. Stop checking further. (is it needed at all? probably not)\r\n1. If there is a `*testcase*.*outputstyle*.error` file, please compare it against the standard error, fail if different. A zero-length `*testcase*.*outputstyle*.error` indicates the error output should be empty and no additional checks are to be made. Stop checking further if did a check here..\r\n1. If there is a `*testcase*.*outputstyle*.clean` file, please perform the next test against a \"cleaned\" output.\r\n1. If there is a `*testcase*.*outputstyle*.css` file, please compare it against the standard output.\r\n\r\nplus checking of some flags. \r\n\r\nAs I understand the ruby code right now, options 1, 4 and 5 are currently implemented.\r\n\r\nThere is also a possibility to generate \"expected\" output files with a command line option; we should decide the logic in which order those files are written, for example:\r\n1. If the script generates some error output, write `.error` file. Stop.\r\n2. If the script generates no error output, write stdout to the`.css` file.\r\n\r\nAs I understand the ruby code right now, option 2 is currently implemented.\r\n\r\nSo once we agree how to specify the expected error output, it should be trivial to implement."}}