{"action":"created","issue":{"url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/167","labels_url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/167/labels{/name}","comments_url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/167/comments","events_url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/167/events","html_url":"https://github.com/test-kitchen/test-kitchen/issues/167","id":16454466,"number":167,"title":"Suites that should fail","user":{"login":"coderanger","id":128243,"avatar_url":"https://avatars.githubusercontent.com/u/128243?v=3","gravatar_id":"","url":"https://api.github.com/users/coderanger","html_url":"https://github.com/coderanger","followers_url":"https://api.github.com/users/coderanger/followers","following_url":"https://api.github.com/users/coderanger/following{/other_user}","gists_url":"https://api.github.com/users/coderanger/gists{/gist_id}","starred_url":"https://api.github.com/users/coderanger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coderanger/subscriptions","organizations_url":"https://api.github.com/users/coderanger/orgs","repos_url":"https://api.github.com/users/coderanger/repos","events_url":"https://api.github.com/users/coderanger/events{/privacy}","received_events_url":"https://api.github.com/users/coderanger/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/test-kitchen/test-kitchen/labels/New%20Feature","name":"New Feature","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/test-kitchen/test-kitchen/milestones/3","html_url":"https://github.com/test-kitchen/test-kitchen/milestones/2.0","labels_url":"https://api.github.com/repos/test-kitchen/test-kitchen/milestones/3/labels","id":438776,"number":3,"title":"2.0","description":"","creator":{"login":"sethvargo","id":408570,"avatar_url":"https://avatars.githubusercontent.com/u/408570?v=3","gravatar_id":"","url":"https://api.github.com/users/sethvargo","html_url":"https://github.com/sethvargo","followers_url":"https://api.github.com/users/sethvargo/followers","following_url":"https://api.github.com/users/sethvargo/following{/other_user}","gists_url":"https://api.github.com/users/sethvargo/gists{/gist_id}","starred_url":"https://api.github.com/users/sethvargo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sethvargo/subscriptions","organizations_url":"https://api.github.com/users/sethvargo/orgs","repos_url":"https://api.github.com/users/sethvargo/repos","events_url":"https://api.github.com/users/sethvargo/events{/privacy}","received_events_url":"https://api.github.com/users/sethvargo/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":14,"state":"open","created_at":"2013-09-27T16:28:55Z","updated_at":"2015-03-18T21:49:18Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2013-07-08T04:13:04Z","updated_at":"2015-07-28T15:31:17Z","closed_at":null,"body":"There is not currently a great way to check that a given recipe fails, sometimes important to test that error detection is working."},"comment":{"url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/comments/125655074","html_url":"https://github.com/test-kitchen/test-kitchen/issues/167#issuecomment-125655074","issue_url":"https://api.github.com/repos/test-kitchen/test-kitchen/issues/167","id":125655074,"user":{"login":"steversmith","id":8061122,"avatar_url":"https://avatars.githubusercontent.com/u/8061122?v=3","gravatar_id":"","url":"https://api.github.com/users/steversmith","html_url":"https://github.com/steversmith","followers_url":"https://api.github.com/users/steversmith/followers","following_url":"https://api.github.com/users/steversmith/following{/other_user}","gists_url":"https://api.github.com/users/steversmith/gists{/gist_id}","starred_url":"https://api.github.com/users/steversmith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steversmith/subscriptions","organizations_url":"https://api.github.com/users/steversmith/orgs","repos_url":"https://api.github.com/users/steversmith/repos","events_url":"https://api.github.com/users/steversmith/events{/privacy}","received_events_url":"https://api.github.com/users/steversmith/received_events","type":"User","site_admin":false},"created_at":"2015-07-28T15:31:17Z","updated_at":"2015-07-28T15:31:17Z","body":"I have a similar need to perform verification tests after a convergence fails. As a workaround, I made a couple of changes to the test-kitchen.\r\n\r\nIn the kitchen.yml, I added a `ignore_convergence_failure` setting to the suite definition:\r\n```\r\nsuites:\r\n- name: should_fail\r\n  ignore_convergence_failure: true\r\n```\r\n\r\nIn the lib/kitchen/suite.rb, I added a `ignore_convergence_failure?` method\r\n```\r\ndef initialize(options = {})\r\n  ...\r\n  option = options.fetch(:ignore_convergence_failure, false)\r\n  @ignore_convergence_failure = option==true || option==\"true\"\r\nend\r\n\r\n# @return [Boolean] True if test suite should ignore convergence failures\r\ndef ignore_convergence_failure?\r\n  @ignore_convergence_failure\r\nend\r\n```\r\n\r\nIn the lib/kitchen/instance.rb, I added a rescue block to the `converge_action` method:\r\n```\r\ndef converge_action\r\n  ...\r\n      begin\r\n        provisioner.call(state)\r\n      rescue ActionFailed => e\r\n        if suite.ignore_convergence_failure?\r\n          # produce a backtrace in the instance's file logger\r\n          # to aid in debugging\r\n          log_failure(:converge, e)\r\n\r\n          # produce a high level message on the console, the kitchen log,\r\n          # and the instance's file logger to provide visibility that\r\n          # the error is being ignored\r\n          banner \"ignore_convergence_failure on #{to_str}! \" \\\r\n            \"Please see .kitchen/logs/#{name}.log for more details\"\r\n        else\r\n          raise e\r\n        end\r\n      end\r\n  ...\r\nend\r\n```\r\n\r\nIn my serverspec, I added a test of the `chef-stacktrace.out` to confirm the expected error:\r\n```\r\n# validate the stack trace\r\ndescribe file(\"/tmp/kitchen/cache/chef-stacktrace.out\") do\r\n  its(:content) { should match /my expected error/ }\r\nend\r\n```\r\n\r\n***What do you think?  Should I create a pull request with my changes?***\r\n\r\nNote that I have manually tested this, but I cannot get the automated tests to run at all, even without any edits to the code, so I have not added any unit tests..."}}