{"action":"closed","issue":{"url":"https://api.github.com/repos/confluentinc/ducktape/issues/20","labels_url":"https://api.github.com/repos/confluentinc/ducktape/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/confluentinc/ducktape/issues/20/comments","events_url":"https://api.github.com/repos/confluentinc/ducktape/issues/20/events","html_url":"https://github.com/confluentinc/ducktape/issues/20","id":58474371,"number":20,"title":"Automatic validation of Camus performance output","user":{"login":"ewencp","id":47366,"avatar_url":"https://avatars.githubusercontent.com/u/47366?v=3","gravatar_id":"","url":"https://api.github.com/users/ewencp","html_url":"https://github.com/ewencp","followers_url":"https://api.github.com/users/ewencp/followers","following_url":"https://api.github.com/users/ewencp/following{/other_user}","gists_url":"https://api.github.com/users/ewencp/gists{/gist_id}","starred_url":"https://api.github.com/users/ewencp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ewencp/subscriptions","organizations_url":"https://api.github.com/users/ewencp/orgs","repos_url":"https://api.github.com/users/ewencp/repos","events_url":"https://api.github.com/users/ewencp/events{/privacy}","received_events_url":"https://api.github.com/users/ewencp/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-21T20:49:26Z","updated_at":"2015-04-16T04:25:53Z","closed_at":"2015-04-16T04:25:53Z","body":"These tests are currently called \"Performance\" tests, but they really just test the basic functionality. They should be validating the output of the Hadoop job to check for at least a few things:\r\n\r\n1. That we're even running where expected! We previously discovered one was running on a local runner because of a misconfiguration.\r\n2. That the expected records were handled, which we should be able to do by picking out the right counters from the job output.\r\n2.a. Even better would be if we could read the output file from HDFS and validate its contents.\r\n3. The test intentionally triggers an error by publishing some null data, which Camus has to ignore. We want to see the report of this specific exception but still see the job finish.\r\n\r\nThe last item makes this especially important -- we should probably move the output to debug level since the exception output is expected but misleading.\r\n\r\nWhile implementing this, it'd probably also be worthwhile to intentionally trigger some other type of error and make sure we're detecting unexpected exceptions from the job, i.e. that we can properly detect if the job did not finish as expected."}}