{"action":"created","issue":{"url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/2678","labels_url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/2678/labels{/name}","comments_url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/2678/comments","events_url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/2678/events","html_url":"https://github.com/shogun-toolbox/shogun/issues/2678","id":55276627,"number":2678,"title":"evaluate multiple metrics during cross validation","user":{"login":"HAShihab","id":1406926,"avatar_url":"https://avatars.githubusercontent.com/u/1406926?v=3","gravatar_id":"","url":"https://api.github.com/users/HAShihab","html_url":"https://github.com/HAShihab","followers_url":"https://api.github.com/users/HAShihab/followers","following_url":"https://api.github.com/users/HAShihab/following{/other_user}","gists_url":"https://api.github.com/users/HAShihab/gists{/gist_id}","starred_url":"https://api.github.com/users/HAShihab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HAShihab/subscriptions","organizations_url":"https://api.github.com/users/HAShihab/orgs","repos_url":"https://api.github.com/users/HAShihab/repos","events_url":"https://api.github.com/users/HAShihab/events{/privacy}","received_events_url":"https://api.github.com/users/HAShihab/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-23T12:30:54Z","updated_at":"2015-01-23T12:45:20Z","closed_at":null,"body":"The examples show how to perform cross-validation and evaluate a single metric (e.g. accuracy).  If we want to evaluate multiple metrics, we have to call the cross-validation multiple times (once for each metric).  I think it would be useful for the cross-valiation to give us the option to evaluate multiple metrics at once (e.g. accuracy, precision, AUC etc.) without having to call the cross-valiadtion multiple times."},"comment":{"url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/comments/71188456","html_url":"https://github.com/shogun-toolbox/shogun/issues/2678#issuecomment-71188456","issue_url":"https://api.github.com/repos/shogun-toolbox/shogun/issues/2678","id":71188456,"user":{"login":"HAShihab","id":1406926,"avatar_url":"https://avatars.githubusercontent.com/u/1406926?v=3","gravatar_id":"","url":"https://api.github.com/users/HAShihab","html_url":"https://github.com/HAShihab","followers_url":"https://api.github.com/users/HAShihab/followers","following_url":"https://api.github.com/users/HAShihab/following{/other_user}","gists_url":"https://api.github.com/users/HAShihab/gists{/gist_id}","starred_url":"https://api.github.com/users/HAShihab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HAShihab/subscriptions","organizations_url":"https://api.github.com/users/HAShihab/orgs","repos_url":"https://api.github.com/users/HAShihab/repos","events_url":"https://api.github.com/users/HAShihab/events{/privacy}","received_events_url":"https://api.github.com/users/HAShihab/received_events","type":"User","site_admin":false},"created_at":"2015-01-23T12:45:20Z","updated_at":"2015-01-23T12:45:20Z","body":"@abinashmeher999 sure: taking the Python example from here: http://www.shogun-toolbox.org/static/notebook/current/xval_modelselection.html\r\n\r\n```\r\nmetric=AccuracyMeasure()\r\ncross=CrossValidation(classifier, features, labels, stratified_split, metric)\r\n\r\n# perform the cross-validation, note that this call involved a lot of computation\r\nresult=cross.evaluate()\r\n\r\n# the result needs to be casted to CrossValidationResult\r\nresult=CrossValidationResult.obtain_from_generic(result)\r\n\r\n# this class contains a field \"mean\" which contain the mean performance metric\r\nprint \"Testing\", metric.get_name(), result.mean\r\n```\r\n\r\nIf I want to test a suite of metrics, it seems as though I would have to call CrossValidation and evaluate for each metric.  I think having an all-encompassing metric type that returns as many metrics as possible would be useful, e.g.\r\n\r\n```\r\nmetric=AllMeasure()\r\ncross=CrossValidation(classifier, features, labels, stratified_split, metric)\r\nresult=cross.evaluate()\r\nresult=CrossValidationResult.obtain_from_generic(result)\r\n\r\nprint “Accuracy:”, result['Accuracy'].mean\r\nprint “Precision:”, result['Precision'].mean\r\n```"}}