{"action":"created","issue":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/431","labels_url":"https://api.github.com/repos/mlr-org/mlr/issues/431/labels{/name}","comments_url":"https://api.github.com/repos/mlr-org/mlr/issues/431/comments","events_url":"https://api.github.com/repos/mlr-org/mlr/issues/431/events","html_url":"https://github.com/mlr-org/mlr/issues/431","id":99720453,"number":431,"title":"Bug in performance calculation for a ResamplePrediction?","user":{"login":"schiffner","id":8244303,"avatar_url":"https://avatars.githubusercontent.com/u/8244303?v=3","gravatar_id":"","url":"https://api.github.com/users/schiffner","html_url":"https://github.com/schiffner","followers_url":"https://api.github.com/users/schiffner/followers","following_url":"https://api.github.com/users/schiffner/following{/other_user}","gists_url":"https://api.github.com/users/schiffner/gists{/gist_id}","starred_url":"https://api.github.com/users/schiffner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schiffner/subscriptions","organizations_url":"https://api.github.com/users/schiffner/orgs","repos_url":"https://api.github.com/users/schiffner/repos","events_url":"https://api.github.com/users/schiffner/events{/privacy}","received_events_url":"https://api.github.com/users/schiffner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/mlr-org/mlr/milestones/1","html_url":"https://github.com/mlr-org/mlr/milestones/v2.5","labels_url":"https://api.github.com/repos/mlr-org/mlr/milestones/1/labels","id":1318474,"number":1,"title":"v2.5","description":"","creator":{"login":"mllg","id":1260920,"avatar_url":"https://avatars.githubusercontent.com/u/1260920?v=3","gravatar_id":"","url":"https://api.github.com/users/mllg","html_url":"https://github.com/mllg","followers_url":"https://api.github.com/users/mllg/followers","following_url":"https://api.github.com/users/mllg/following{/other_user}","gists_url":"https://api.github.com/users/mllg/gists{/gist_id}","starred_url":"https://api.github.com/users/mllg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mllg/subscriptions","organizations_url":"https://api.github.com/users/mllg/orgs","repos_url":"https://api.github.com/users/mllg/repos","events_url":"https://api.github.com/users/mllg/events{/privacy}","received_events_url":"https://api.github.com/users/mllg/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":6,"state":"open","created_at":"2015-09-22T14:27:45Z","updated_at":"2015-10-23T17:54:19Z","due_on":"2015-10-11T07:00:00Z","closed_at":null},"comments":4,"created_at":"2015-08-07T19:58:35Z","updated_at":"2015-10-23T19:00:14Z","closed_at":null,"body":"Some functions like `tuneThreshold` or `generateThreshVsPerfData` call `performance` on a `ResamplePrediction`.\r\nFor example `tuneThreshold` and `generateThreshVsPerfData` both contain something like\r\n\r\n```\r\nperformance(setThreshold(pred, ...))\r\n```\r\n\r\nwhere `pred` and thus `setThreshold(pred, ...)` are `Prediction`s (and therefore can also be `ResamplePrediction`s) to calculate performance values for different threshold settings.\r\n\r\n`performance` seems to treat the `ResamplePrediction` as a `Prediction`, the aggregation functions in the `Measure`s are not taken into account, which can lead to totally wrong performance values, for example:\r\n\r\n```\r\nset.seed(123)\r\nm = mmce\r\nm = setAggregation(m, b632)\r\n\r\nrdesc = makeResampleDesc(\"Bootstrap\", iters = 5, predict = \"both\")\r\nr1 = resample(makeLearner(\"classif.lda\", predict.type = \"prob\"), sonar.task, rdesc, list(mmce, m))\r\nr1$aggr\r\n# mmce.test.mean      mmce.b632 \r\n#     0.2695533      0.1898192 \r\n\r\nd1 = generateThreshVsPerfData(r1, list(mmce, m))\r\nplotThreshVsPerf(d1)\r\n\r\nperformance(r1$pred, list(mmce, m))\r\n#     mmce     mmce \r\n# 0.109375 0.109375\r\nperformance(setThreshold(r1$pred, 0.5), list(mmce, m))\r\n    # mmce     mmce \r\n# 0.109375 0.109375\r\n\r\nset.seed(123)\r\nrdesc = makeResampleDesc(\"Bootstrap\", iters = 5)\r\nr2 = resample(makeLearner(\"classif.lda\", predict.type = \"prob\"), sonar.task, rdesc)\r\nr2$aggr\r\n# mmce.test.mean \r\n#      0.2695533 \r\n\r\nd2 = generateThreshVsPerfData(r2)\r\nplotThreshVsPerf(d2)\r\n\r\nperformance(r2$pred)\r\n#      mmce \r\n# 0.2690217 \r\nperformance(setThreshold(r2$pred, 0.5))\r\n#      mmce \r\n# 0.2690217\r\n```\r\n\r\nI hope I'm not doing something wrong.\r\nIn case of `tuneThreshold` I'm not sure if it was originally intended for `ResamplePrediction`s."},"comment":{"url":"https://api.github.com/repos/mlr-org/mlr/issues/comments/150663993","html_url":"https://github.com/mlr-org/mlr/issues/431#issuecomment-150663993","issue_url":"https://api.github.com/repos/mlr-org/mlr/issues/431","id":150663993,"user":{"login":"schiffner","id":8244303,"avatar_url":"https://avatars.githubusercontent.com/u/8244303?v=3","gravatar_id":"","url":"https://api.github.com/users/schiffner","html_url":"https://github.com/schiffner","followers_url":"https://api.github.com/users/schiffner/followers","following_url":"https://api.github.com/users/schiffner/following{/other_user}","gists_url":"https://api.github.com/users/schiffner/gists{/gist_id}","starred_url":"https://api.github.com/users/schiffner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schiffner/subscriptions","organizations_url":"https://api.github.com/users/schiffner/orgs","repos_url":"https://api.github.com/users/schiffner/repos","events_url":"https://api.github.com/users/schiffner/events{/privacy}","received_events_url":"https://api.github.com/users/schiffner/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T19:00:13Z","updated_at":"2015-10-23T19:00:13Z","body":"I don't recall that we found a concrete solution. What I do remember is: We agreed that the `ResamplePrediction` should be composed of several `Predictions` as opposed to the current state, where it is derived/inherits from a `Prediction`.\r\nI am very much willing to help/discuss more."}}