{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5237","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5237/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5237/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5237/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5237","id":105716020,"number":5237,"title":"In roc_curve: drop some suboptimal thresholds","user":{"login":"gclenaghan","id":8988581,"avatar_url":"https://avatars.githubusercontent.com/u/8988581?v=3","gravatar_id":"","url":"https://api.github.com/users/gclenaghan","html_url":"https://github.com/gclenaghan","followers_url":"https://api.github.com/users/gclenaghan/followers","following_url":"https://api.github.com/users/gclenaghan/following{/other_user}","gists_url":"https://api.github.com/users/gclenaghan/gists{/gist_id}","starred_url":"https://api.github.com/users/gclenaghan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gclenaghan/subscriptions","organizations_url":"https://api.github.com/users/gclenaghan/orgs","repos_url":"https://api.github.com/users/gclenaghan/repos","events_url":"https://api.github.com/users/gclenaghan/events{/privacy}","received_events_url":"https://api.github.com/users/gclenaghan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-09-10T01:40:09Z","updated_at":"2015-09-25T16:27:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5237","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5237","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5237.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5237.patch"},"body":"Currently, the roc_curve function returns a value for every distinct possible threshold. For many classification routines this is close to the number of data points, and the results of this can get very unwieldy. For instance, with a large test set, the plotted ROC curve is unsuitable for a vector graphics format since the image file will be large and slow to load. Moreover, many of the thresholds are suboptimal: one could choose a different threshold for a higher TPR at the same FPR or a lower FPR at the same TPR.\r\n\r\nI've added an option keep_all_thresholds to the roc_curve function which drops thresholds which are invisible on a plotted ROC curve when set to false (by default it is set to true to ensure compatibility). Basically, in the sorted list of scores, if two of the same class appear consecutively, it will drop threshold corresponding to the first.\r\n\r\nHere are some tests:\r\n\r\n\tIn [21]:\r\n\tt1, t2\r\n\timport sklearn.datasets\r\n\timport sklearn.metrics as met\r\n\timport time\r\n\tprobas_pred, y_true = sklearn.datasets.make_classification(n_samples=1000000,\r\n\t\t\t\t\t\t\t\t\t\tn_features=1, n_informative=1,\r\n\t\t\t\t\t\t\t\t\t\tn_redundant=0, n_clusters_per_class=1,\r\n\t\t\t\t\t\t\t\t\t\trandom_state=1031)\r\n\t​\r\n\tt0 = time.time()\r\n\tfpr, tpr, thr = met.roc_curve(y_true, probas_pred)\r\n\tt1 = time.time() - t0\r\n\t​\r\n\tt0 = time.time()\r\n\tfpr2, tpr2, thr2 = met.roc_curve(y_true, probas_pred, keep_all_thresholds=False)\r\n\tt2 = time.time() - t0\r\n\t​\r\n\tt1, t2\r\n\r\n\tOut[21]:\r\n\t(0.2661151885986328, 0.24276185035705566)\r\n\r\n\tIn [18]:\r\n\t\r\n\tfpr.shape, fpr2.shape\r\n\r\n\tOut[18]:\r\n\t((990748,), (9843,))\r\n\r\n\tIn [19]:\r\n\t\r\n\tmet.auc(fpr, tpr) - met.auc(fpr2, tpr2)\r\n\r\n\tOut[19]:\r\n\t-2.2459999415858078e-09\r\n\r\nWith this option roc_curve is faster and returns many fewer results (how many fewer depends on how good the classifier is, results may vary).\r\n\r\nI've also changed some tests to use the new option.\r\n\r\nI've been doing something similar to this with the results of roc_curve but it would be quicker and easier to have it directly in the function. Please let me know if there is anything I forgot to do!"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/143268991","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5237#issuecomment-143268991","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5237","id":143268991,"user":{"login":"gclenaghan","id":8988581,"avatar_url":"https://avatars.githubusercontent.com/u/8988581?v=3","gravatar_id":"","url":"https://api.github.com/users/gclenaghan","html_url":"https://github.com/gclenaghan","followers_url":"https://api.github.com/users/gclenaghan/followers","following_url":"https://api.github.com/users/gclenaghan/following{/other_user}","gists_url":"https://api.github.com/users/gclenaghan/gists{/gist_id}","starred_url":"https://api.github.com/users/gclenaghan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gclenaghan/subscriptions","organizations_url":"https://api.github.com/users/gclenaghan/orgs","repos_url":"https://api.github.com/users/gclenaghan/repos","events_url":"https://api.github.com/users/gclenaghan/events{/privacy}","received_events_url":"https://api.github.com/users/gclenaghan/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T16:27:08Z","updated_at":"2015-09-25T16:27:08Z","body":"Sure, I modified the comment to be more verbose."}}