{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4822","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4822/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4822/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4822/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4822","id":85483093,"number":4822,"title":"[WIP] ROCCH calibration method","user":{"login":"albahnsen","id":3617224,"avatar_url":"https://avatars.githubusercontent.com/u/3617224?v=3","gravatar_id":"","url":"https://api.github.com/users/albahnsen","html_url":"https://github.com/albahnsen","followers_url":"https://api.github.com/users/albahnsen/followers","following_url":"https://api.github.com/users/albahnsen/following{/other_user}","gists_url":"https://api.github.com/users/albahnsen/gists{/gist_id}","starred_url":"https://api.github.com/users/albahnsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/albahnsen/subscriptions","organizations_url":"https://api.github.com/users/albahnsen/orgs","repos_url":"https://api.github.com/users/albahnsen/repos","events_url":"https://api.github.com/users/albahnsen/events{/privacy}","received_events_url":"https://api.github.com/users/albahnsen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-06-05T09:34:41Z","updated_at":"2015-06-05T18:01:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/4822","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4822","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/4822.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/4822.patch"},"body":"Implementation of the ROCCH calibration method, as described in http://www.jmlr.org/papers/volume13/hernandez-orallo12a/hernandez-orallo12a.pdf and http://link.springer.com/article/10.1007/s10994-007-5011-0\r\n\r\nIt is implemented as an other method within the CalibratedClassifierCV class. Moreover, the documentation is expanded to include the results.\r\n\r\nThe method performs similarly to Isotonic calibration when measured by Brier loss, F1 Score or AUC. But  with an speedup of up to 1.3X\r\n\r\nHere an example\r\n```\r\nIn[2]: %paste\r\n\r\nimport pandas as pd\r\nfrom sklearn import datasets\r\nfrom sklearn.naive_bayes import GaussianNB\r\nfrom sklearn.svm import LinearSVC\r\nfrom sklearn.calibration import CalibratedClassifierCV\r\nfrom sklearn.cross_validation import train_test_split\r\nfrom sklearn.metrics import brier_score_loss, f1_score, roc_auc_score\r\n\r\nX, y = datasets.make_classification(n_samples=100000, n_features=20,\r\n                                    n_informative=2, n_redundant=10,\r\n                                    random_state=42)\r\n\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.99,\r\n                                                    random_state=42)\r\n\r\ndef test_times(method, est):\r\n    clf =  CalibratedClassifierCV(est, cv=5, method=method)\r\n    clf.fit(X_train, y_train)\r\n    y_pred = clf.predict(X_test)\r\n    y_prob = clf.predict_proba(X_test)[:,1]\r\n    brier = brier_score_loss(y_test, y_prob, pos_label=y.max())\r\n    f1 = f1_score(y_test, y_pred)\r\n    auc = roc_auc_score(y_test, y_prob)\r\n    return brier, f1, auc\r\n\r\nres = pd.DataFrame(columns=['brier_loss','f1_score','auc_score'])\r\n## -- End pasted text --\r\nBackend TkAgg is interactive backend. Turning interactive mode on.\r\nIn[3]: %timeit res.loc['GNB+isotonic'] = test_times(method='isotonic', est=GaussianNB())\r\n1 loops, best of 3: 823 ms per loop\r\nIn[4]: %timeit res.loc['GNB+sigmoid'] = test_times(method='sigmoid', est=GaussianNB())\r\n1 loops, best of 3: 776 ms per loop\r\nIn[5]: %timeit res.loc['GNB+rocch'] = test_times(method='rocch', est=GaussianNB())\r\n1 loops, best of 3: 752 ms per loop\r\nIn[6]: %timeit res.loc['LSVC+isotonic'] = test_times(method='isotonic', est=LinearSVC())\r\n1 loops, best of 3: 416 ms per loop\r\nIn[7]: %timeit res.loc['LSVC+sigmoid'] = test_times(method='sigmoid', est=LinearSVC())\r\n1 loops, best of 3: 367 ms per loop\r\nIn[8]: %timeit res.loc['LSVC+rocch'] = test_times(method='rocch', est=LinearSVC())\r\n1 loops, best of 3: 324 ms per loop\r\nIn[9]: res['time'] = [823, 776, 752, 416, 367, 324]\r\nIn[10]: print res\r\n```\r\nbrier_loss|   f1_score | auc_score | time \r\n------------ | ------------ | ------------ |------------\r\nGNB+isotonic    | 0.098487  |0.854058   |0.939305  | 823\r\nGNB+sigmoid     | 0.108937 | 0.866259  | 0.937025|   776\r\nGNB+rocch       | 0.098752 | 0.851963   |0.939027   |752\r\nLSVC+isotonic  |  0.099401|  0.864936 | 0.936852 |  416\r\nLSVC+sigmoid  |   0.098603 | 0.862041 |  0.937657|   367\r\nLSVC+rocch      | 0.099584 | 0.864046|   0.936733|   324"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/109381728","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4822#issuecomment-109381728","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4822","id":109381728,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2015-06-05T18:01:18Z","updated_at":"2015-06-05T18:01:18Z","body":"This is a relatively common method, right?\r\nWe discussed finding different threshold here: #4813.\r\nThat would target more the ``predict`` not the ``predict_proba``, though."}}