{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2904","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2904/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2904/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2904/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/2904","id":28350886,"number":2904,"title":"data-independent CV iterators","user":{"login":"mblondel","id":233706,"avatar_url":"https://avatars.githubusercontent.com/u/233706?v=3","gravatar_id":"","url":"https://api.github.com/users/mblondel","html_url":"https://github.com/mblondel","followers_url":"https://api.github.com/users/mblondel/followers","following_url":"https://api.github.com/users/mblondel/following{/other_user}","gists_url":"https://api.github.com/users/mblondel/gists{/gist_id}","starred_url":"https://api.github.com/users/mblondel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mblondel/subscriptions","organizations_url":"https://api.github.com/users/mblondel/orgs","repos_url":"https://api.github.com/users/mblondel/repos","events_url":"https://api.github.com/users/mblondel/events{/privacy}","received_events_url":"https://api.github.com/users/mblondel/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels/API","name":"API","color":"bfd4f2"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":31,"created_at":"2014-02-26T17:11:47Z","updated_at":"2015-10-24T21:00:53Z","closed_at":null,"body":"In many situations, you don't have a test set so you would like to use CV for *both* evaluation and hyper-parameter tuning. Therefore, you need to do nested cross-validation:\r\n```python\r\nfor train, test in cv1:\r\n   # Find the best hyper-parameters for this split\r\n    for train, val in cv2:\r\n        [...]\r\n   # Retrain using the best hyper-parameters\r\n   [...]\r\n# Return best scores for each split\r\n```\r\nThis is very difficult to implement in a generic way with our current API because CV iterators are tied to a particular data. For example, when doing `cv = KFold(n_samples)`, `cv` will only work with a dataset of the specified size. \r\n\r\nIdeally, we would need something closer to the estimator API: use constructor parameters for data-independent options (n_folds, shuffle, random_state, train / test proportion, etc) and a `run` method that takes `y` as argument (the reason to take `y` is for stratified schemes). This would look something like this:\r\n\r\n```python\r\n# deprecated usage\r\nfor train, test in KFold(n, n_folds):\r\n    print train, test\r\n\r\n# new usage\r\nfor train, test in KFold(n_folds).run(y):\r\n    print train, test\r\n```"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/150851175","html_url":"https://github.com/scikit-learn/scikit-learn/issues/2904#issuecomment-150851175","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/2904","id":150851175,"user":{"login":"jnothman","id":78827,"avatar_url":"https://avatars.githubusercontent.com/u/78827?v=3","gravatar_id":"","url":"https://api.github.com/users/jnothman","html_url":"https://github.com/jnothman","followers_url":"https://api.github.com/users/jnothman/followers","following_url":"https://api.github.com/users/jnothman/following{/other_user}","gists_url":"https://api.github.com/users/jnothman/gists{/gist_id}","starred_url":"https://api.github.com/users/jnothman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnothman/subscriptions","organizations_url":"https://api.github.com/users/jnothman/orgs","repos_url":"https://api.github.com/users/jnothman/repos","events_url":"https://api.github.com/users/jnothman/events{/privacy}","received_events_url":"https://api.github.com/users/jnothman/received_events","type":"User","site_admin":false},"created_at":"2015-10-24T21:00:53Z","updated_at":"2015-10-24T21:00:53Z","body":"Is this resolved?"}}