{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4583","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4583/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4583/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4583/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4583","id":68130841,"number":4583,"title":"[MRG + 1] ShuffleLabelsOut cross-validation iterator","user":{"login":"bmcfee","id":1190540,"avatar_url":"https://avatars.githubusercontent.com/u/1190540?v=3","gravatar_id":"","url":"https://api.github.com/users/bmcfee","html_url":"https://github.com/bmcfee","followers_url":"https://api.github.com/users/bmcfee/followers","following_url":"https://api.github.com/users/bmcfee/following{/other_user}","gists_url":"https://api.github.com/users/bmcfee/gists{/gist_id}","starred_url":"https://api.github.com/users/bmcfee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmcfee/subscriptions","organizations_url":"https://api.github.com/users/bmcfee/orgs","repos_url":"https://api.github.com/users/bmcfee/repos","events_url":"https://api.github.com/users/bmcfee/events{/privacy}","received_events_url":"https://api.github.com/users/bmcfee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-04-13T15:47:41Z","updated_at":"2015-07-03T14:10:56Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/4583","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4583","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/4583.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/4583.patch"},"body":"This PR implements a stochastic variant of leave-p-labels-out CV splitting.  Rather than iterating over all `n_classes \\choose p` subsets of labels for the test points, a user-specified number (fraction) of the labels are randomly selected.  The number of iterations is also user-specified, as in `ShuffleSplit`.\r\n\r\n---\r\n\r\nThis is similar to #4444 , except that no balance is maintained at the level of individual samples.\r\n\r\nI'm not sure if there's another PR implementing this exactly this functionality already, but I couldn't find any.  However, if I missed it, and someone's already implemented this, feel free to ignore!\r\n\r\n---\r\n\r\nA realistic example use case for this is in music information retrieval, where it is common to have a collection of recordings indexed by artist, each artist has a small number of data points, and all of an artist's points should reside on one side of the train/test split.  In this setting, the number of labels is large, so an exhaustive `LeavePLabelsOut` search is undesirable."},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/118360897","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4583#issuecomment-118360897","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4583","id":118360897,"user":{"login":"ogrisel","id":89061,"avatar_url":"https://avatars.githubusercontent.com/u/89061?v=3","gravatar_id":"","url":"https://api.github.com/users/ogrisel","html_url":"https://github.com/ogrisel","followers_url":"https://api.github.com/users/ogrisel/followers","following_url":"https://api.github.com/users/ogrisel/following{/other_user}","gists_url":"https://api.github.com/users/ogrisel/gists{/gist_id}","starred_url":"https://api.github.com/users/ogrisel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ogrisel/subscriptions","organizations_url":"https://api.github.com/users/ogrisel/orgs","repos_url":"https://api.github.com/users/ogrisel/repos","events_url":"https://api.github.com/users/ogrisel/events{/privacy}","received_events_url":"https://api.github.com/users/ogrisel/received_events","type":"User","site_admin":false},"created_at":"2015-07-03T14:10:56Z","updated_at":"2015-07-03T14:10:56Z","body":"Also I wonder if we should not rename this class `ShuffleDisjointLabelSplit` to be more in-line with the new `DisjointLabelKFold` class while being consistent with the basic `ShuffleSplit` alternative."}}