{"action":"created","issue":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/1848","labels_url":"https://api.github.com/repos/mne-tools/mne-python/issues/1848/labels{/name}","comments_url":"https://api.github.com/repos/mne-tools/mne-python/issues/1848/comments","events_url":"https://api.github.com/repos/mne-tools/mne-python/issues/1848/events","html_url":"https://github.com/mne-tools/mne-python/issues/1848","id":60821201,"number":1848,"title":"ENH: stats for ERF decoding and time generalization","user":{"login":"dengemann","id":1908618,"avatar_url":"https://avatars.githubusercontent.com/u/1908618?v=3","gravatar_id":"","url":"https://api.github.com/users/dengemann","html_url":"https://github.com/dengemann","followers_url":"https://api.github.com/users/dengemann/followers","following_url":"https://api.github.com/users/dengemann/following{/other_user}","gists_url":"https://api.github.com/users/dengemann/gists{/gist_id}","starred_url":"https://api.github.com/users/dengemann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dengemann/subscriptions","organizations_url":"https://api.github.com/users/dengemann/orgs","repos_url":"https://api.github.com/users/dengemann/repos","events_url":"https://api.github.com/users/dengemann/events{/privacy}","received_events_url":"https://api.github.com/users/dengemann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-03-12T13:49:47Z","updated_at":"2015-12-17T15:28:08Z","closed_at":null,"body":"With @kingjr we're currently thinking about the right way to handle cross-trial significance testing for the analyses demonstrated in our decoding examples. This is basically overlapping with [one of our GSOC topics](https://github.com/mne-tools/mne-python/wiki/GSOC-Ideas#improve-multivariate-stats-and-machine-learning-support), and the [discussion in the comments over here](http://blogs.discovermagazine.com/neuroskeptic/2015/01/18/machine-learning-exceeding-chance-by-chance/#.VQGUmYHF-6A). Now practically, permutation testing or bootstrapping seems pretty heavy, if it means that you resample every model you fit. This will render these kind of analyses intractable.  On the flip side there seems to be a problem in using any test that makes assumptions about the degrees of freedom, as for single trial probability output, for example, the samples are not independend because of cross-validation. It seems, instead of bootstrapping the entire decoding or incorporating permutation testing into cross-validation, as suggested by Martin Hebart (first comment [here](http://blogs.discovermagazine.com/neuroskeptic/2015/01/18/machine-learning-exceeding-chance-by-chance/#.VQGUmYHF-6A)), we could just compute a permutation test post-hoc on the predictions. This would be much more tractable.  \r\nI was wondering what you think might be the right way to go. Also for the GSOC project it would be nice to have some discussion about this.\r\n\r\ncc @agramfort @Eric89GXL @banilo"},"comment":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/comments/165483809","html_url":"https://github.com/mne-tools/mne-python/issues/1848#issuecomment-165483809","issue_url":"https://api.github.com/repos/mne-tools/mne-python/issues/1848","id":165483809,"user":{"login":"topepo","id":5731043,"avatar_url":"https://avatars.githubusercontent.com/u/5731043?v=3","gravatar_id":"","url":"https://api.github.com/users/topepo","html_url":"https://github.com/topepo","followers_url":"https://api.github.com/users/topepo/followers","following_url":"https://api.github.com/users/topepo/following{/other_user}","gists_url":"https://api.github.com/users/topepo/gists{/gist_id}","starred_url":"https://api.github.com/users/topepo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/topepo/subscriptions","organizations_url":"https://api.github.com/users/topepo/orgs","repos_url":"https://api.github.com/users/topepo/repos","events_url":"https://api.github.com/users/topepo/events{/privacy}","received_events_url":"https://api.github.com/users/topepo/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T15:28:07Z","updated_at":"2015-12-17T15:28:07Z","body":"Without knowing too much about the details of the problem, my thoughts are:\r\n\r\n * comparisons between resamples is probably preferable since it can drastically improve the power of the comparison (relative to a single static test set). Controlling for the resample-to-resample variability, which can be huge, is easy to do. \r\n * permutation tests, depending on how yo do them, tend to test the hypothesis that the signal is above the noise. If you are using them to evaluate a single model, knowing that the performance is better than noise can be a very low bar and measuring the uncertainty of the performance metric (via confidence or credible intervals) is probably a better idea. A permutation test might be more applicable if you jointly permute two models and see if the difference in performance between them is above the experimental noise. "}}