{"action":"created","issue":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/2401","labels_url":"https://api.github.com/repos/mne-tools/mne-python/issues/2401/labels{/name}","comments_url":"https://api.github.com/repos/mne-tools/mne-python/issues/2401/comments","events_url":"https://api.github.com/repos/mne-tools/mne-python/issues/2401/events","html_url":"https://github.com/mne-tools/mne-python/pull/2401","id":100618129,"number":2401,"title":"EHN: GAT parallel across chunks for predict & score","user":{"login":"kingjr","id":4881164,"avatar_url":"https://avatars.githubusercontent.com/u/4881164?v=3","gravatar_id":"","url":"https://api.github.com/users/kingjr","html_url":"https://github.com/kingjr","followers_url":"https://api.github.com/users/kingjr/followers","following_url":"https://api.github.com/users/kingjr/following{/other_user}","gists_url":"https://api.github.com/users/kingjr/gists{/gist_id}","starred_url":"https://api.github.com/users/kingjr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kingjr/subscriptions","organizations_url":"https://api.github.com/users/kingjr/orgs","repos_url":"https://api.github.com/users/kingjr/repos","events_url":"https://api.github.com/users/kingjr/events{/privacy}","received_events_url":"https://api.github.com/users/kingjr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-08-12T19:35:47Z","updated_at":"2015-09-07T13:18:23Z","closed_at":"2015-09-06T20:28:01Z","pull_request":{"url":"https://api.github.com/repos/mne-tools/mne-python/pulls/2401","html_url":"https://github.com/mne-tools/mne-python/pull/2401","diff_url":"https://github.com/mne-tools/mne-python/pull/2401.diff","patch_url":"https://github.com/mne-tools/mne-python/pull/2401.patch"},"body":"@dengemann @jona-sassenhagen @cmoutard\r\n\r\nThe previous implementation was parallelized across time samples, which was highly suboptimal when dealing with simple classifiers (e.g. based on a low number of features, like with eeg) and a lot of time points, because of constant jumps across CPU threads.\r\n\r\nWith the current implementation, the `predict` and `score` parallelization is made with chunked data. This is analogous to the method used for `fit`, but the chunked data is across time (not estimators), which is slightly more difficult to follow, but allows dispatching `X` across threads, as opposed to copy it in each thread.\r\n\r\nThe cpu and memory gain is pretty high for large datasets. (@agramfort No, I haven't done the profiling, but I see it by eye ;))\r\n\r\nI apologize in advance: the syntax remains quite hard to follow, most of the time because I made the stupid decision early on to accept irregular training and testing times (which can be useful, but not that often). There may be room for improvement and clarification (notably by using a single function to chunk and parallelize the data), but this will be for a later PR.\r\n\r\n(Also: I've had problems with the git history, some old commits were omitted; I'm not sure why, I tried rebasing a few time, keeping silent features, checked the history,  etc; in vain. I think it's because i) I started this branch a while ago and/or ii) I continued from https://github.com/mne-tools/mne-python/pull/2398)\r\n\r\nLet me know what you think."},"comment":{"url":"https://api.github.com/repos/mne-tools/mne-python/issues/comments/138295734","html_url":"https://github.com/mne-tools/mne-python/pull/2401#issuecomment-138295734","issue_url":"https://api.github.com/repos/mne-tools/mne-python/issues/2401","id":138295734,"user":{"login":"kingjr","id":4881164,"avatar_url":"https://avatars.githubusercontent.com/u/4881164?v=3","gravatar_id":"","url":"https://api.github.com/users/kingjr","html_url":"https://github.com/kingjr","followers_url":"https://api.github.com/users/kingjr/followers","following_url":"https://api.github.com/users/kingjr/following{/other_user}","gists_url":"https://api.github.com/users/kingjr/gists{/gist_id}","starred_url":"https://api.github.com/users/kingjr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kingjr/subscriptions","organizations_url":"https://api.github.com/users/kingjr/orgs","repos_url":"https://api.github.com/users/kingjr/repos","events_url":"https://api.github.com/users/kingjr/events{/privacy}","received_events_url":"https://api.github.com/users/kingjr/received_events","type":"User","site_admin":false},"created_at":"2015-09-07T13:18:23Z","updated_at":"2015-09-07T13:18:23Z","body":"no worries & thank you ;)"}}