{"action":"created","issue":{"url":"https://api.github.com/repos/xray/xray/issues/585","labels_url":"https://api.github.com/repos/xray/xray/issues/585/labels{/name}","comments_url":"https://api.github.com/repos/xray/xray/issues/585/comments","events_url":"https://api.github.com/repos/xray/xray/issues/585/events","html_url":"https://github.com/xray/xray/issues/585","id":107424151,"number":585,"title":"Parallel map/apply powered by dask.array","user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/xray/xray/labels/dask","name":"dask","color":"bfdadc"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/xray/xray/milestones/5","html_url":"https://github.com/xray/xray/milestones/before%201.0","labels_url":"https://api.github.com/repos/xray/xray/milestones/5/labels","id":741199,"number":5,"title":"before 1.0","description":"","creator":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":4,"state":"open","created_at":"2014-08-03T19:39:34Z","updated_at":"2015-09-20T23:28:11Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-09-20T23:27:55Z","updated_at":"2015-09-28T09:43:17Z","closed_at":null,"body":"Dask is awesome, but it isn't always easy to use it for parallel operations. In many cases, especially when wrapping routines from external libraries, it is most straightforward to express operations in terms of a function that expects and returns xray objects loaded into memory.\r\n\r\nDask array has a `map_blocks` function/method, but it's applicability is limited because dask.array doesn't have axis names for unambiguously identifying dimensions. `da.atop` can handle many of these cases, but it's not the easiest to use. Fortunately, we have sufficient metadata in xray that we could probably parallelize many `atop` operations automatically by inferring result dimensions and dtypes from applying the function once. See here for more discussion on the dask side: https://github.com/blaze/dask/issues/702\r\n\r\nSo I would like to add some convenience methods for automatic parallelization with dask of a function defined on xray objects loaded into memory. In addition to a `map_blocks` method/function, it would be useful to add some sort of `parallel_apply` method to groupby objects that works very similarly, by lazily applying a function that takes and returns xray objects loaded into memory."},"comment":{"url":"https://api.github.com/repos/xray/xray/issues/comments/143692567","html_url":"https://github.com/xray/xray/issues/585#issuecomment-143692567","issue_url":"https://api.github.com/repos/xray/xray/issues/585","id":143692567,"user":{"login":"rabernat","id":1197350,"avatar_url":"https://avatars.githubusercontent.com/u/1197350?v=3","gravatar_id":"","url":"https://api.github.com/users/rabernat","html_url":"https://github.com/rabernat","followers_url":"https://api.github.com/users/rabernat/followers","following_url":"https://api.github.com/users/rabernat/following{/other_user}","gists_url":"https://api.github.com/users/rabernat/gists{/gist_id}","starred_url":"https://api.github.com/users/rabernat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rabernat/subscriptions","organizations_url":"https://api.github.com/users/rabernat/orgs","repos_url":"https://api.github.com/users/rabernat/repos","events_url":"https://api.github.com/users/rabernat/events{/privacy}","received_events_url":"https://api.github.com/users/rabernat/received_events","type":"User","site_admin":false},"created_at":"2015-09-28T09:43:17Z","updated_at":"2015-09-28T09:43:17Z","body":":+1: Very useful idea!"}}