{"action":"created","issue":{"url":"https://api.github.com/repos/ContinuumIO/dask/issues/287","labels_url":"https://api.github.com/repos/ContinuumIO/dask/issues/287/labels{/name}","comments_url":"https://api.github.com/repos/ContinuumIO/dask/issues/287/comments","events_url":"https://api.github.com/repos/ContinuumIO/dask/issues/287/events","html_url":"https://github.com/ContinuumIO/dask/issues/287","id":86653433,"number":287,"title":"Creating several partitions on a single file","user":{"login":"chdoig","id":4940549,"avatar_url":"https://avatars.githubusercontent.com/u/4940549?v=3","gravatar_id":"","url":"https://api.github.com/users/chdoig","html_url":"https://github.com/chdoig","followers_url":"https://api.github.com/users/chdoig/followers","following_url":"https://api.github.com/users/chdoig/following{/other_user}","gists_url":"https://api.github.com/users/chdoig/gists{/gist_id}","starred_url":"https://api.github.com/users/chdoig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chdoig/subscriptions","organizations_url":"https://api.github.com/users/chdoig/orgs","repos_url":"https://api.github.com/users/chdoig/repos","events_url":"https://api.github.com/users/chdoig/events{/privacy}","received_events_url":"https://api.github.com/users/chdoig/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-06-09T16:16:55Z","updated_at":"2015-06-15T04:16:32Z","closed_at":null,"body":"**tl;dr:** Is it/Would it be possible to handle parallelism for processing a single large file with dask?\r\n\r\nUse case example: Processing wikipedia articles dump.\r\n\r\nI'm trying to reproduce a tokenization task from gensim with dask:  http://radimrehurek.com/gensim/wiki.html#preparing-the-corpus\r\n\r\n*Convert the articles to plain text (process Wiki markup) and store the result as sparse TF-IDF vectors. In Python, this is easy to do on-the-fly and we don’t even need to uncompress the whole archive to disk. There is a script included in gensim that does just that, run:*\r\n\r\n    $ python -m gensim.scripts.make_wiki\r\n\r\nThat script uses the class WikiCorpus, which uses multiprocessing the following way:\r\n\r\nhttps://github.com/piskvorky/gensim/blob/f1a904d61cdea8e44681342678a5eefd178d7b12/gensim/corpora/wikicorpus.py#L291-L308\r\n\r\nThe file is currently ~11GB compressed.\r\n\r\nThis problem could also help this library to extract wiki dumps:\r\nhttps://github.com/attardi/wikiextractor\r\n\r\nwhere they were also asking for help on a multi-threading/multi-processing that same task: https://github.com/attardi/wikiextractor/issues/4\r\n\r\nI'd like to tackle this problem with dask but, I don't know how/where I should start. Is it possible to create several bags after parsing a single file? Similarly to what gensim in that script does?\r\n\r\ncc: @mrocklin "},"comment":{"url":"https://api.github.com/repos/ContinuumIO/dask/issues/comments/111916655","html_url":"https://github.com/ContinuumIO/dask/issues/287#issuecomment-111916655","issue_url":"https://api.github.com/repos/ContinuumIO/dask/issues/287","id":111916655,"user":{"login":"eriknw","id":2058401,"avatar_url":"https://avatars.githubusercontent.com/u/2058401?v=3","gravatar_id":"","url":"https://api.github.com/users/eriknw","html_url":"https://github.com/eriknw","followers_url":"https://api.github.com/users/eriknw/followers","following_url":"https://api.github.com/users/eriknw/following{/other_user}","gists_url":"https://api.github.com/users/eriknw/gists{/gist_id}","starred_url":"https://api.github.com/users/eriknw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eriknw/subscriptions","organizations_url":"https://api.github.com/users/eriknw/orgs","repos_url":"https://api.github.com/users/eriknw/repos","events_url":"https://api.github.com/users/eriknw/events{/privacy}","received_events_url":"https://api.github.com/users/eriknw/received_events","type":"User","site_admin":false},"created_at":"2015-06-15T04:16:32Z","updated_at":"2015-06-15T04:16:32Z","body":"I spent some time this weekend digging into this.\r\n\r\nThe approach I have in mind for dask.bag relies heavily on a couple of components I envision for an alternative implementation of dask.distributed: notably, a hub on a distributed node that receives tasks \"on-the-fly\" from a master coordinator such that the entire DAG need not be known by the hub (and, in this case, the coordinator) in advanced.  However, developing a second version of dask.distributed is probably not the best use of my time right now, and the patterns needed for `dask.bag.from_iterable` are more general than what easily fits into dask.bag.\r\n\r\nSo, I decided to direct my effort into a parallel pipe in `toolz.sandbox.ppipe` (still being developed off-line for a little bit longer--I dove a little too deeply into the rabbit hole).  Splitters and reducers/combiners are callables passed to `ppipe` that have `_toolz_split_` and `_toolz_combine_` attributes defined respectively (and decorators can make defining these easier).  I hope to allow for additional functionality found in other stream processing and Map-Reduce-*esque* workflows, but the proof-of-concept will try to keep the style of `toolz`, and, hence, will look a lot like `dask.bag`.\r\n\r\nIt's been fun digging into `dask.bag`.  Feel free to send tasks for it my way.\r\n\r\nAnyway, we can probably close the original issue unless you want to keep this open to continue the discussion.  Supporting multiple partitions from a file is possible *if* the file is indexable/partitionable.  Line-delimited text is a good example of this.  Right now, an end-user probably isn't expected to make a custom partitioner."}}