{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/842","labels_url":"https://api.github.com/repos/blaze/dask/issues/842/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/842/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/842/events","html_url":"https://github.com/blaze/dask/pull/842","id":118294020,"number":842,"title":"Significant performance improvement in bag.core._chunk_read_file","user":{"login":"sisp","id":2206639,"avatar_url":"https://avatars.githubusercontent.com/u/2206639?v=3","gravatar_id":"","url":"https://api.github.com/users/sisp","html_url":"https://github.com/sisp","followers_url":"https://api.github.com/users/sisp/followers","following_url":"https://api.github.com/users/sisp/following{/other_user}","gists_url":"https://api.github.com/users/sisp/gists{/gist_id}","starred_url":"https://api.github.com/users/sisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sisp/subscriptions","organizations_url":"https://api.github.com/users/sisp/orgs","repos_url":"https://api.github.com/users/sisp/repos","events_url":"https://api.github.com/users/sisp/events{/privacy}","received_events_url":"https://api.github.com/users/sisp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-11-22T22:28:31Z","updated_at":"2015-11-23T16:58:37Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/blaze/dask/pulls/842","html_url":"https://github.com/blaze/dask/pull/842","diff_url":"https://github.com/blaze/dask/pull/842.diff","patch_url":"https://github.com/blaze/dask/pull/842.patch"},"body":"This PR changes the way a single large text file is processed in parallel using ``from_filenames`` and the ``chunkbytes`` argument. Before, in ``utils.textblock`` ``utils.next_linesep`` determined the exact start and end bytes of each block by actually streaming over the corresponding region in the file and finally the exact region was loaded into memory. This is very inefficient because the region is read twice and loaded into memory while the remaining graph may only require streaming over the bag elements. I noticed the performance problem when processing both a large and even very moderately sized (a couple of MBs) gzipped text file. This PR makes the chunked processing significantly faster.\r\n\r\nI have only made changes to the ``bag`` module but saw that ``utils.textblock`` is also used in the ``dataframe`` module. Perhaps my ``bag.core._textblock`` function can be moved to ``utils`` and then ``dataframe`` may be adapted to benefit from the speed-up, too.\r\n\r\nI welcome any comments and suggestions."},"comment":{"url":"https://api.github.com/repos/blaze/dask/issues/comments/158995595","html_url":"https://github.com/blaze/dask/pull/842#issuecomment-158995595","issue_url":"https://api.github.com/repos/blaze/dask/issues/842","id":158995595,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-11-23T16:58:37Z","updated_at":"2015-11-23T16:58:37Z","body":"I also don't know of a good solution for the gzip issue.  The best I can think of is better warnings and better documentation.  \r\n\r\n*  Perhaps we should say something more explicit about this issue in docstrings like `from_filenames`\r\n*  Perhaps we should warn when using large-ish gzip files (like larger than 500 MB)\r\n\r\nBut as you suggest, there is no way to really properly defend against this fail case."}}