{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/dask/issues/842","labels_url":"https://api.github.com/repos/blaze/dask/issues/842/labels{/name}","comments_url":"https://api.github.com/repos/blaze/dask/issues/842/comments","events_url":"https://api.github.com/repos/blaze/dask/issues/842/events","html_url":"https://github.com/blaze/dask/pull/842","id":118294020,"number":842,"title":"Significant performance improvement in bag.core._chunk_read_file","user":{"login":"sisp","id":2206639,"avatar_url":"https://avatars.githubusercontent.com/u/2206639?v=3","gravatar_id":"","url":"https://api.github.com/users/sisp","html_url":"https://github.com/sisp","followers_url":"https://api.github.com/users/sisp/followers","following_url":"https://api.github.com/users/sisp/following{/other_user}","gists_url":"https://api.github.com/users/sisp/gists{/gist_id}","starred_url":"https://api.github.com/users/sisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sisp/subscriptions","organizations_url":"https://api.github.com/users/sisp/orgs","repos_url":"https://api.github.com/users/sisp/repos","events_url":"https://api.github.com/users/sisp/events{/privacy}","received_events_url":"https://api.github.com/users/sisp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":27,"created_at":"2015-11-22T22:28:31Z","updated_at":"2015-11-28T15:39:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/blaze/dask/pulls/842","html_url":"https://github.com/blaze/dask/pull/842","diff_url":"https://github.com/blaze/dask/pull/842.diff","patch_url":"https://github.com/blaze/dask/pull/842.patch"},"body":"This PR changes the way a single large text file is processed in parallel using ``from_filenames`` and the ``chunkbytes`` argument. Before, in ``utils.textblock`` ``utils.next_linesep`` determined the exact start and end bytes of each block by actually streaming over the corresponding region in the file and finally the exact region was loaded into memory. This is very inefficient because the region is read twice and loaded into memory while the remaining graph may only require streaming over the bag elements. I noticed the performance problem when processing both a large and even very moderately sized (a couple of MBs) gzipped text file. This PR makes the chunked processing significantly faster.\r\n\r\nI have only made changes to the ``bag`` module but saw that ``utils.textblock`` is also used in the ``dataframe`` module. Perhaps my ``bag.core._textblock`` function can be moved to ``utils`` and then ``dataframe`` may be adapted to benefit from the speed-up, too.\r\n\r\nI welcome any comments and suggestions."},"comment":{"url":"https://api.github.com/repos/blaze/dask/issues/comments/160312218","html_url":"https://github.com/blaze/dask/pull/842#issuecomment-160312218","issue_url":"https://api.github.com/repos/blaze/dask/issues/842","id":160312218,"user":{"login":"sisp","id":2206639,"avatar_url":"https://avatars.githubusercontent.com/u/2206639?v=3","gravatar_id":"","url":"https://api.github.com/users/sisp","html_url":"https://github.com/sisp","followers_url":"https://api.github.com/users/sisp/followers","following_url":"https://api.github.com/users/sisp/following{/other_user}","gists_url":"https://api.github.com/users/sisp/gists{/gist_id}","starred_url":"https://api.github.com/users/sisp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sisp/subscriptions","organizations_url":"https://api.github.com/users/sisp/orgs","repos_url":"https://api.github.com/users/sisp/repos","events_url":"https://api.github.com/users/sisp/events{/privacy}","received_events_url":"https://api.github.com/users/sisp/received_events","type":"User","site_admin":false},"created_at":"2015-11-28T15:39:22Z","updated_at":"2015-11-28T15:39:22Z","body":"Do you mean we would read the entire chunk into memory? Probably into something like ``StringIO``? If, so, that may increase memory usage (significantly, if the chunk size is large)."}}