{"action":"reopened","issue":{"url":"https://api.github.com/repos/nipy/nibabel/issues/209","labels_url":"https://api.github.com/repos/nipy/nibabel/issues/209/labels{/name}","comments_url":"https://api.github.com/repos/nipy/nibabel/issues/209/comments","events_url":"https://api.github.com/repos/nipy/nibabel/issues/209/events","html_url":"https://github.com/nipy/nibabel/issues/209","id":22661028,"number":209,"title":"ENH: Improve read pattern in gzip","user":{"login":"GaelVaroquaux","id":208217,"avatar_url":"https://avatars.githubusercontent.com/u/208217?v=3","gravatar_id":"","url":"https://api.github.com/users/GaelVaroquaux","html_url":"https://github.com/GaelVaroquaux","followers_url":"https://api.github.com/users/GaelVaroquaux/followers","following_url":"https://api.github.com/users/GaelVaroquaux/following{/other_user}","gists_url":"https://api.github.com/users/GaelVaroquaux/gists{/gist_id}","starred_url":"https://api.github.com/users/GaelVaroquaux/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GaelVaroquaux/subscriptions","organizations_url":"https://api.github.com/users/GaelVaroquaux/orgs","repos_url":"https://api.github.com/users/GaelVaroquaux/repos","events_url":"https://api.github.com/users/GaelVaroquaux/events{/privacy}","received_events_url":"https://api.github.com/users/GaelVaroquaux/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2013-11-14T12:59:57Z","updated_at":"2015-11-16T00:45:24Z","closed_at":null,"body":"**Problems**\r\n\r\n*Memory usage*\r\n\r\nWhen opening big gzip files, as for instance those shipped by the human connectome project, nibabel takes too much memory (see for instance #208 ).\r\n\r\nThe breakage happens in gzip.open and can be explained as follow (see the traceback in #208 to understand better):\r\n\r\n- gzip does not know how much memory it needs to allocate, so it uses 1) for decompression a growing chunk size, with powers of two and capped at gzip.GzipFile.max_read_chunk, 2) for storing the output a buffer that it grows progressively by appending the chunk read.\r\n- for memory problems, that buffer is the worst problem as its memory footprint gets doubled when the append happens (it is not an append: a new buffer is created, and the old one removed).\r\n- if the final data has 4Gb + epsilon, the gzip opener will have at a given instant 2 4Gb buffers, and, in @rphlypo's case blow the memory\r\n\r\n*Read is slow*\r\n\r\nIn addition to the memory usage, this pattern leads to very inefficient code, as the temporary buffer is grown again and again, which leads to costly memory copies. These are especially costly since the size of the memory copy isn't small compared to the total size of the memory, and thus the system has to move other memory pages belonging to other programs to allocate this memory. This is typically the behavior that renders systems unresponsive when the memory usage gets close 100%. To witness how bad the I/O speed is with data of a size on the same order of magnitude than the total memory, an easy experiment is to compare the time a 'nibabel.get_data()' takes to the time required to:\r\n - gzip.GzipFile('file.nii.gz').read()    (slow)\r\n - gunzip file.nii.gz -c > /dev/null      (almost twice as fast)\r\n - open in fslview                              (almost twice as fast)\r\n\r\n**Proposed improvement**\r\n\r\nThe only way that I can think of to improve the memory usage would be to preallocate a buffer big enough. We roughly know the size. However, the gzip module does not allow us to do this, and it would require fairly deep monkey patching that I think is not reasonable. I give up on this aspect of the problem.\r\n\r\nFor the speed, the simple option is to increase the 'max_read_chunk' of the GzipFile object. With a file that is 400Mb big, here is an experiment that shows the benefit:\r\n<pre>\r\nimport gzip\r\ngzip_file = gzip.open('0010042_rest_tshift_RPI_voreg_mni.nii.gz')\r\ngzip_file.max_read_chunk = max_read_chunk = 100 * 1024 * 1024 # 100Mb\r\ndata = gzip_file.read()\r\n</pre>\r\nThe extra line (third line) makes a factor of 2 of speed in the above example.\r\n\r\nTo see the impact on nibabel, a little bit of monkey-patching enables experimentation:\r\n<pre>\r\nimport gzip\r\ngzip.GzipFile.max_read_chunk = max_read_chunk = 200 * 1024 * 1024 # 200Mb\r\n</pre>\r\nRoughly factors of two speedups\r\n\r\n**Action point**\r\n\r\nI am proposing to submit a patch to nibabel that uses the following strategy for an opener:\r\n<pre>\r\ngzip_file = gzip.open('0010042_rest_tshift_RPI_voreg_mni.nii.gz')\r\ngzip_file.max_read_chunk = max_read_chunk = 100 * 1024 * 1024 # 100Mb\r\ndata = gzip_file.read()\r\n</pre>\r\n\r\nIt should be an easy, local, modification. @matthew-brett : would you be in favor of such a patch?"}}