{"action":"created","issue":{"url":"https://api.github.com/repos/Blosc/python-blosc/issues/65","labels_url":"https://api.github.com/repos/Blosc/python-blosc/issues/65/labels{/name}","comments_url":"https://api.github.com/repos/Blosc/python-blosc/issues/65/comments","events_url":"https://api.github.com/repos/Blosc/python-blosc/issues/65/events","html_url":"https://github.com/Blosc/python-blosc/pull/65","id":34657411,"number":65,"title":"buffer type support","user":{"login":"esc","id":41563,"avatar_url":"https://avatars.githubusercontent.com/u/41563?v=3","gravatar_id":"","url":"https://api.github.com/users/esc","html_url":"https://github.com/esc","followers_url":"https://api.github.com/users/esc/followers","following_url":"https://api.github.com/users/esc/following{/other_user}","gists_url":"https://api.github.com/users/esc/gists{/gist_id}","starred_url":"https://api.github.com/users/esc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/esc/subscriptions","organizations_url":"https://api.github.com/users/esc/orgs","repos_url":"https://api.github.com/users/esc/repos","events_url":"https://api.github.com/users/esc/events{/privacy}","received_events_url":"https://api.github.com/users/esc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2014-05-30T15:53:35Z","updated_at":"2015-05-26T09:19:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Blosc/python-blosc/pulls/65","html_url":"https://github.com/Blosc/python-blosc/pull/65","diff_url":"https://github.com/Blosc/python-blosc/pull/65.diff","patch_url":"https://github.com/Blosc/python-blosc/pull/65.patch"},"body":"Under certain circumstances it is advantageous to supply a ``buffer`` type to the compression and decompression routines. My use case is decompressing Numpy arrays from string in Bloscpack. The input data is a string and during decompression the individual chunks need to be taken from the string. In order to do this you could for example slice the string. However, AFAIU this necessitates a memory copy. The current implementation is such that I use a cStringIO, since much of the stuff in Bloscpack is implemented using file pointers. However,  from what I understand, reading from a cStringIO object also requires a memory copy, it would seem. The alternative is to use the ``buffer`` builtin to emulate a file:\r\n\r\nhttps://github.com/esc/bloscpack/commit/dd77456754aa68b34639ec26539ae4c34c57a5b0\r\n\r\nAnd then use that as a source for the compressed data.\r\n\r\nInitial benchmarks look promising:\r\n\r\n```\r\nIn [2]: %cpaste\r\nPasting code; enter '--' alone on the line to stop or use Ctrl-D.\r\n:import numpy as np\r\n:import bloscpack as bp\r\n:a = np.linspace(0, 100, 2e8)\r\n:shuffle = True\r\n:clevel = 9\r\n:cname = 'lz4'\r\n:bargs = bp.args.BloscArgs(clevel=clevel, shuffle=shuffle, cname=cname)\r\n:bpargs = bp.BloscpackArgs(checksum='None', offsets=False, max_app_chunks=0)\r\n:bpc = bp.pack_ndarray_str(a, blosc_args=bargs, bloscpack_args=bpargs,\r\n:        chunk_size='0.5G')\r\n:--\r\n\r\nIn [4]: %timeit a3 = bp.unpack_ndarray_str(bpc)\r\n1 loops, best of 3: 390 ms per loop\r\n\r\nIn [5]: %timeit a3 = bp.unpack_ndarray_str(bpc)\r\n1 loops, best of 3: 389 ms per loop\r\n\r\nIn [6]: %timeit a3 = bp.fast_unpack_ndarray_str(bpc)\r\n1 loops, best of 3: 336 ms per loop\r\n\r\nIn [7]: %timeit a3 = bp.fast_unpack_ndarray_str(bpc)\r\n1 loops, best of 3: 337 ms per loop\r\n```\r\n\r\nWhere the ``fast_unpack_ndarray_str`` is using the ``buffer`` under the hood. In fact this solution is of the same order as the plain ``compress_ptr`` method:\r\n\r\n```\r\nIn [9]: import blosc\r\n\r\nIn [10]: c = blosc.compress_ptr(a.__array_interface__['data'][0], a.size, a.dtype.itemsize, clevel=clevel, shuffle=shuffle, cname=cname)\r\n\r\nIn [11]: %timeit a2 = np.empty_like(a) ; bytes_written = blosc.decompress_ptr(c, a2.__array_interface__['data'][0])\r\n1 loops, best of 3: 334 ms per loop\r\n\r\nIn [12]: %timeit a2 = np.empty_like(a) ; bytes_written = blosc.decompress_ptr(c, a2.__array_interface__['data'][0])\r\n1 loops, best of 3: 334 ms per loop\r\n```\r\n\r\nSupport was easy to implemet since we use ``s#`` in the ``PyArg_ParseTuple`` which can accept a ``buffer`` as input and will cast that into a c-string:\r\n\r\n```\r\ns# (string, Unicode or any read buffer compatible object) [const char *, int (or Py_ssize_t, see below)]\r\n\r\n    This variant on s stores into two C variables, the first one a pointer to a character string, the\r\nsecond one its length. In this case the Python string may contain embedded null bytes.\r\n Unicode objects pass back a pointer to the default encoded string version of the object if such\r\n a conversion is possible. All other read-buffer compatible objects pass back a reference to the\r\n raw internal data representation.\r\n```\r\n\r\nSo they only thing left to do, was to allow this from the ``toplevel.py``\r\n\r\nA remaining issue is that ``buffer`` was removed in Python 3 and the documentation seems to suggest using a ``memoryview`` but I can't get it to work. The error is something about requiring a 'pinnned' read-only buffer. I hope to look into a Py3 solution soon but wanted to float this idea already anyway."},"comment":{"url":"https://api.github.com/repos/Blosc/python-blosc/issues/comments/105460026","html_url":"https://github.com/Blosc/python-blosc/pull/65#issuecomment-105460026","issue_url":"https://api.github.com/repos/Blosc/python-blosc/issues/65","id":105460026,"user":{"login":"esc","id":41563,"avatar_url":"https://avatars.githubusercontent.com/u/41563?v=3","gravatar_id":"","url":"https://api.github.com/users/esc","html_url":"https://github.com/esc","followers_url":"https://api.github.com/users/esc/followers","following_url":"https://api.github.com/users/esc/following{/other_user}","gists_url":"https://api.github.com/users/esc/gists{/gist_id}","starred_url":"https://api.github.com/users/esc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/esc/subscriptions","organizations_url":"https://api.github.com/users/esc/orgs","repos_url":"https://api.github.com/users/esc/repos","events_url":"https://api.github.com/users/esc/events{/privacy}","received_events_url":"https://api.github.com/users/esc/received_events","type":"User","site_admin":false},"created_at":"2015-05-26T09:19:07Z","updated_at":"2015-05-26T09:19:07Z","body":"Probably superseded by #80 "}}