{"action":"created","issue":{"url":"https://api.github.com/repos/Blosc/bcolz/issues/265","labels_url":"https://api.github.com/repos/Blosc/bcolz/issues/265/labels{/name}","comments_url":"https://api.github.com/repos/Blosc/bcolz/issues/265/comments","events_url":"https://api.github.com/repos/Blosc/bcolz/issues/265/events","html_url":"https://github.com/Blosc/bcolz/issues/265","id":111105691,"number":265,"title":"Question on bcolz to numpy array conversion performance","user":{"login":"monkeybutter","id":2526498,"avatar_url":"https://avatars.githubusercontent.com/u/2526498?v=3","gravatar_id":"","url":"https://api.github.com/users/monkeybutter","html_url":"https://github.com/monkeybutter","followers_url":"https://api.github.com/users/monkeybutter/followers","following_url":"https://api.github.com/users/monkeybutter/following{/other_user}","gists_url":"https://api.github.com/users/monkeybutter/gists{/gist_id}","starred_url":"https://api.github.com/users/monkeybutter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monkeybutter/subscriptions","organizations_url":"https://api.github.com/users/monkeybutter/orgs","repos_url":"https://api.github.com/users/monkeybutter/repos","events_url":"https://api.github.com/users/monkeybutter/events{/privacy}","received_events_url":"https://api.github.com/users/monkeybutter/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-10-13T05:13:43Z","updated_at":"2015-10-14T22:12:59Z","closed_at":null,"body":"I've being doing some benchmarks comparing different serialization formats and how fast we can read from disk. The comparison includes numpy's fromfile, hdf5 and bcolz:\r\n\r\nhttps://gist.github.com/monkeybutter/ed72220be48333ce0dde\r\n\r\nbcolz is amazingly fast at reading the file and creating the carray object, one order of magnitude faster than the second best, h5py. However, I would like to understand why the conversion of a bcolz into a numpy array is so expensive (gist cell no. 8)\r\n\r\nI know bcolz is intended to be used to perform operations internally using the functionality it exposes, but to be fair with the benchmark I wanted to do the comparison using the same numpy array as the output for all the cases.\r\n\r\nIs there anything that can be done to improve the speed of this conversion?"},"comment":{"url":"https://api.github.com/repos/Blosc/bcolz/issues/comments/148218299","html_url":"https://github.com/Blosc/bcolz/issues/265#issuecomment-148218299","issue_url":"https://api.github.com/repos/Blosc/bcolz/issues/265","id":148218299,"user":{"login":"monkeybutter","id":2526498,"avatar_url":"https://avatars.githubusercontent.com/u/2526498?v=3","gravatar_id":"","url":"https://api.github.com/users/monkeybutter","html_url":"https://github.com/monkeybutter","followers_url":"https://api.github.com/users/monkeybutter/followers","following_url":"https://api.github.com/users/monkeybutter/following{/other_user}","gists_url":"https://api.github.com/users/monkeybutter/gists{/gist_id}","starred_url":"https://api.github.com/users/monkeybutter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/monkeybutter/subscriptions","organizations_url":"https://api.github.com/users/monkeybutter/orgs","repos_url":"https://api.github.com/users/monkeybutter/repos","events_url":"https://api.github.com/users/monkeybutter/events{/privacy}","received_events_url":"https://api.github.com/users/monkeybutter/received_events","type":"User","site_admin":false},"created_at":"2015-10-14T22:12:59Z","updated_at":"2015-10-14T22:12:59Z","body":"Thank you both @FrancescElies and @FrancescAlted for all the information you have shared. I find really interesting knowing more about the internals of blosc and understanding what is really happening under the hood.\r\n\r\nReading your comments I've realised the results will be highly dependent on the data, machine architecture and all the internal tuning parameters of bcolz. Finding the sweet spot would require traversing these parameters in every dimension [compressor, clevel, chunk size, concurrency, ...] for any particular data set and machine.\r\n\r\nI understand I'm mixing pears and apples in my benchmark trying to compare contiguous non compressed arrays with blocked compressed arrays. Knowing they are different kinds of fruit, I had the idea that reading from disk, blocked compressed arrays could beat, in some cases, non compressed contiguous arrays . If the data happens to be highly compressible [and shuffleble :-) ] and the chunk size matches the cache size of the CPU, is it possible to beat plain disk contiguous arrays?"}}