{"action":"created","issue":{"url":"https://api.github.com/repos/kwikteam/phy/issues/129","labels_url":"https://api.github.com/repos/kwikteam/phy/issues/129/labels{/name}","comments_url":"https://api.github.com/repos/kwikteam/phy/issues/129/comments","events_url":"https://api.github.com/repos/kwikteam/phy/issues/129/events","html_url":"https://github.com/kwikteam/phy/issues/129","id":65712819,"number":129,"title":"Benchmark and optimize on large datasets","user":{"login":"rossant","id":1942359,"avatar_url":"https://avatars.githubusercontent.com/u/1942359?v=3","gravatar_id":"","url":"https://api.github.com/users/rossant","html_url":"https://github.com/rossant","followers_url":"https://api.github.com/users/rossant/followers","following_url":"https://api.github.com/users/rossant/following{/other_user}","gists_url":"https://api.github.com/users/rossant/gists{/gist_id}","starred_url":"https://api.github.com/users/rossant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rossant/subscriptions","organizations_url":"https://api.github.com/users/rossant/orgs","repos_url":"https://api.github.com/users/rossant/repos","events_url":"https://api.github.com/users/rossant/events{/privacy}","received_events_url":"https://api.github.com/users/rossant/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kwikteam/phy/labels/performance","name":"performance","color":"d4c5f9"},{"url":"https://api.github.com/repos/kwikteam/phy/labels/ready","name":"ready","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"rossant","id":1942359,"avatar_url":"https://avatars.githubusercontent.com/u/1942359?v=3","gravatar_id":"","url":"https://api.github.com/users/rossant","html_url":"https://github.com/rossant","followers_url":"https://api.github.com/users/rossant/followers","following_url":"https://api.github.com/users/rossant/following{/other_user}","gists_url":"https://api.github.com/users/rossant/gists{/gist_id}","starred_url":"https://api.github.com/users/rossant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rossant/subscriptions","organizations_url":"https://api.github.com/users/rossant/orgs","repos_url":"https://api.github.com/users/rossant/repos","events_url":"https://api.github.com/users/rossant/events{/privacy}","received_events_url":"https://api.github.com/users/rossant/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/kwikteam/phy/milestones/4","html_url":"https://github.com/kwikteam/phy/milestones/Alpha%20version","labels_url":"https://api.github.com/repos/kwikteam/phy/milestones/4/labels","id":1048956,"number":4,"title":"Alpha version","description":"","creator":{"login":"rossant","id":1942359,"avatar_url":"https://avatars.githubusercontent.com/u/1942359?v=3","gravatar_id":"","url":"https://api.github.com/users/rossant","html_url":"https://github.com/rossant","followers_url":"https://api.github.com/users/rossant/followers","following_url":"https://api.github.com/users/rossant/following{/other_user}","gists_url":"https://api.github.com/users/rossant/gists{/gist_id}","starred_url":"https://api.github.com/users/rossant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rossant/subscriptions","organizations_url":"https://api.github.com/users/rossant/orgs","repos_url":"https://api.github.com/users/rossant/repos","events_url":"https://api.github.com/users/rossant/events{/privacy}","received_events_url":"https://api.github.com/users/rossant/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":9,"state":"open","created_at":"2015-04-01T15:34:01Z","updated_at":"2015-04-07T11:54:14Z","due_on":"2015-04-15T22:00:00Z","closed_at":null},"comments":2,"created_at":"2015-04-01T15:33:33Z","updated_at":"2015-04-07T21:00:45Z","closed_at":null,"body":"* [x] Choose and download locally one of Nick's huge datasets.\r\n* [x] Optimize the cluster store (generation time and loading time)\r\n    * Initialize all arrays in the cluster store\r\n    * Loop over all spikes\r\n    * Copy over the features and masks in the cluster store\r\n* [ ] Benchmark"},"comment":{"url":"https://api.github.com/repos/kwikteam/phy/issues/comments/90731210","html_url":"https://github.com/kwikteam/phy/issues/129#issuecomment-90731210","issue_url":"https://api.github.com/repos/kwikteam/phy/issues/129","id":90731210,"user":{"login":"rossant","id":1942359,"avatar_url":"https://avatars.githubusercontent.com/u/1942359?v=3","gravatar_id":"","url":"https://api.github.com/users/rossant","html_url":"https://github.com/rossant","followers_url":"https://api.github.com/users/rossant/followers","following_url":"https://api.github.com/users/rossant/following{/other_user}","gists_url":"https://api.github.com/users/rossant/gists{/gist_id}","starred_url":"https://api.github.com/users/rossant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rossant/subscriptions","organizations_url":"https://api.github.com/users/rossant/orgs","repos_url":"https://api.github.com/users/rossant/repos","events_url":"https://api.github.com/users/rossant/events{/privacy}","received_events_url":"https://api.github.com/users/rossant/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T21:00:45Z","updated_at":"2015-04-07T21:00:45Z","body":"Some more notes:\r\n\r\n* More benchmarks make me thing that HDF5 is too damn slow for the cluster store. I think we need something else.\r\n* One possibility would be to design our own file format, but that's risky.\r\n* [bcolz](http://bcolz.blosc.org) is an interesting option. It can be easily installed with [conda](https://binstar.org/blaze/bcolz).\r\n* Here is a [gist](https://gist.github.com/rossant/02dda655690ab3a2a92b) with a quick and dirty benchmark script.\r\n\r\n@cgestes is that something you'd want to look into once you're done with the GUI? That would give me more time to make progress on the rest of the Python API.\r\n\r\nLet me explain what to benchmark (see the gist above):\r\n\r\n* Start from a big `(n_spikes, n_channels)` array in HDF5 (the KWX file, we can't change the format for now)\r\n* You have a cluster assigned to each spike\r\n* The end-result must be a set of files, one per cluster, containing the sub-array corresponding to the spikes in that cluster\r\n* Eventually there will be several arrays in each file (typically 2, features and masks), but for benchmark it should be enough to use only one\r\n* In the gist above, I process the data chunk by chunk, and I save the results incrementally on the disk (appending is fast on flat binary files)\r\n* Finally, I put back the binary files into HDF5 (this is slow)\r\n\r\nIt would be better if we could get rid of the last step. So we want a unique file format such that:\r\n\r\n* appending data is fast\r\n* I/O is fast\r\n* don't care about file size\r\n\r\nFor profiling you can use line_profiler. Don't forget to clear the OS cache too."}}