{"action":"created","issue":{"url":"https://api.github.com/repos/bartvm/fuel/issues/56","labels_url":"https://api.github.com/repos/bartvm/fuel/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/bartvm/fuel/issues/56/comments","events_url":"https://api.github.com/repos/bartvm/fuel/issues/56/events","html_url":"https://github.com/bartvm/fuel/pull/56","id":64679976,"number":56,"title":"WIP: Data processing server","user":{"login":"bartvm","id":2299595,"avatar_url":"https://avatars.githubusercontent.com/u/2299595?v=3","gravatar_id":"","url":"https://api.github.com/users/bartvm","html_url":"https://github.com/bartvm","followers_url":"https://api.github.com/users/bartvm/followers","following_url":"https://api.github.com/users/bartvm/following{/other_user}","gists_url":"https://api.github.com/users/bartvm/gists{/gist_id}","starred_url":"https://api.github.com/users/bartvm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartvm/subscriptions","organizations_url":"https://api.github.com/users/bartvm/orgs","repos_url":"https://api.github.com/users/bartvm/repos","events_url":"https://api.github.com/users/bartvm/events{/privacy}","received_events_url":"https://api.github.com/users/bartvm/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-27T03:01:11Z","updated_at":"2015-03-27T15:50:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bartvm/fuel/pulls/56","html_url":"https://github.com/bartvm/fuel/pull/56","diff_url":"https://github.com/bartvm/fuel/pull/56.diff","patch_url":"https://github.com/bartvm/fuel/pull/56.patch"},"body":"As requested in #7. We already have some basic multi-processing, but we've already run into issues with HDF5 files (#47) so I figured it was worth revisiting the idea of having a separate server that pushes data around through sockets (optimized for NumPy data).\r\n\r\nAs I'm kind of figuring out ZMQ and messaging frameworks as I go along, this is a very rough draft, and I might be doing it completely wrong. I'm basically abusing the extended [request-and-reply pattern](http://zguide.zeromq.org/page:all#Shared-Queue-DEALER-and-ROUTER-sockets), making the broker act as a buffer. The idea is that the user creates his data stream in one file and calls `start_server(data_stream)` at that end end. In the script with the main loop the data stream is then given as `ServerDataStream()`.\r\n\r\nLong to-do list:\r\n\r\n- [ ] If the client dies, it can't reconnect to the server. If we want to allow this, we need something like the [lazy pirate pattern](http://zguide.zeromq.org/page:all#Client-Side-Reliability-Lazy-Pirate-Pattern) I think.\r\n- [x] Support for multiple sources.\r\n- [x] Server needs to send stop iteration message to client\r\n- [x] Documentation\r\n- [x] Client interface (`ServerDataStream`)\r\n- [x] Benchmarking\r\n- [x] Python 3 compatibility (byte strings, `buffer` interface)\r\n- [ ] Checkpointing support\r\n\r\nAny feedback welcome. Like I said, making this up as I go along."},"comment":{"url":"https://api.github.com/repos/bartvm/fuel/issues/comments/86982471","html_url":"https://github.com/bartvm/fuel/pull/56#issuecomment-86982471","issue_url":"https://api.github.com/repos/bartvm/fuel/issues/56","id":86982471,"user":{"login":"bartvm","id":2299595,"avatar_url":"https://avatars.githubusercontent.com/u/2299595?v=3","gravatar_id":"","url":"https://api.github.com/users/bartvm","html_url":"https://github.com/bartvm","followers_url":"https://api.github.com/users/bartvm/followers","following_url":"https://api.github.com/users/bartvm/following{/other_user}","gists_url":"https://api.github.com/users/bartvm/gists{/gist_id}","starred_url":"https://api.github.com/users/bartvm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartvm/subscriptions","organizations_url":"https://api.github.com/users/bartvm/orgs","repos_url":"https://api.github.com/users/bartvm/repos","events_url":"https://api.github.com/users/bartvm/events{/privacy}","received_events_url":"https://api.github.com/users/bartvm/received_events","type":"User","site_admin":false},"created_at":"2015-03-27T15:50:29Z","updated_at":"2015-03-27T15:50:29Z","body":"Whoever wants to have a look, this should be good for initial review (@dwf, @vdumoulin, @rizar)"}}