{"action":"created","issue":{"url":"https://api.github.com/repos/bartvm/blocks/issues/281","labels_url":"https://api.github.com/repos/bartvm/blocks/issues/281/labels{/name}","comments_url":"https://api.github.com/repos/bartvm/blocks/issues/281/comments","events_url":"https://api.github.com/repos/bartvm/blocks/issues/281/events","html_url":"https://github.com/bartvm/blocks/pull/281","id":57654447,"number":281,"title":"added SlidingWindowDataStream","user":{"login":"pbrakel","id":1336772,"avatar_url":"https://avatars.githubusercontent.com/u/1336772?v=3","gravatar_id":"","url":"https://api.github.com/users/pbrakel","html_url":"https://github.com/pbrakel","followers_url":"https://api.github.com/users/pbrakel/followers","following_url":"https://api.github.com/users/pbrakel/following{/other_user}","gists_url":"https://api.github.com/users/pbrakel/gists{/gist_id}","starred_url":"https://api.github.com/users/pbrakel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbrakel/subscriptions","organizations_url":"https://api.github.com/users/pbrakel/orgs","repos_url":"https://api.github.com/users/pbrakel/repos","events_url":"https://api.github.com/users/pbrakel/events{/privacy}","received_events_url":"https://api.github.com/users/pbrakel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-02-13T21:09:59Z","updated_at":"2015-02-18T11:50:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/bartvm/blocks/pulls/281","html_url":"https://github.com/bartvm/blocks/pull/281","diff_url":"https://github.com/bartvm/blocks/pull/281.diff","patch_url":"https://github.com/bartvm/blocks/pull/281.patch"},"body":"I implemented this with the future application in mind of keeping track of a pool that is sorted on size to generate batches of sequences of similar lengths. I'm not entirely happy with the name yet because it can be confused with an iterator that moves a sliding window over the frames of a sequence. "},"comment":{"url":"https://api.github.com/repos/bartvm/blocks/issues/comments/74852214","html_url":"https://github.com/bartvm/blocks/pull/281#issuecomment-74852214","issue_url":"https://api.github.com/repos/bartvm/blocks/issues/281","id":74852214,"user":{"login":"rizar","id":654434,"avatar_url":"https://avatars.githubusercontent.com/u/654434?v=3","gravatar_id":"","url":"https://api.github.com/users/rizar","html_url":"https://github.com/rizar","followers_url":"https://api.github.com/users/rizar/followers","following_url":"https://api.github.com/users/rizar/following{/other_user}","gists_url":"https://api.github.com/users/rizar/gists{/gist_id}","starred_url":"https://api.github.com/users/rizar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rizar/subscriptions","organizations_url":"https://api.github.com/users/rizar/orgs","repos_url":"https://api.github.com/users/rizar/repos","events_url":"https://api.github.com/users/rizar/events{/privacy}","received_events_url":"https://api.github.com/users/rizar/received_events","type":"User","site_admin":false},"created_at":"2015-02-18T11:50:29Z","updated_at":"2015-02-18T11:50:29Z","body":"Regarding sorting the data samples: this can be done very easily in Blocks. I would solve this problem by adding two data stream wrappers:\r\n\r\n```\r\nclass Sort(DataStreamWrapper):\r\n    \"\"\"Sorts the examples in a batch according to a given key function.\"\"\"\r\n    ...\r\n\r\nclass Unpack(DataStreamWrapper):\r\n     \"\"\"Frrm a data stream of batches switch to a data stream of samples.\"\"\"\r\n     ....\r\n```\r\n\r\nIn this case to have your data consist of sorted portions you add BatchDataStream -> Sort -> Unpack combinations in the beginning of the wrappers chain. That might sound complicated, but the advantage of such scheme is that every wrapper on its own is extremely simple.\r\n\r\nPlease let me know if you plan to implement sorting in this or another way, because this one of the thing we are definitely missing. It brings very cheap speedup for very time consuming applications.\r\n"}}