{"action":"created","issue":{"url":"https://api.github.com/repos/blaze/castra/issues/36","labels_url":"https://api.github.com/repos/blaze/castra/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/blaze/castra/issues/36/comments","events_url":"https://api.github.com/repos/blaze/castra/issues/36/events","html_url":"https://github.com/blaze/castra/issues/36","id":103305424,"number":36,"title":"Rework of castra","user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars.githubusercontent.com/u/2783717?v=3","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-08-26T15:59:16Z","updated_at":"2015-08-26T19:21:20Z","closed_at":null,"body":"I've been working on a refactor of Castra - before I spend any more time on this, I should probably get some feedback. Here's the plan:\r\n\r\nIssues I'm attempting to solve:\r\n\r\n1. Castra provides no validation that partitions split evenly - indices like `[[1, 2, 3, 3], [3, 3, 4, 5, 6], ...]` were possible (and happened to me)\r\n\r\n2. Castra provides no easy way to say \"partition weekly\", without manually doing the partitioning elsewhere (issue #3)\r\n\r\nThe plan:\r\n\r\n1. Add `partitionby=None` to the `init` signature. This will live in `meta`. If `None`, no repartitioning is done by Castra. Can also be a time period (things you can pass to `resample` in pandas).\r\n\r\n2. `extend` checks current partitions for equality overlap (even if `partitionby=None`). There are 3 cases that can happen here:\r\n  1. Start of new frame is before end of existing partition. This errors\r\n  2. Start of new frame is equal to end of existing partition. The equal parts are split off and appended to existing partition. Remainder is stored as new partition.\r\n  3. Start of new frame is after existing partition. New frame is written to disk (current behavior)\r\n\r\n3. If `partitionby != None`, then data is partitioned by Castra into blocks. `extend` should still take large dataframes (calling extend on a row is a bad idea), but will group them into partitions based on the rule passed to `partitionby`. Using the functionality provided by bloscpack, the on disk partitions can be appended to with little overhead. This makes writing in cases where this happens slightly slower, but has no penalty on reads.\r\n\r\n4. Add `extend_sequence` function. This takes an iterable of dataframes (can be a generator), and does the partitioning in memory instead of on disk. This will be faster than calling `extend` in a loop (no on disk appends), but will result in the same disk file format.\r\n\r\nThis method means that the disk will match what's in memory after calls to `extend` or `extend_sequence` complete, will allow castra to do partitioning for the user, and will ensure that the partitions are valid. I have a crude version of this working now, and have found writes to be only slightly penalized when appends happen (no penalty if they don't), and no penalty for reading from disk."},"comment":{"url":"https://api.github.com/repos/blaze/castra/issues/comments/135145033","html_url":"https://github.com/blaze/castra/issues/36#issuecomment-135145033","issue_url":"https://api.github.com/repos/blaze/castra/issues/36","id":135145033,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2015-08-26T19:21:20Z","updated_at":"2015-08-26T19:21:20Z","body":"If there is an application where the second case ends up being really important (e.g. log files that come in slightly out of order) then that sounds like a motivating use case.  Do you have such a case?"}}