{"action":"created","issue":{"url":"https://api.github.com/repos/esjewett/reductio/issues/12","labels_url":"https://api.github.com/repos/esjewett/reductio/issues/12/labels{/name}","comments_url":"https://api.github.com/repos/esjewett/reductio/issues/12/comments","events_url":"https://api.github.com/repos/esjewett/reductio/issues/12/events","html_url":"https://github.com/esjewett/reductio/issues/12","id":55940274,"number":12,"title":"groupAll Performance","user":{"login":"JackStouffer","id":680068,"avatar_url":"https://avatars.githubusercontent.com/u/680068?v=3","gravatar_id":"","url":"https://api.github.com/users/JackStouffer","html_url":"https://github.com/JackStouffer","followers_url":"https://api.github.com/users/JackStouffer/followers","following_url":"https://api.github.com/users/JackStouffer/following{/other_user}","gists_url":"https://api.github.com/users/JackStouffer/gists{/gist_id}","starred_url":"https://api.github.com/users/JackStouffer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JackStouffer/subscriptions","organizations_url":"https://api.github.com/users/JackStouffer/orgs","repos_url":"https://api.github.com/users/JackStouffer/repos","events_url":"https://api.github.com/users/JackStouffer/events{/privacy}","received_events_url":"https://api.github.com/users/JackStouffer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-01-29T18:52:45Z","updated_at":"2015-01-30T02:37:36Z","closed_at":null,"body":"From https://github.com/square/crossfilter/issues/100\r\n\r\nI added a column in the data via js with date.toDateSting() to test your theory. Here is the changed code.\r\n\r\n```\r\nvar testDim = cs_data.dimension(function(d) {return d.DateString;});\r\nvar test_shipments = testDim.group().reduceSum(dc.pluck('DollarValue'));\r\n\r\nreducer = reductio().groupAll(function(record) {\r\n    var idx = mapped_date_array.indexOf(record.DateString);\r\n\r\n    if (record.ShipDate < date_array[30]) {\r\n        return [mapped_date_array[idx]];\r\n    } else {\r\n        var i = 0;\r\n        var return_array = [];\r\n                    \r\n        // we are finding the 30 day moving avg \r\n        for (i = 30; i >= 0; i--) {\r\n              if (typeof(mapped_date_array[idx - i]) != \"undefined\") {\r\n                   return_array.push(mapped_date_array[idx - i]);\r\n              }\r\n        }\r\n\r\n        return return_array;\r\n    }\r\n}).count(true).sum(function(d) { return d.DollarValue; }).avg(true);\r\n```\r\n\r\nand ran\r\n\r\n```\r\nconsole.time(\"values\");\r\nshipments_moving_avg.value();\r\nconsole.timeEnd(\"values\");\r\n```\r\n\r\nto test.\r\n\r\nThis yielded \"values: 9869.228ms\""},"comment":{"url":"https://api.github.com/repos/esjewett/reductio/issues/comments/72144687","html_url":"https://github.com/esjewett/reductio/issues/12#issuecomment-72144687","issue_url":"https://api.github.com/repos/esjewett/reductio/issues/12","id":72144687,"user":{"login":"esjewett","id":4822,"avatar_url":"https://avatars.githubusercontent.com/u/4822?v=3","gravatar_id":"","url":"https://api.github.com/users/esjewett","html_url":"https://github.com/esjewett","followers_url":"https://api.github.com/users/esjewett/followers","following_url":"https://api.github.com/users/esjewett/following{/other_user}","gists_url":"https://api.github.com/users/esjewett/gists{/gist_id}","starred_url":"https://api.github.com/users/esjewett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/esjewett/subscriptions","organizations_url":"https://api.github.com/users/esjewett/orgs","repos_url":"https://api.github.com/users/esjewett/repos","events_url":"https://api.github.com/users/esjewett/events{/privacy}","received_events_url":"https://api.github.com/users/esjewett/received_events","type":"User","site_admin":false},"created_at":"2015-01-30T02:37:36Z","updated_at":"2015-01-30T02:37:36Z","body":"Try this gist: https://gist.github.com/esjewett/8291936a4e4054342ba4\r\n\r\nIt's about 3x faster than the initial approach in my tests, taking about 5 seconds on my machine for the initial ```shipments_moving_avg.value()``` vs. 15-25 for different variations of the initial approach. I put in some comments regarding the performance and date-handling tricks I'm using. They are applicable to all Crossfilter applications, not just this one.\r\n\r\n![screen shot 2015-01-29 at 8 34 55 pm](https://cloud.githubusercontent.com/assets/4822/5970265/5ada2562-a7f6-11e4-8f4e-a768e95f0a64.png)\r\n\r\nKeep in mind that a 30-day moving average requires aggregating 30x the number of records because each record is placed in 30 groups and run through the reducers for each of those groups to aggregate it. So it's as if your data set actually has over 3 million records in it.\r\n\r\nIf you slowly add back your other dimensions while keeping an eye on the performance, you should be in OK shape to do interactive data analysis with this data set using Crossfilter and Reductio. There may be a few seconds lag for the big chunky filters (like if you filter by country), but dragging on timelines and such should work pretty interactively.\r\n\r\nI'll try to keep using this data as a test-case for performance, but I think at this point further performance improvements may be marginal."}}