{"action":"created","comment":{"url":"https://api.github.com/repos/quantopian/zipline/pulls/comments/26523202","id":26523202,"diff_hunk":"@@ -111,46 +112,43 @@ def __init__(self, data, **kwargs):\n \n         # Hash_value for downstream sorting.\n         self.arg_string = hash_args(data, **kwargs)\n-\n-        self._raw_data = None\n-\n-    @property\n-    def mapping(self):\n-        mapping = {\n-            'dt': (lambda x: x, 'dt'),\n-            'sid': (lambda x: x, 'sid'),\n-            'price': (float, 'price'),\n-            'volume': (int, 'volume'),\n-        }\n-\n-        # Add additional fields.\n-        for field_name in self.data.minor_axis:\n-            if field_name in ['price', 'volume', 'dt', 'sid']:\n-                continue\n-            mapping[field_name] = (lambda x: x, field_name)\n-\n-        return mapping\n+        self.it = self.mapped_data\n \n     @property\n     def instance_hash(self):\n         return self.arg_string\n \n-    def raw_data_gen(self):\n-        for dt in self.data.major_axis:\n-            df = self.data.major_xs(dt)\n-            for sid, series in df.iteritems():\n-                if sid in self.sids:\n-                    event = {\n-                        'dt': dt,\n-                        'sid': sid,\n-                    }\n-                    for field_name, value in series.iteritems():\n-                        event[field_name] = value\n-\n-                    yield event\n+    def raw_data(self):\n+        # treat yoself\n+        pass\n \n     @property\n-    def raw_data(self):\n-        if not self._raw_data:\n-            self._raw_data = self.raw_data_gen()\n-        return self._raw_data\n+    def mapped_data(self):\n+        # currently volume is same dtype as prices, float64","path":"zipline/sources/data_frame_source.py","position":60,"original_position":60,"commit_id":"2f8cd408875f8ca9d818583e5bb39a06b214fdde","original_commit_id":"2f8cd408875f8ca9d818583e5bb39a06b214fdde","user":{"login":"ehebert","id":185744,"avatar_url":"https://avatars.githubusercontent.com/u/185744?v=3","gravatar_id":"","url":"https://api.github.com/users/ehebert","html_url":"https://github.com/ehebert","followers_url":"https://api.github.com/users/ehebert/followers","following_url":"https://api.github.com/users/ehebert/following{/other_user}","gists_url":"https://api.github.com/users/ehebert/gists{/gist_id}","starred_url":"https://api.github.com/users/ehebert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ehebert/subscriptions","organizations_url":"https://api.github.com/users/ehebert/orgs","repos_url":"https://api.github.com/users/ehebert/repos","events_url":"https://api.github.com/users/ehebert/events{/privacy}","received_events_url":"https://api.github.com/users/ehebert/received_events","type":"User","site_admin":false},"body":"It's probably better, or at least ok, to have the volumes as float64.\r\n\r\nFor places where volume needs to be multiplied by the price, it should be faster to do the float*float.\r\n\r\nAlso, allows for shaping data in both directions, i.e. ohlcv can be a vector... if we care about that.","created_at":"2015-03-16T20:18:01Z","updated_at":"2015-03-16T20:18:01Z","html_url":"https://github.com/quantopian/zipline/pull/534#discussion_r26523202","pull_request_url":"https://api.github.com/repos/quantopian/zipline/pulls/534","_links":{"self":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/comments/26523202"},"html":{"href":"https://github.com/quantopian/zipline/pull/534#discussion_r26523202"},"pull_request":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/534"}}},"pull_request":{"url":"https://api.github.com/repos/quantopian/zipline/pulls/534","id":31163289,"html_url":"https://github.com/quantopian/zipline/pull/534","diff_url":"https://github.com/quantopian/zipline/pull/534.diff","patch_url":"https://github.com/quantopian/zipline/pull/534.patch","issue_url":"https://api.github.com/repos/quantopian/zipline/issues/534","number":534,"state":"open","locked":false,"title":"Dataverse","user":{"login":"dalejung","id":610115,"avatar_url":"https://avatars.githubusercontent.com/u/610115?v=3","gravatar_id":"","url":"https://api.github.com/users/dalejung","html_url":"https://github.com/dalejung","followers_url":"https://api.github.com/users/dalejung/followers","following_url":"https://api.github.com/users/dalejung/following{/other_user}","gists_url":"https://api.github.com/users/dalejung/gists{/gist_id}","starred_url":"https://api.github.com/users/dalejung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dalejung/subscriptions","organizations_url":"https://api.github.com/users/dalejung/orgs","repos_url":"https://api.github.com/users/dalejung/repos","events_url":"https://api.github.com/users/dalejung/events{/privacy}","received_events_url":"https://api.github.com/users/dalejung/received_events","type":"User","site_admin":false},"body":"WIP \r\n### WIP \r\n## WIP \r\n# WIP.\r\n\r\nBased off of the position_tracker refactor. Once that gets merged it'll cut down the noise.\r\n\r\nExecutive Summary: We have array backed datasources. All path-*independent* data should **not** be calculated iteratively. Vectorize and fast-path the execution.\r\n\r\n* [ ] Refactor the HistoryChunker into a UniverseChunker. In general, I'd like the BarData to be based off of that.\r\n* [ ] Have a MetaDataverse that choses the BaseDataverse / BacktestDataverse depending on what datasource it gets. Right now it's chosen explicitly, which would normally be fine, except the RandomWalkSource balks since it's not an array.\r\n* [ ] Change position tracker to literally only track positions. Position values should be handled higher up and not require update_last_sale.\r\n* [ ] Build out a matrix of the types of TradeEvents/Dataverses we can have. Technically speaking, the DataPanelSource should be allowed to emit WideTradeBars while using the BaseDataverse.\r\n* [ ] Create a DataSource that is compatible with BacktestDataverse, this will only emit TradeEvents that are used to signifiy new data, but won't necessarily carry any data with them, be it dicts or ndarrays.\r\n* [ ] PositionTracker should emit new Position events. Somewhere we need some sort of pubsub so that when Universes get ported, we are not polling the Positions for sids, but instead only updating when positions are created/deleted.\r\n* [ ] Once Magic BacktestBarData is setup, also wire it to expose vectorized computations from Research.\r\n* [ ] `history` should return a DataFrame proxy so that we can fast-path lines like `history(10, '1m', 'price').mean()`. The goal is to not even create the DataFrame in the first place.\r\n* [ ] Outline the require immutability of objects like BarData and HistoryFrame. This can be opt-in for speed.\r\n* [ ] When the chunked historycontainer rolls over, carry over the last known values for ffill.","created_at":"2015-03-13T20:58:50Z","updated_at":"2015-03-16T20:18:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c19cc4520ac6b29e44210187394b61dc0a5ed1c5","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/quantopian/zipline/pulls/534/commits","review_comments_url":"https://api.github.com/repos/quantopian/zipline/pulls/534/comments","review_comment_url":"https://api.github.com/repos/quantopian/zipline/pulls/comments{/number}","comments_url":"https://api.github.com/repos/quantopian/zipline/issues/534/comments","statuses_url":"https://api.github.com/repos/quantopian/zipline/statuses/2f8cd408875f8ca9d818583e5bb39a06b214fdde","head":{"label":"dalejung:dataverse","ref":"dataverse","sha":"2f8cd408875f8ca9d818583e5bb39a06b214fdde","user":{"login":"dalejung","id":610115,"avatar_url":"https://avatars.githubusercontent.com/u/610115?v=3","gravatar_id":"","url":"https://api.github.com/users/dalejung","html_url":"https://github.com/dalejung","followers_url":"https://api.github.com/users/dalejung/followers","following_url":"https://api.github.com/users/dalejung/following{/other_user}","gists_url":"https://api.github.com/users/dalejung/gists{/gist_id}","starred_url":"https://api.github.com/users/dalejung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dalejung/subscriptions","organizations_url":"https://api.github.com/users/dalejung/orgs","repos_url":"https://api.github.com/users/dalejung/repos","events_url":"https://api.github.com/users/dalejung/events{/privacy}","received_events_url":"https://api.github.com/users/dalejung/received_events","type":"User","site_admin":false},"repo":{"id":24394752,"name":"zipline","full_name":"dalejung/zipline","owner":{"login":"dalejung","id":610115,"avatar_url":"https://avatars.githubusercontent.com/u/610115?v=3","gravatar_id":"","url":"https://api.github.com/users/dalejung","html_url":"https://github.com/dalejung","followers_url":"https://api.github.com/users/dalejung/followers","following_url":"https://api.github.com/users/dalejung/following{/other_user}","gists_url":"https://api.github.com/users/dalejung/gists{/gist_id}","starred_url":"https://api.github.com/users/dalejung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dalejung/subscriptions","organizations_url":"https://api.github.com/users/dalejung/orgs","repos_url":"https://api.github.com/users/dalejung/repos","events_url":"https://api.github.com/users/dalejung/events{/privacy}","received_events_url":"https://api.github.com/users/dalejung/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/dalejung/zipline","description":"Zipline, a Pythonic Algorithmic Trading Library","fork":true,"url":"https://api.github.com/repos/dalejung/zipline","forks_url":"https://api.github.com/repos/dalejung/zipline/forks","keys_url":"https://api.github.com/repos/dalejung/zipline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dalejung/zipline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dalejung/zipline/teams","hooks_url":"https://api.github.com/repos/dalejung/zipline/hooks","issue_events_url":"https://api.github.com/repos/dalejung/zipline/issues/events{/number}","events_url":"https://api.github.com/repos/dalejung/zipline/events","assignees_url":"https://api.github.com/repos/dalejung/zipline/assignees{/user}","branches_url":"https://api.github.com/repos/dalejung/zipline/branches{/branch}","tags_url":"https://api.github.com/repos/dalejung/zipline/tags","blobs_url":"https://api.github.com/repos/dalejung/zipline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dalejung/zipline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dalejung/zipline/git/refs{/sha}","trees_url":"https://api.github.com/repos/dalejung/zipline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dalejung/zipline/statuses/{sha}","languages_url":"https://api.github.com/repos/dalejung/zipline/languages","stargazers_url":"https://api.github.com/repos/dalejung/zipline/stargazers","contributors_url":"https://api.github.com/repos/dalejung/zipline/contributors","subscribers_url":"https://api.github.com/repos/dalejung/zipline/subscribers","subscription_url":"https://api.github.com/repos/dalejung/zipline/subscription","commits_url":"https://api.github.com/repos/dalejung/zipline/commits{/sha}","git_commits_url":"https://api.github.com/repos/dalejung/zipline/git/commits{/sha}","comments_url":"https://api.github.com/repos/dalejung/zipline/comments{/number}","issue_comment_url":"https://api.github.com/repos/dalejung/zipline/issues/comments{/number}","contents_url":"https://api.github.com/repos/dalejung/zipline/contents/{+path}","compare_url":"https://api.github.com/repos/dalejung/zipline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dalejung/zipline/merges","archive_url":"https://api.github.com/repos/dalejung/zipline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dalejung/zipline/downloads","issues_url":"https://api.github.com/repos/dalejung/zipline/issues{/number}","pulls_url":"https://api.github.com/repos/dalejung/zipline/pulls{/number}","milestones_url":"https://api.github.com/repos/dalejung/zipline/milestones{/number}","notifications_url":"https://api.github.com/repos/dalejung/zipline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dalejung/zipline/labels{/name}","releases_url":"https://api.github.com/repos/dalejung/zipline/releases{/id}","created_at":"2014-09-24T00:33:53Z","updated_at":"2015-03-11T18:22:01Z","pushed_at":"2015-03-13T20:53:28Z","git_url":"git://github.com/dalejung/zipline.git","ssh_url":"git@github.com:dalejung/zipline.git","clone_url":"https://github.com/dalejung/zipline.git","svn_url":"https://github.com/dalejung/zipline","homepage":"","size":7675,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"quantopian:master","ref":"master","sha":"6cd2f4f1f2f4a4dca74fcfd6484278cc90e6f77a","user":{"login":"quantopian","id":1393215,"avatar_url":"https://avatars.githubusercontent.com/u/1393215?v=3","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"repo":{"id":6297520,"name":"zipline","full_name":"quantopian/zipline","owner":{"login":"quantopian","id":1393215,"avatar_url":"https://avatars.githubusercontent.com/u/1393215?v=3","gravatar_id":"","url":"https://api.github.com/users/quantopian","html_url":"https://github.com/quantopian","followers_url":"https://api.github.com/users/quantopian/followers","following_url":"https://api.github.com/users/quantopian/following{/other_user}","gists_url":"https://api.github.com/users/quantopian/gists{/gist_id}","starred_url":"https://api.github.com/users/quantopian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/quantopian/subscriptions","organizations_url":"https://api.github.com/users/quantopian/orgs","repos_url":"https://api.github.com/users/quantopian/repos","events_url":"https://api.github.com/users/quantopian/events{/privacy}","received_events_url":"https://api.github.com/users/quantopian/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/quantopian/zipline","description":"Zipline, a Pythonic Algorithmic Trading Library","fork":false,"url":"https://api.github.com/repos/quantopian/zipline","forks_url":"https://api.github.com/repos/quantopian/zipline/forks","keys_url":"https://api.github.com/repos/quantopian/zipline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/quantopian/zipline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/quantopian/zipline/teams","hooks_url":"https://api.github.com/repos/quantopian/zipline/hooks","issue_events_url":"https://api.github.com/repos/quantopian/zipline/issues/events{/number}","events_url":"https://api.github.com/repos/quantopian/zipline/events","assignees_url":"https://api.github.com/repos/quantopian/zipline/assignees{/user}","branches_url":"https://api.github.com/repos/quantopian/zipline/branches{/branch}","tags_url":"https://api.github.com/repos/quantopian/zipline/tags","blobs_url":"https://api.github.com/repos/quantopian/zipline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/quantopian/zipline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/quantopian/zipline/git/refs{/sha}","trees_url":"https://api.github.com/repos/quantopian/zipline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/quantopian/zipline/statuses/{sha}","languages_url":"https://api.github.com/repos/quantopian/zipline/languages","stargazers_url":"https://api.github.com/repos/quantopian/zipline/stargazers","contributors_url":"https://api.github.com/repos/quantopian/zipline/contributors","subscribers_url":"https://api.github.com/repos/quantopian/zipline/subscribers","subscription_url":"https://api.github.com/repos/quantopian/zipline/subscription","commits_url":"https://api.github.com/repos/quantopian/zipline/commits{/sha}","git_commits_url":"https://api.github.com/repos/quantopian/zipline/git/commits{/sha}","comments_url":"https://api.github.com/repos/quantopian/zipline/comments{/number}","issue_comment_url":"https://api.github.com/repos/quantopian/zipline/issues/comments{/number}","contents_url":"https://api.github.com/repos/quantopian/zipline/contents/{+path}","compare_url":"https://api.github.com/repos/quantopian/zipline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/quantopian/zipline/merges","archive_url":"https://api.github.com/repos/quantopian/zipline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/quantopian/zipline/downloads","issues_url":"https://api.github.com/repos/quantopian/zipline/issues{/number}","pulls_url":"https://api.github.com/repos/quantopian/zipline/pulls{/number}","milestones_url":"https://api.github.com/repos/quantopian/zipline/milestones{/number}","notifications_url":"https://api.github.com/repos/quantopian/zipline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/quantopian/zipline/labels{/name}","releases_url":"https://api.github.com/repos/quantopian/zipline/releases{/id}","created_at":"2012-10-19T15:50:29Z","updated_at":"2015-03-16T11:37:42Z","pushed_at":"2015-03-16T17:36:12Z","git_url":"git://github.com/quantopian/zipline.git","ssh_url":"git@github.com:quantopian/zipline.git","clone_url":"https://github.com/quantopian/zipline.git","svn_url":"https://github.com/quantopian/zipline","homepage":"","size":17888,"stargazers_count":2065,"watchers_count":2065,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":509,"mirror_url":null,"open_issues_count":106,"forks":509,"open_issues":106,"watchers":2065,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/534"},"html":{"href":"https://github.com/quantopian/zipline/pull/534"},"issue":{"href":"https://api.github.com/repos/quantopian/zipline/issues/534"},"comments":{"href":"https://api.github.com/repos/quantopian/zipline/issues/534/comments"},"review_comments":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/534/comments"},"review_comment":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/quantopian/zipline/pulls/534/commits"},"statuses":{"href":"https://api.github.com/repos/quantopian/zipline/statuses/2f8cd408875f8ca9d818583e5bb39a06b214fdde"}}}}