{"action":"created","issue":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/175","labels_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/175/labels{/name}","comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/175/comments","events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/175/events","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/175","id":117462618,"number":175,"title":"Student module engagement","user":{"login":"mulby","id":627177,"avatar_url":"https://avatars.githubusercontent.com/u/627177?v=3","gravatar_id":"","url":"https://api.github.com/users/mulby","html_url":"https://github.com/mulby","followers_url":"https://api.github.com/users/mulby/followers","following_url":"https://api.github.com/users/mulby/following{/other_user}","gists_url":"https://api.github.com/users/mulby/gists{/gist_id}","starred_url":"https://api.github.com/users/mulby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mulby/subscriptions","organizations_url":"https://api.github.com/users/mulby/orgs","repos_url":"https://api.github.com/users/mulby/repos","events_url":"https://api.github.com/users/mulby/events{/privacy}","received_events_url":"https://api.github.com/users/mulby/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-17T22:09:33Z","updated_at":"2015-12-01T14:44:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/175","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/175","diff_url":"https://github.com/edx/edx-analytics-pipeline/pull/175.diff","patch_url":"https://github.com/edx/edx-analytics-pipeline/pull/175.patch"},"body":"@brianhw here is an early review for preliminary feedback\r\n\r\nTodo:\r\n\r\n- [x] A LOT of unit tests (pretty much all of record.py and big chunks of engagement.py)\r\n- [x] An acceptance test\r\n- [x] pep8/pylint\r\n- [x] more documentation on the workflow\r\n- [ ] figure out overwrite semantics\r\n\r\nQuestions I have that I would love your opinion/thoughts on:\r\n\r\n1. How do you feel about the concept of a \"Record\" where we represent the schema *once* and then use it to handle reading and writing tuples that are Hive compatible as well as generate the schema for the other systems? I just spiked this through to see how it felt... let me know what you think. If we like it I'll write an extensive test suite for it. You can see example usage in engagement.py\r\n2. How do you feel about the new Hive utilities? One of the goals was to allow us to keep partitions as output targets, but also allow us to use the partitions as input sources to hadoop jobs. I also wanted to tease apart the concept of \"table\" and partition to allow us to more flexibly combine them. In this case, allowing us to have many partitions and only one table. Again, there are example usages in engagement.py\r\n3. Should I keep the sparse course engagement stuff in this PR? I feel like it might make more sense to put it into the PR that computes the roster table since that's where it's used primarily."},"comment":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/comments/160987112","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/175#issuecomment-160987112","issue_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/175","id":160987112,"user":{"login":"mulby","id":627177,"avatar_url":"https://avatars.githubusercontent.com/u/627177?v=3","gravatar_id":"","url":"https://api.github.com/users/mulby","html_url":"https://github.com/mulby","followers_url":"https://api.github.com/users/mulby/followers","following_url":"https://api.github.com/users/mulby/following{/other_user}","gists_url":"https://api.github.com/users/mulby/gists{/gist_id}","starred_url":"https://api.github.com/users/mulby/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mulby/subscriptions","organizations_url":"https://api.github.com/users/mulby/orgs","repos_url":"https://api.github.com/users/mulby/repos","events_url":"https://api.github.com/users/mulby/events{/privacy}","received_events_url":"https://api.github.com/users/mulby/received_events","type":"User","site_admin":false},"created_at":"2015-12-01T14:44:10Z","updated_at":"2015-12-01T14:44:10Z","body":"@brianhw this is ready for review now"}}