{"action":"created","comment":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/comments/37328874","id":37328874,"diff_hunk":"@@ -0,0 +1,135 @@\n+\"\"\"\n+Loads the d_user table into the warehouse through the pipeline via Hive.\n+\n+On the roadmap is to write a task that runs validation queries on the aggregated Hive data pre-load.\n+\"\"\"\n+from edx.analytics.tasks.vertica_load import VerticaCopyTask\n+from edx.analytics.tasks.location_per_course import ImportLastCountryOfUserToHiveTask\n+from edx.analytics.tasks.database_imports import ImportAuthUserProfileTask, ImportAuthUserTask\n+import luigi\n+from edx.analytics.tasks.util.hive import HiveTableFromQueryTask, WarehouseMixin, HivePartition\n+\n+\n+class AggregateDUserTableHive(HiveTableFromQueryTask):\n+    \"\"\"Aggregate the d_user table in Hive.\"\"\"\n+    interval = luigi.DateIntervalParameter()","path":"edx/analytics/tasks/load_warehouse_d_user.py","position":null,"original_position":15,"commit_id":"81847bb32508c99cbf84d36e636c80a5ee89707c","original_commit_id":"68f15f3bb7a04a60d5d4926713985e77b0456d0e","user":{"login":"jrowan","id":5571464,"avatar_url":"https://avatars.githubusercontent.com/u/5571464?v=3","gravatar_id":"","url":"https://api.github.com/users/jrowan","html_url":"https://github.com/jrowan","followers_url":"https://api.github.com/users/jrowan/followers","following_url":"https://api.github.com/users/jrowan/following{/other_user}","gists_url":"https://api.github.com/users/jrowan/gists{/gist_id}","starred_url":"https://api.github.com/users/jrowan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrowan/subscriptions","organizations_url":"https://api.github.com/users/jrowan/orgs","repos_url":"https://api.github.com/users/jrowan/repos","events_url":"https://api.github.com/users/jrowan/events{/privacy}","received_events_url":"https://api.github.com/users/jrowan/received_events","type":"User","site_admin":false},"body":"I made user_country_output read from the config by default.\r\n\r\nI went through the list of prod jenkins jobs and it seems most of the workflows that we are calling (all but answer_distribution, which asks for a manifest directly, and user_activity) ask for an interval directly on the CLI level.  Many of them have the Jenkins configuration set up with start and end dates, and then build the interval string passed in as a parameter on the command line in the initial script before launching the tasks.  Additionally, interval is the parameter used by the EventLogSelectionDownstreamMixin.  It may be that reworking these arguments is the correct thing to do in the future as part of a larger re-working and re-configuring, but the approach I'm currently taking (build the interval command-line parameter from provided from_date and to_date values) seemed to me like the most how-we-currently-do-things workflow for now.","created_at":"2015-08-18T17:41:52Z","updated_at":"2015-08-18T17:41:52Z","html_url":"https://github.com/edx/edx-analytics-pipeline/pull/143#discussion_r37328874","pull_request_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143","_links":{"self":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/comments/37328874"},"html":{"href":"https://github.com/edx/edx-analytics-pipeline/pull/143#discussion_r37328874"},"pull_request":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143"}}},"pull_request":{"url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143","id":42155787,"html_url":"https://github.com/edx/edx-analytics-pipeline/pull/143","diff_url":"https://github.com/edx/edx-analytics-pipeline/pull/143.diff","patch_url":"https://github.com/edx/edx-analytics-pipeline/pull/143.patch","issue_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/143","number":143,"state":"open","locked":false,"title":"Load the tables d_user and f_user_activity into the warehouse via the pipeline.","user":{"login":"jrowan","id":5571464,"avatar_url":"https://avatars.githubusercontent.com/u/5571464?v=3","gravatar_id":"","url":"https://api.github.com/users/jrowan","html_url":"https://github.com/jrowan","followers_url":"https://api.github.com/users/jrowan/followers","following_url":"https://api.github.com/users/jrowan/following{/other_user}","gists_url":"https://api.github.com/users/jrowan/gists{/gist_id}","starred_url":"https://api.github.com/users/jrowan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrowan/subscriptions","organizations_url":"https://api.github.com/users/jrowan/orgs","repos_url":"https://api.github.com/users/jrowan/repos","events_url":"https://api.github.com/users/jrowan/events{/privacy}","received_events_url":"https://api.github.com/users/jrowan/received_events","type":"User","site_admin":false},"body":"Port of two pentaho tasks for loading data warehouse tables.  Goes through the pipeline following the architecture (pipeline computation with results stored in hive | read from foreign database into hive)* -> (build hive table from hive query) -> (load hive table into the data warehouse).  Contains acceptance tests for the workflows, but note that as more tasks are built out, we will likely eventually change the workflow to have a single entry point and table loading tasks depending on prior table loading tasks.\r\n\r\n@mulby @jab5569 @brianhw ","created_at":"2015-08-11T17:35:22Z","updated_at":"2015-08-18T17:41:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2b312e5654579087c3c07fca74512c71d491fa80","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143/commits","review_comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143/comments","review_comment_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/comments{/number}","comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/143/comments","statuses_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/statuses/81847bb32508c99cbf84d36e636c80a5ee89707c","head":{"label":"edx:jamesrowan/retire-pentaho-d-user-and-f-user-activity","ref":"jamesrowan/retire-pentaho-d-user-and-f-user-activity","sha":"81847bb32508c99cbf84d36e636c80a5ee89707c","user":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?v=3","gravatar_id":"","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"repo":{"id":22929261,"name":"edx-analytics-pipeline","full_name":"edx/edx-analytics-pipeline","owner":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?v=3","gravatar_id":"","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/edx/edx-analytics-pipeline","description":"","fork":false,"url":"https://api.github.com/repos/edx/edx-analytics-pipeline","forks_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/forks","keys_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/teams","hooks_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/hooks","issue_events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/events{/number}","events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/events","assignees_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/assignees{/user}","branches_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/branches{/branch}","tags_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/tags","blobs_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/refs{/sha}","trees_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/statuses/{sha}","languages_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/languages","stargazers_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/stargazers","contributors_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/contributors","subscribers_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/subscribers","subscription_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/subscription","commits_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/commits{/sha}","git_commits_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/commits{/sha}","comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/comments{/number}","issue_comment_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/comments{/number}","contents_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/contents/{+path}","compare_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/merges","archive_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/downloads","issues_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues{/number}","pulls_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls{/number}","milestones_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/milestones{/number}","notifications_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/labels{/name}","releases_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/releases{/id}","created_at":"2014-08-13T19:57:17Z","updated_at":"2015-08-12T21:02:22Z","pushed_at":"2015-08-18T17:36:12Z","git_url":"git://github.com/edx/edx-analytics-pipeline.git","ssh_url":"git@github.com:edx/edx-analytics-pipeline.git","clone_url":"https://github.com/edx/edx-analytics-pipeline.git","svn_url":"https://github.com/edx/edx-analytics-pipeline","homepage":null,"size":6005,"stargazers_count":10,"watchers_count":10,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":8,"forks":21,"open_issues":8,"watchers":10,"default_branch":"master"}},"base":{"label":"edx:master","ref":"master","sha":"c033b89c424db4da67c2db99fb2b3db8089970bd","user":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?v=3","gravatar_id":"","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"repo":{"id":22929261,"name":"edx-analytics-pipeline","full_name":"edx/edx-analytics-pipeline","owner":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?v=3","gravatar_id":"","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/edx/edx-analytics-pipeline","description":"","fork":false,"url":"https://api.github.com/repos/edx/edx-analytics-pipeline","forks_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/forks","keys_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/teams","hooks_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/hooks","issue_events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/events{/number}","events_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/events","assignees_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/assignees{/user}","branches_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/branches{/branch}","tags_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/tags","blobs_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/refs{/sha}","trees_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/statuses/{sha}","languages_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/languages","stargazers_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/stargazers","contributors_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/contributors","subscribers_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/subscribers","subscription_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/subscription","commits_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/commits{/sha}","git_commits_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/git/commits{/sha}","comments_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/comments{/number}","issue_comment_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/comments{/number}","contents_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/contents/{+path}","compare_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/merges","archive_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/downloads","issues_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues{/number}","pulls_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls{/number}","milestones_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/milestones{/number}","notifications_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/labels{/name}","releases_url":"https://api.github.com/repos/edx/edx-analytics-pipeline/releases{/id}","created_at":"2014-08-13T19:57:17Z","updated_at":"2015-08-12T21:02:22Z","pushed_at":"2015-08-18T17:36:12Z","git_url":"git://github.com/edx/edx-analytics-pipeline.git","ssh_url":"git@github.com:edx/edx-analytics-pipeline.git","clone_url":"https://github.com/edx/edx-analytics-pipeline.git","svn_url":"https://github.com/edx/edx-analytics-pipeline","homepage":null,"size":6005,"stargazers_count":10,"watchers_count":10,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":21,"mirror_url":null,"open_issues_count":8,"forks":21,"open_issues":8,"watchers":10,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143"},"html":{"href":"https://github.com/edx/edx-analytics-pipeline/pull/143"},"issue":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/143"},"comments":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/issues/143/comments"},"review_comments":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143/comments"},"review_comment":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/pulls/143/commits"},"statuses":{"href":"https://api.github.com/repos/edx/edx-analytics-pipeline/statuses/81847bb32508c99cbf84d36e636c80a5ee89707c"}}}}