{"action":"created","comment":{"url":"https://api.github.com/repos/allenai/pipeline/pulls/comments/31543925","id":31543925,"diff_hunk":"@@ -0,0 +1,42 @@\n+package org.allenai.pipeline.spark","path":"spark/src/main/scala/org/allenai/pipeline/spark/PartitionedRddFileArtifact.scala","position":1,"original_position":1,"commit_id":"e63ea47f47c4e8a5153ad3e018ace229a86884f2","original_commit_id":"e63ea47f47c4e8a5153ad3e018ace229a86884f2","user":{"login":"rodneykinney","id":3767011,"avatar_url":"https://avatars.githubusercontent.com/u/3767011?v=3","gravatar_id":"","url":"https://api.github.com/users/rodneykinney","html_url":"https://github.com/rodneykinney","followers_url":"https://api.github.com/users/rodneykinney/followers","following_url":"https://api.github.com/users/rodneykinney/following{/other_user}","gists_url":"https://api.github.com/users/rodneykinney/gists{/gist_id}","starred_url":"https://api.github.com/users/rodneykinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodneykinney/subscriptions","organizations_url":"https://api.github.com/users/rodneykinney/orgs","repos_url":"https://api.github.com/users/rodneykinney/repos","events_url":"https://api.github.com/users/rodneykinney/events{/privacy}","received_events_url":"https://api.github.com/users/rodneykinney/received_events","type":"User","site_admin":false},"body":"This Artifact class is a common interface for saving RDDs.  Previously, we had separate implementations for File- and S3-based artifacts, which required some ugly run-time dynamic casting.","created_at":"2015-06-02T16:49:39Z","updated_at":"2015-06-02T16:49:39Z","html_url":"https://github.com/allenai/pipeline/pull/31#discussion_r31543925","pull_request_url":"https://api.github.com/repos/allenai/pipeline/pulls/31","_links":{"self":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/comments/31543925"},"html":{"href":"https://github.com/allenai/pipeline/pull/31#discussion_r31543925"},"pull_request":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/31"}}},"pull_request":{"url":"https://api.github.com/repos/allenai/pipeline/pulls/31","id":36744907,"html_url":"https://github.com/allenai/pipeline/pull/31","diff_url":"https://github.com/allenai/pipeline/pull/31.diff","patch_url":"https://github.com/allenai/pipeline/pull/31.patch","issue_url":"https://api.github.com/repos/allenai/pipeline/issues/31","number":31,"state":"open","locked":false,"title":"Refactor into sub-projects","user":{"login":"rodneykinney","id":3767011,"avatar_url":"https://avatars.githubusercontent.com/u/3767011?v=3","gravatar_id":"","url":"https://api.github.com/users/rodneykinney","html_url":"https://github.com/rodneykinney","followers_url":"https://api.github.com/users/rodneykinney/followers","following_url":"https://api.github.com/users/rodneykinney/following{/other_user}","gists_url":"https://api.github.com/users/rodneykinney/gists{/gist_id}","starred_url":"https://api.github.com/users/rodneykinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodneykinney/subscriptions","organizations_url":"https://api.github.com/users/rodneykinney/orgs","repos_url":"https://api.github.com/users/rodneykinney/repos","events_url":"https://api.github.com/users/rodneykinney/events{/privacy}","received_events_url":"https://api.github.com/users/rodneykinney/received_events","type":"User","site_admin":false},"body":"Breaks up the pipeline project into the following sub-projects:\r\n\r\n - core (no cloud computing dependencies)\r\n - s3 (support for reading/writing to S3)\r\n - spark (support for running Spark-based pipelines)\r\n\r\nThe spark project contains much of the code from the scholar/offline project that was specific to Spark but not specific to S2.\r\n\r\nThe core project has been enhanced to support the naming of pipeline steps.","created_at":"2015-06-02T16:00:26Z","updated_at":"2015-06-02T16:49:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c38e9a0f792085fda3d9e4f2c0116e3123e47e02","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/allenai/pipeline/pulls/31/commits","review_comments_url":"https://api.github.com/repos/allenai/pipeline/pulls/31/comments","review_comment_url":"https://api.github.com/repos/allenai/pipeline/pulls/comments{/number}","comments_url":"https://api.github.com/repos/allenai/pipeline/issues/31/comments","statuses_url":"https://api.github.com/repos/allenai/pipeline/statuses/e63ea47f47c4e8a5153ad3e018ace229a86884f2","head":{"label":"rodneykinney:master","ref":"master","sha":"e63ea47f47c4e8a5153ad3e018ace229a86884f2","user":{"login":"rodneykinney","id":3767011,"avatar_url":"https://avatars.githubusercontent.com/u/3767011?v=3","gravatar_id":"","url":"https://api.github.com/users/rodneykinney","html_url":"https://github.com/rodneykinney","followers_url":"https://api.github.com/users/rodneykinney/followers","following_url":"https://api.github.com/users/rodneykinney/following{/other_user}","gists_url":"https://api.github.com/users/rodneykinney/gists{/gist_id}","starred_url":"https://api.github.com/users/rodneykinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodneykinney/subscriptions","organizations_url":"https://api.github.com/users/rodneykinney/orgs","repos_url":"https://api.github.com/users/rodneykinney/repos","events_url":"https://api.github.com/users/rodneykinney/events{/privacy}","received_events_url":"https://api.github.com/users/rodneykinney/received_events","type":"User","site_admin":false},"repo":{"id":32085607,"name":"pipeline","full_name":"rodneykinney/pipeline","owner":{"login":"rodneykinney","id":3767011,"avatar_url":"https://avatars.githubusercontent.com/u/3767011?v=3","gravatar_id":"","url":"https://api.github.com/users/rodneykinney","html_url":"https://github.com/rodneykinney","followers_url":"https://api.github.com/users/rodneykinney/followers","following_url":"https://api.github.com/users/rodneykinney/following{/other_user}","gists_url":"https://api.github.com/users/rodneykinney/gists{/gist_id}","starred_url":"https://api.github.com/users/rodneykinney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rodneykinney/subscriptions","organizations_url":"https://api.github.com/users/rodneykinney/orgs","repos_url":"https://api.github.com/users/rodneykinney/repos","events_url":"https://api.github.com/users/rodneykinney/events{/privacy}","received_events_url":"https://api.github.com/users/rodneykinney/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/rodneykinney/pipeline","description":"Library for building reproducible data pipelines to support experimentation","fork":true,"url":"https://api.github.com/repos/rodneykinney/pipeline","forks_url":"https://api.github.com/repos/rodneykinney/pipeline/forks","keys_url":"https://api.github.com/repos/rodneykinney/pipeline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/rodneykinney/pipeline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/rodneykinney/pipeline/teams","hooks_url":"https://api.github.com/repos/rodneykinney/pipeline/hooks","issue_events_url":"https://api.github.com/repos/rodneykinney/pipeline/issues/events{/number}","events_url":"https://api.github.com/repos/rodneykinney/pipeline/events","assignees_url":"https://api.github.com/repos/rodneykinney/pipeline/assignees{/user}","branches_url":"https://api.github.com/repos/rodneykinney/pipeline/branches{/branch}","tags_url":"https://api.github.com/repos/rodneykinney/pipeline/tags","blobs_url":"https://api.github.com/repos/rodneykinney/pipeline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/rodneykinney/pipeline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/rodneykinney/pipeline/git/refs{/sha}","trees_url":"https://api.github.com/repos/rodneykinney/pipeline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/rodneykinney/pipeline/statuses/{sha}","languages_url":"https://api.github.com/repos/rodneykinney/pipeline/languages","stargazers_url":"https://api.github.com/repos/rodneykinney/pipeline/stargazers","contributors_url":"https://api.github.com/repos/rodneykinney/pipeline/contributors","subscribers_url":"https://api.github.com/repos/rodneykinney/pipeline/subscribers","subscription_url":"https://api.github.com/repos/rodneykinney/pipeline/subscription","commits_url":"https://api.github.com/repos/rodneykinney/pipeline/commits{/sha}","git_commits_url":"https://api.github.com/repos/rodneykinney/pipeline/git/commits{/sha}","comments_url":"https://api.github.com/repos/rodneykinney/pipeline/comments{/number}","issue_comment_url":"https://api.github.com/repos/rodneykinney/pipeline/issues/comments{/number}","contents_url":"https://api.github.com/repos/rodneykinney/pipeline/contents/{+path}","compare_url":"https://api.github.com/repos/rodneykinney/pipeline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/rodneykinney/pipeline/merges","archive_url":"https://api.github.com/repos/rodneykinney/pipeline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/rodneykinney/pipeline/downloads","issues_url":"https://api.github.com/repos/rodneykinney/pipeline/issues{/number}","pulls_url":"https://api.github.com/repos/rodneykinney/pipeline/pulls{/number}","milestones_url":"https://api.github.com/repos/rodneykinney/pipeline/milestones{/number}","notifications_url":"https://api.github.com/repos/rodneykinney/pipeline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/rodneykinney/pipeline/labels{/name}","releases_url":"https://api.github.com/repos/rodneykinney/pipeline/releases{/id}","created_at":"2015-03-12T15:46:16Z","updated_at":"2015-03-30T21:54:04Z","pushed_at":"2015-06-02T15:47:36Z","git_url":"git://github.com/rodneykinney/pipeline.git","ssh_url":"git@github.com:rodneykinney/pipeline.git","clone_url":"https://github.com/rodneykinney/pipeline.git","svn_url":"https://github.com/rodneykinney/pipeline","homepage":"","size":828,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"allenai:master","ref":"master","sha":"53bc10a7d9d5b1ee125a6d688e476daed805d358","user":{"login":"allenai","id":5667695,"avatar_url":"https://avatars.githubusercontent.com/u/5667695?v=3","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"repo":{"id":31559978,"name":"pipeline","full_name":"allenai/pipeline","owner":{"login":"allenai","id":5667695,"avatar_url":"https://avatars.githubusercontent.com/u/5667695?v=3","gravatar_id":"","url":"https://api.github.com/users/allenai","html_url":"https://github.com/allenai","followers_url":"https://api.github.com/users/allenai/followers","following_url":"https://api.github.com/users/allenai/following{/other_user}","gists_url":"https://api.github.com/users/allenai/gists{/gist_id}","starred_url":"https://api.github.com/users/allenai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allenai/subscriptions","organizations_url":"https://api.github.com/users/allenai/orgs","repos_url":"https://api.github.com/users/allenai/repos","events_url":"https://api.github.com/users/allenai/events{/privacy}","received_events_url":"https://api.github.com/users/allenai/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/allenai/pipeline","description":"Library for building reproducible data pipelines to support experimentation","fork":false,"url":"https://api.github.com/repos/allenai/pipeline","forks_url":"https://api.github.com/repos/allenai/pipeline/forks","keys_url":"https://api.github.com/repos/allenai/pipeline/keys{/key_id}","collaborators_url":"https://api.github.com/repos/allenai/pipeline/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/allenai/pipeline/teams","hooks_url":"https://api.github.com/repos/allenai/pipeline/hooks","issue_events_url":"https://api.github.com/repos/allenai/pipeline/issues/events{/number}","events_url":"https://api.github.com/repos/allenai/pipeline/events","assignees_url":"https://api.github.com/repos/allenai/pipeline/assignees{/user}","branches_url":"https://api.github.com/repos/allenai/pipeline/branches{/branch}","tags_url":"https://api.github.com/repos/allenai/pipeline/tags","blobs_url":"https://api.github.com/repos/allenai/pipeline/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/allenai/pipeline/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/allenai/pipeline/git/refs{/sha}","trees_url":"https://api.github.com/repos/allenai/pipeline/git/trees{/sha}","statuses_url":"https://api.github.com/repos/allenai/pipeline/statuses/{sha}","languages_url":"https://api.github.com/repos/allenai/pipeline/languages","stargazers_url":"https://api.github.com/repos/allenai/pipeline/stargazers","contributors_url":"https://api.github.com/repos/allenai/pipeline/contributors","subscribers_url":"https://api.github.com/repos/allenai/pipeline/subscribers","subscription_url":"https://api.github.com/repos/allenai/pipeline/subscription","commits_url":"https://api.github.com/repos/allenai/pipeline/commits{/sha}","git_commits_url":"https://api.github.com/repos/allenai/pipeline/git/commits{/sha}","comments_url":"https://api.github.com/repos/allenai/pipeline/comments{/number}","issue_comment_url":"https://api.github.com/repos/allenai/pipeline/issues/comments{/number}","contents_url":"https://api.github.com/repos/allenai/pipeline/contents/{+path}","compare_url":"https://api.github.com/repos/allenai/pipeline/compare/{base}...{head}","merges_url":"https://api.github.com/repos/allenai/pipeline/merges","archive_url":"https://api.github.com/repos/allenai/pipeline/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/allenai/pipeline/downloads","issues_url":"https://api.github.com/repos/allenai/pipeline/issues{/number}","pulls_url":"https://api.github.com/repos/allenai/pipeline/pulls{/number}","milestones_url":"https://api.github.com/repos/allenai/pipeline/milestones{/number}","notifications_url":"https://api.github.com/repos/allenai/pipeline/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/allenai/pipeline/labels{/name}","releases_url":"https://api.github.com/repos/allenai/pipeline/releases{/id}","created_at":"2015-03-02T19:57:02Z","updated_at":"2015-04-09T03:00:49Z","pushed_at":"2015-06-02T16:00:27Z","git_url":"git://github.com/allenai/pipeline.git","ssh_url":"git@github.com:allenai/pipeline.git","clone_url":"https://github.com/allenai/pipeline.git","svn_url":"https://github.com/allenai/pipeline","homepage":"","size":1304,"stargazers_count":2,"watchers_count":2,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":5,"mirror_url":null,"open_issues_count":4,"forks":5,"open_issues":4,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/31"},"html":{"href":"https://github.com/allenai/pipeline/pull/31"},"issue":{"href":"https://api.github.com/repos/allenai/pipeline/issues/31"},"comments":{"href":"https://api.github.com/repos/allenai/pipeline/issues/31/comments"},"review_comments":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/31/comments"},"review_comment":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/allenai/pipeline/pulls/31/commits"},"statuses":{"href":"https://api.github.com/repos/allenai/pipeline/statuses/e63ea47f47c4e8a5153ad3e018ace229a86884f2"}}}}