{"action":"created","comment":{"url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/comments/29114050","id":29114050,"diff_hunk":"@@ -0,0 +1,73 @@\n+package com.databricks.examples.redshift.output\n+\n+import org.apache.hadoop.fs.Path\n+import org.apache.hadoop.io.{NullWritable, Text}\n+import org.apache.hadoop.mapreduce.lib.output.{FileOutputCommitter, FileOutputFormat, TextOutputFormat}\n+import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}\n+import org.apache.spark.rdd.RDD\n+import org.apache.spark.SparkContext\n+import scala.reflect.ClassTag\n+\n+/**\n+ * Similar to the file output committer, but this creates a redshift-style\n+ * manifest of the results. This can be used to enforce consistency on S3.\n+ */\n+class ManifestFileOutputCommitter(outputPath: Path, context: TaskAttemptContext)\n+    extends FileOutputCommitter(outputPath, context) {\n+  override def commitJob(jobContext: JobContext): Unit = {\n+\n+    val jobAttemptPath = getJobAttemptPath(context)\n+    val fs = jobAttemptPath.getFileSystem(context.getConfiguration())\n+    val allOutputFiles = fs.listStatus(jobAttemptPath)","path":"src/main/scala/com/databricks/examples/redshift/output/ManifestFileOutputCommitter.scala","position":21,"original_position":21,"commit_id":"e01dce10d425951fca481e96c0e4e85d31c52090","original_commit_id":"e01dce10d425951fca481e96c0e4e85d31c52090","user":{"login":"derrley","id":1234652,"avatar_url":"https://avatars.githubusercontent.com/u/1234652?v=3","gravatar_id":"","url":"https://api.github.com/users/derrley","html_url":"https://github.com/derrley","followers_url":"https://api.github.com/users/derrley/followers","following_url":"https://api.github.com/users/derrley/following{/other_user}","gists_url":"https://api.github.com/users/derrley/gists{/gist_id}","starred_url":"https://api.github.com/users/derrley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derrley/subscriptions","organizations_url":"https://api.github.com/users/derrley/orgs","repos_url":"https://api.github.com/users/derrley/repos","events_url":"https://api.github.com/users/derrley/events{/privacy}","received_events_url":"https://api.github.com/users/derrley/received_events","type":"User","site_admin":false},"body":"Fuck.\n\nSlate / Firehose output format here we come.\n\nMight be cool to just talk to firehose instead of s3.\nOn Apr 26, 2015 4:35 PM, \"Nick Buroojy\" <notifications@github.com> wrote:\n\n> In\n> src/main/scala/com/databricks/examples/redshift/output/ManifestFileOutputCommitter.scala\n> <https://github.com/civitaslearning/spark-redshift/pull/4#discussion_r29114022>\n> :\n>\n> > +import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}\n> > +import org.apache.spark.rdd.RDD\n> > +import org.apache.spark.SparkContext\n> > +import scala.reflect.ClassTag\n> > +\n> > +/**\n> > + * Similar to the file output committer, but this creates a redshift-style\n> > + * manifest of the results. This can be used to enforce consistency on S3.\n> > + */\n> > +class ManifestFileOutputCommitter(outputPath: Path, context: TaskAttemptContext)\n> > +    extends FileOutputCommitter(outputPath, context) {\n> > +  override def commitJob(jobContext: JobContext): Unit = {\n> > +\n> > +    val jobAttemptPath = getJobAttemptPath(context)\n> > +    val fs = jobAttemptPath.getFileSystem(context.getConfiguration())\n> > +    val allOutputFiles = fs.listStatus(jobAttemptPath)\n>\n> My point is that we can't trust the Hadoop file output format to\n> consistently write to s3. The file output committer lists files in the\n> _temporary directory on s3 before it copies them to the final output. If\n> that list is eventually consistent, the output will be permanently\n> inconsistent (not just eventually consistent).\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/civitaslearning/spark-redshift/pull/4/files#r29114022>\n> .\n>\n","created_at":"2015-04-26T21:37:16Z","updated_at":"2015-04-26T21:37:16Z","html_url":"https://github.com/civitaslearning/spark-redshift/pull/4#discussion_r29114050","pull_request_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4","_links":{"self":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/comments/29114050"},"html":{"href":"https://github.com/civitaslearning/spark-redshift/pull/4#discussion_r29114050"},"pull_request":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4"}}},"pull_request":{"url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4","id":33707255,"html_url":"https://github.com/civitaslearning/spark-redshift/pull/4","diff_url":"https://github.com/civitaslearning/spark-redshift/pull/4.diff","patch_url":"https://github.com/civitaslearning/spark-redshift/pull/4.patch","issue_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/4","number":4,"state":"open","locked":false,"title":"Added input and output formats for redshift manifested data","user":{"login":"nburoojy","id":6707096,"avatar_url":"https://avatars.githubusercontent.com/u/6707096?v=3","gravatar_id":"","url":"https://api.github.com/users/nburoojy","html_url":"https://github.com/nburoojy","followers_url":"https://api.github.com/users/nburoojy/followers","following_url":"https://api.github.com/users/nburoojy/following{/other_user}","gists_url":"https://api.github.com/users/nburoojy/gists{/gist_id}","starred_url":"https://api.github.com/users/nburoojy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nburoojy/subscriptions","organizations_url":"https://api.github.com/users/nburoojy/orgs","repos_url":"https://api.github.com/users/nburoojy/repos","events_url":"https://api.github.com/users/nburoojy/events{/privacy}","received_events_url":"https://api.github.com/users/nburoojy/received_events","type":"User","site_admin":false},"body":"@derrley @dkolbly \r\n\r\nRedshift ensures load and unload consistency using manifest files.\r\nThis is especially useful on eventually-consistent filesystems like S3.\r\n\r\nThis change adds an input format which loads a manifest file, then rewrites\r\nthe job input to be the contents of the manifest. Since it explicitly defines\r\nthe input paths, a missing file will throw an exception and the job will fail.\r\nThis is intended to ensure consistency in Redshift -> S3 -> Hadoop workflows.\r\n\r\nIt also adds an output format which is very similar to the text output, but\r\nwrites a redshift-style manifest in addition to the _SUCCESS marker. This is\r\nintended to ensure consistency in Hadoop -> S3 -> Redshift workflows. ","created_at":"2015-04-20T21:18:27Z","updated_at":"2015-04-26T21:37:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ec61b21330aae6e845dda74f5460a650ff332486","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4/commits","review_comments_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4/comments","review_comment_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/comments{/number}","comments_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/4/comments","statuses_url":"https://api.github.com/repos/civitaslearning/spark-redshift/statuses/e01dce10d425951fca481e96c0e4e85d31c52090","head":{"label":"civitaslearning:nick/manifestin","ref":"nick/manifestin","sha":"e01dce10d425951fca481e96c0e4e85d31c52090","user":{"login":"civitaslearning","id":1000428,"avatar_url":"https://avatars.githubusercontent.com/u/1000428?v=3","gravatar_id":"","url":"https://api.github.com/users/civitaslearning","html_url":"https://github.com/civitaslearning","followers_url":"https://api.github.com/users/civitaslearning/followers","following_url":"https://api.github.com/users/civitaslearning/following{/other_user}","gists_url":"https://api.github.com/users/civitaslearning/gists{/gist_id}","starred_url":"https://api.github.com/users/civitaslearning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/civitaslearning/subscriptions","organizations_url":"https://api.github.com/users/civitaslearning/orgs","repos_url":"https://api.github.com/users/civitaslearning/repos","events_url":"https://api.github.com/users/civitaslearning/events{/privacy}","received_events_url":"https://api.github.com/users/civitaslearning/received_events","type":"Organization","site_admin":false},"repo":{"id":32090851,"name":"spark-redshift","full_name":"civitaslearning/spark-redshift","owner":{"login":"civitaslearning","id":1000428,"avatar_url":"https://avatars.githubusercontent.com/u/1000428?v=3","gravatar_id":"","url":"https://api.github.com/users/civitaslearning","html_url":"https://github.com/civitaslearning","followers_url":"https://api.github.com/users/civitaslearning/followers","following_url":"https://api.github.com/users/civitaslearning/following{/other_user}","gists_url":"https://api.github.com/users/civitaslearning/gists{/gist_id}","starred_url":"https://api.github.com/users/civitaslearning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/civitaslearning/subscriptions","organizations_url":"https://api.github.com/users/civitaslearning/orgs","repos_url":"https://api.github.com/users/civitaslearning/repos","events_url":"https://api.github.com/users/civitaslearning/events{/privacy}","received_events_url":"https://api.github.com/users/civitaslearning/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/civitaslearning/spark-redshift","description":"Spark and Redshift integration","fork":true,"url":"https://api.github.com/repos/civitaslearning/spark-redshift","forks_url":"https://api.github.com/repos/civitaslearning/spark-redshift/forks","keys_url":"https://api.github.com/repos/civitaslearning/spark-redshift/keys{/key_id}","collaborators_url":"https://api.github.com/repos/civitaslearning/spark-redshift/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/civitaslearning/spark-redshift/teams","hooks_url":"https://api.github.com/repos/civitaslearning/spark-redshift/hooks","issue_events_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/events{/number}","events_url":"https://api.github.com/repos/civitaslearning/spark-redshift/events","assignees_url":"https://api.github.com/repos/civitaslearning/spark-redshift/assignees{/user}","branches_url":"https://api.github.com/repos/civitaslearning/spark-redshift/branches{/branch}","tags_url":"https://api.github.com/repos/civitaslearning/spark-redshift/tags","blobs_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/refs{/sha}","trees_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/trees{/sha}","statuses_url":"https://api.github.com/repos/civitaslearning/spark-redshift/statuses/{sha}","languages_url":"https://api.github.com/repos/civitaslearning/spark-redshift/languages","stargazers_url":"https://api.github.com/repos/civitaslearning/spark-redshift/stargazers","contributors_url":"https://api.github.com/repos/civitaslearning/spark-redshift/contributors","subscribers_url":"https://api.github.com/repos/civitaslearning/spark-redshift/subscribers","subscription_url":"https://api.github.com/repos/civitaslearning/spark-redshift/subscription","commits_url":"https://api.github.com/repos/civitaslearning/spark-redshift/commits{/sha}","git_commits_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/commits{/sha}","comments_url":"https://api.github.com/repos/civitaslearning/spark-redshift/comments{/number}","issue_comment_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/comments{/number}","contents_url":"https://api.github.com/repos/civitaslearning/spark-redshift/contents/{+path}","compare_url":"https://api.github.com/repos/civitaslearning/spark-redshift/compare/{base}...{head}","merges_url":"https://api.github.com/repos/civitaslearning/spark-redshift/merges","archive_url":"https://api.github.com/repos/civitaslearning/spark-redshift/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/civitaslearning/spark-redshift/downloads","issues_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues{/number}","pulls_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls{/number}","milestones_url":"https://api.github.com/repos/civitaslearning/spark-redshift/milestones{/number}","notifications_url":"https://api.github.com/repos/civitaslearning/spark-redshift/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/civitaslearning/spark-redshift/labels{/name}","releases_url":"https://api.github.com/repos/civitaslearning/spark-redshift/releases{/id}","created_at":"2015-03-12T17:23:15Z","updated_at":"2015-03-16T18:02:51Z","pushed_at":"2015-04-20T21:18:27Z","git_url":"git://github.com/civitaslearning/spark-redshift.git","ssh_url":"git@github.com:civitaslearning/spark-redshift.git","clone_url":"https://github.com/civitaslearning/spark-redshift.git","svn_url":"https://github.com/civitaslearning/spark-redshift","homepage":null,"size":277,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"base":{"label":"civitaslearning:master","ref":"master","sha":"5bc4d5e5367926337ee0229b3bc033920caaeac2","user":{"login":"civitaslearning","id":1000428,"avatar_url":"https://avatars.githubusercontent.com/u/1000428?v=3","gravatar_id":"","url":"https://api.github.com/users/civitaslearning","html_url":"https://github.com/civitaslearning","followers_url":"https://api.github.com/users/civitaslearning/followers","following_url":"https://api.github.com/users/civitaslearning/following{/other_user}","gists_url":"https://api.github.com/users/civitaslearning/gists{/gist_id}","starred_url":"https://api.github.com/users/civitaslearning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/civitaslearning/subscriptions","organizations_url":"https://api.github.com/users/civitaslearning/orgs","repos_url":"https://api.github.com/users/civitaslearning/repos","events_url":"https://api.github.com/users/civitaslearning/events{/privacy}","received_events_url":"https://api.github.com/users/civitaslearning/received_events","type":"Organization","site_admin":false},"repo":{"id":32090851,"name":"spark-redshift","full_name":"civitaslearning/spark-redshift","owner":{"login":"civitaslearning","id":1000428,"avatar_url":"https://avatars.githubusercontent.com/u/1000428?v=3","gravatar_id":"","url":"https://api.github.com/users/civitaslearning","html_url":"https://github.com/civitaslearning","followers_url":"https://api.github.com/users/civitaslearning/followers","following_url":"https://api.github.com/users/civitaslearning/following{/other_user}","gists_url":"https://api.github.com/users/civitaslearning/gists{/gist_id}","starred_url":"https://api.github.com/users/civitaslearning/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/civitaslearning/subscriptions","organizations_url":"https://api.github.com/users/civitaslearning/orgs","repos_url":"https://api.github.com/users/civitaslearning/repos","events_url":"https://api.github.com/users/civitaslearning/events{/privacy}","received_events_url":"https://api.github.com/users/civitaslearning/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/civitaslearning/spark-redshift","description":"Spark and Redshift integration","fork":true,"url":"https://api.github.com/repos/civitaslearning/spark-redshift","forks_url":"https://api.github.com/repos/civitaslearning/spark-redshift/forks","keys_url":"https://api.github.com/repos/civitaslearning/spark-redshift/keys{/key_id}","collaborators_url":"https://api.github.com/repos/civitaslearning/spark-redshift/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/civitaslearning/spark-redshift/teams","hooks_url":"https://api.github.com/repos/civitaslearning/spark-redshift/hooks","issue_events_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/events{/number}","events_url":"https://api.github.com/repos/civitaslearning/spark-redshift/events","assignees_url":"https://api.github.com/repos/civitaslearning/spark-redshift/assignees{/user}","branches_url":"https://api.github.com/repos/civitaslearning/spark-redshift/branches{/branch}","tags_url":"https://api.github.com/repos/civitaslearning/spark-redshift/tags","blobs_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/refs{/sha}","trees_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/trees{/sha}","statuses_url":"https://api.github.com/repos/civitaslearning/spark-redshift/statuses/{sha}","languages_url":"https://api.github.com/repos/civitaslearning/spark-redshift/languages","stargazers_url":"https://api.github.com/repos/civitaslearning/spark-redshift/stargazers","contributors_url":"https://api.github.com/repos/civitaslearning/spark-redshift/contributors","subscribers_url":"https://api.github.com/repos/civitaslearning/spark-redshift/subscribers","subscription_url":"https://api.github.com/repos/civitaslearning/spark-redshift/subscription","commits_url":"https://api.github.com/repos/civitaslearning/spark-redshift/commits{/sha}","git_commits_url":"https://api.github.com/repos/civitaslearning/spark-redshift/git/commits{/sha}","comments_url":"https://api.github.com/repos/civitaslearning/spark-redshift/comments{/number}","issue_comment_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/comments{/number}","contents_url":"https://api.github.com/repos/civitaslearning/spark-redshift/contents/{+path}","compare_url":"https://api.github.com/repos/civitaslearning/spark-redshift/compare/{base}...{head}","merges_url":"https://api.github.com/repos/civitaslearning/spark-redshift/merges","archive_url":"https://api.github.com/repos/civitaslearning/spark-redshift/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/civitaslearning/spark-redshift/downloads","issues_url":"https://api.github.com/repos/civitaslearning/spark-redshift/issues{/number}","pulls_url":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls{/number}","milestones_url":"https://api.github.com/repos/civitaslearning/spark-redshift/milestones{/number}","notifications_url":"https://api.github.com/repos/civitaslearning/spark-redshift/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/civitaslearning/spark-redshift/labels{/name}","releases_url":"https://api.github.com/repos/civitaslearning/spark-redshift/releases{/id}","created_at":"2015-03-12T17:23:15Z","updated_at":"2015-03-16T18:02:51Z","pushed_at":"2015-04-20T21:18:27Z","git_url":"git://github.com/civitaslearning/spark-redshift.git","ssh_url":"git@github.com:civitaslearning/spark-redshift.git","clone_url":"https://github.com/civitaslearning/spark-redshift.git","svn_url":"https://github.com/civitaslearning/spark-redshift","homepage":null,"size":277,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4"},"html":{"href":"https://github.com/civitaslearning/spark-redshift/pull/4"},"issue":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/4"},"comments":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/issues/4/comments"},"review_comments":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4/comments"},"review_comment":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/pulls/4/commits"},"statuses":{"href":"https://api.github.com/repos/civitaslearning/spark-redshift/statuses/e01dce10d425951fca481e96c0e4e85d31c52090"}}}}