{"action":"created","comment":{"url":"https://api.github.com/repos/databricks/spark-redshift/pulls/comments/43216075","id":43216075,"diff_hunk":"@@ -311,29 +311,25 @@ private[redshift] class RedshiftWriter(\n       // for a description of the manifest file format. The URLs in this manifest must be absolute\n       // and complete.\n \n-      // The saved filenames are going to be of the form part-r-XXXXX-UUID.avro. There isn't a way\n-      // to figure out what this UUID is, besides querying S3. In principle, this file format could\n-      // change in a way that breaks this library, but this seems unlikely and we can patch around\n-      // it if that happens.\n+      // The saved filenames depend on the spark-avro version. In spark-avro 1.0.0, the write\n+      // path uses SparkContext.saveAsHadoopFile(), which produces filenames of the form\n+      // part-XXXXX.avro. In spark-avro 2.0.0+, the partition filenames are of the form\n+      // part-r-XXXXX-UUID.avro.\n       val fs = FileSystem.get(URI.create(tempDir), sqlContext.sparkContext.hadoopConfiguration)\n-      val writeJobUUID = fs.listStatus(new Path(tempDir))","path":"src/main/scala/com/databricks/spark/redshift/RedshiftWriter.scala","position":13,"original_position":13,"commit_id":"2005df25265bc900dac88236590d37dedda17b07","original_commit_id":"8428a2bd4155adfba142497f7d10408bb710a131","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"body":"Actually, this shouldn't be a problem: if you have _millions_ of partitions then it almost certainly makes more sense to coalesce them prior to writing to Redshift. I can add this auto-coalescing in a followup.","created_at":"2015-10-28T03:47:44Z","updated_at":"2015-10-28T03:47:44Z","html_url":"https://github.com/databricks/spark-redshift/pull/114#discussion_r43216075","pull_request_url":"https://api.github.com/repos/databricks/spark-redshift/pulls/114","_links":{"self":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/comments/43216075"},"html":{"href":"https://github.com/databricks/spark-redshift/pull/114#discussion_r43216075"},"pull_request":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/114"}}},"pull_request":{"url":"https://api.github.com/repos/databricks/spark-redshift/pulls/114","id":48957779,"html_url":"https://github.com/databricks/spark-redshift/pull/114","diff_url":"https://github.com/databricks/spark-redshift/pull/114.diff","patch_url":"https://github.com/databricks/spark-redshift/pull/114.patch","issue_url":"https://api.github.com/repos/databricks/spark-redshift/issues/114","number":114,"state":"open","locked":false,"title":"Restore compatibility with spark-avro 1.0.0","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"body":"This patch addresses an issue where `spark-redshift` would run into \"Mandatory url is not present in manifest file\" errors when saving data back to Redshift if a pre-2.0.0 version of `spark-avro` was used at runtime. This problem should not arise for most users of `spark-redshift`, since the proper version of `spark-avro` should automatically be pulled in via Maven or Ivy; the goal of this patch is to provide compatibility for users who cannot upgrade `spark-avro` for other compatibility-related reasons.\r\n\r\n**Cause of the write path bug**: In order to fix a crash that could occur when writing tables containing empty partitions, #99 modified the write path to use manifest files that instruct Redshift to load only the non-empty partitions' Avro files. The code that handled processing of filenames when generating the manifest made assumptions about filenames that hold for spark-avro 2.0.0+ but not for spark-avro 1.0.0.\r\n\r\n**Solution**: use a different method to build the correct list of part file names when constructing the manifest.\r\n\r\nFixes #111.","created_at":"2015-10-28T02:28:03Z","updated_at":"2015-10-28T03:47:44Z","closed_at":null,"merged_at":null,"merge_commit_sha":"16ea41696b08011305367bcffa5922e374d0bd72","assignee":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/databricks/spark-redshift/milestones/5","html_url":"https://github.com/databricks/spark-redshift/milestones/0.5.3","labels_url":"https://api.github.com/repos/databricks/spark-redshift/milestones/5/labels","id":1380286,"number":5,"title":"0.5.3","description":"","creator":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":2,"state":"open","created_at":"2015-10-27T20:02:30Z","updated_at":"2015-10-28T02:28:03Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/databricks/spark-redshift/pulls/114/commits","review_comments_url":"https://api.github.com/repos/databricks/spark-redshift/pulls/114/comments","review_comment_url":"https://api.github.com/repos/databricks/spark-redshift/pulls/comments{/number}","comments_url":"https://api.github.com/repos/databricks/spark-redshift/issues/114/comments","statuses_url":"https://api.github.com/repos/databricks/spark-redshift/statuses/2005df25265bc900dac88236590d37dedda17b07","head":{"label":"databricks:compatibility-with-spark-avro-1.0","ref":"compatibility-with-spark-avro-1.0","sha":"2005df25265bc900dac88236590d37dedda17b07","user":{"login":"databricks","id":4998052,"avatar_url":"https://avatars.githubusercontent.com/u/4998052?v=3","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"repo":{"id":26562411,"name":"spark-redshift","full_name":"databricks/spark-redshift","owner":{"login":"databricks","id":4998052,"avatar_url":"https://avatars.githubusercontent.com/u/4998052?v=3","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/databricks/spark-redshift","description":"Spark and Redshift integration","fork":false,"url":"https://api.github.com/repos/databricks/spark-redshift","forks_url":"https://api.github.com/repos/databricks/spark-redshift/forks","keys_url":"https://api.github.com/repos/databricks/spark-redshift/keys{/key_id}","collaborators_url":"https://api.github.com/repos/databricks/spark-redshift/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/databricks/spark-redshift/teams","hooks_url":"https://api.github.com/repos/databricks/spark-redshift/hooks","issue_events_url":"https://api.github.com/repos/databricks/spark-redshift/issues/events{/number}","events_url":"https://api.github.com/repos/databricks/spark-redshift/events","assignees_url":"https://api.github.com/repos/databricks/spark-redshift/assignees{/user}","branches_url":"https://api.github.com/repos/databricks/spark-redshift/branches{/branch}","tags_url":"https://api.github.com/repos/databricks/spark-redshift/tags","blobs_url":"https://api.github.com/repos/databricks/spark-redshift/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/databricks/spark-redshift/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/databricks/spark-redshift/git/refs{/sha}","trees_url":"https://api.github.com/repos/databricks/spark-redshift/git/trees{/sha}","statuses_url":"https://api.github.com/repos/databricks/spark-redshift/statuses/{sha}","languages_url":"https://api.github.com/repos/databricks/spark-redshift/languages","stargazers_url":"https://api.github.com/repos/databricks/spark-redshift/stargazers","contributors_url":"https://api.github.com/repos/databricks/spark-redshift/contributors","subscribers_url":"https://api.github.com/repos/databricks/spark-redshift/subscribers","subscription_url":"https://api.github.com/repos/databricks/spark-redshift/subscription","commits_url":"https://api.github.com/repos/databricks/spark-redshift/commits{/sha}","git_commits_url":"https://api.github.com/repos/databricks/spark-redshift/git/commits{/sha}","comments_url":"https://api.github.com/repos/databricks/spark-redshift/comments{/number}","issue_comment_url":"https://api.github.com/repos/databricks/spark-redshift/issues/comments{/number}","contents_url":"https://api.github.com/repos/databricks/spark-redshift/contents/{+path}","compare_url":"https://api.github.com/repos/databricks/spark-redshift/compare/{base}...{head}","merges_url":"https://api.github.com/repos/databricks/spark-redshift/merges","archive_url":"https://api.github.com/repos/databricks/spark-redshift/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/databricks/spark-redshift/downloads","issues_url":"https://api.github.com/repos/databricks/spark-redshift/issues{/number}","pulls_url":"https://api.github.com/repos/databricks/spark-redshift/pulls{/number}","milestones_url":"https://api.github.com/repos/databricks/spark-redshift/milestones{/number}","notifications_url":"https://api.github.com/repos/databricks/spark-redshift/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/databricks/spark-redshift/labels{/name}","releases_url":"https://api.github.com/repos/databricks/spark-redshift/releases{/id}","created_at":"2014-11-13T00:08:13Z","updated_at":"2015-10-26T20:55:18Z","pushed_at":"2015-10-28T03:46:08Z","git_url":"git://github.com/databricks/spark-redshift.git","ssh_url":"git@github.com:databricks/spark-redshift.git","clone_url":"https://github.com/databricks/spark-redshift.git","svn_url":"https://github.com/databricks/spark-redshift","homepage":null,"size":1945,"stargazers_count":102,"watchers_count":102,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":35,"mirror_url":null,"open_issues_count":12,"forks":35,"open_issues":12,"watchers":102,"default_branch":"master"}},"base":{"label":"databricks:master","ref":"master","sha":"a0953fe4ab432dcbeca91e36ff7e87b27f7aa76b","user":{"login":"databricks","id":4998052,"avatar_url":"https://avatars.githubusercontent.com/u/4998052?v=3","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"repo":{"id":26562411,"name":"spark-redshift","full_name":"databricks/spark-redshift","owner":{"login":"databricks","id":4998052,"avatar_url":"https://avatars.githubusercontent.com/u/4998052?v=3","gravatar_id":"","url":"https://api.github.com/users/databricks","html_url":"https://github.com/databricks","followers_url":"https://api.github.com/users/databricks/followers","following_url":"https://api.github.com/users/databricks/following{/other_user}","gists_url":"https://api.github.com/users/databricks/gists{/gist_id}","starred_url":"https://api.github.com/users/databricks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/databricks/subscriptions","organizations_url":"https://api.github.com/users/databricks/orgs","repos_url":"https://api.github.com/users/databricks/repos","events_url":"https://api.github.com/users/databricks/events{/privacy}","received_events_url":"https://api.github.com/users/databricks/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/databricks/spark-redshift","description":"Spark and Redshift integration","fork":false,"url":"https://api.github.com/repos/databricks/spark-redshift","forks_url":"https://api.github.com/repos/databricks/spark-redshift/forks","keys_url":"https://api.github.com/repos/databricks/spark-redshift/keys{/key_id}","collaborators_url":"https://api.github.com/repos/databricks/spark-redshift/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/databricks/spark-redshift/teams","hooks_url":"https://api.github.com/repos/databricks/spark-redshift/hooks","issue_events_url":"https://api.github.com/repos/databricks/spark-redshift/issues/events{/number}","events_url":"https://api.github.com/repos/databricks/spark-redshift/events","assignees_url":"https://api.github.com/repos/databricks/spark-redshift/assignees{/user}","branches_url":"https://api.github.com/repos/databricks/spark-redshift/branches{/branch}","tags_url":"https://api.github.com/repos/databricks/spark-redshift/tags","blobs_url":"https://api.github.com/repos/databricks/spark-redshift/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/databricks/spark-redshift/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/databricks/spark-redshift/git/refs{/sha}","trees_url":"https://api.github.com/repos/databricks/spark-redshift/git/trees{/sha}","statuses_url":"https://api.github.com/repos/databricks/spark-redshift/statuses/{sha}","languages_url":"https://api.github.com/repos/databricks/spark-redshift/languages","stargazers_url":"https://api.github.com/repos/databricks/spark-redshift/stargazers","contributors_url":"https://api.github.com/repos/databricks/spark-redshift/contributors","subscribers_url":"https://api.github.com/repos/databricks/spark-redshift/subscribers","subscription_url":"https://api.github.com/repos/databricks/spark-redshift/subscription","commits_url":"https://api.github.com/repos/databricks/spark-redshift/commits{/sha}","git_commits_url":"https://api.github.com/repos/databricks/spark-redshift/git/commits{/sha}","comments_url":"https://api.github.com/repos/databricks/spark-redshift/comments{/number}","issue_comment_url":"https://api.github.com/repos/databricks/spark-redshift/issues/comments{/number}","contents_url":"https://api.github.com/repos/databricks/spark-redshift/contents/{+path}","compare_url":"https://api.github.com/repos/databricks/spark-redshift/compare/{base}...{head}","merges_url":"https://api.github.com/repos/databricks/spark-redshift/merges","archive_url":"https://api.github.com/repos/databricks/spark-redshift/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/databricks/spark-redshift/downloads","issues_url":"https://api.github.com/repos/databricks/spark-redshift/issues{/number}","pulls_url":"https://api.github.com/repos/databricks/spark-redshift/pulls{/number}","milestones_url":"https://api.github.com/repos/databricks/spark-redshift/milestones{/number}","notifications_url":"https://api.github.com/repos/databricks/spark-redshift/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/databricks/spark-redshift/labels{/name}","releases_url":"https://api.github.com/repos/databricks/spark-redshift/releases{/id}","created_at":"2014-11-13T00:08:13Z","updated_at":"2015-10-26T20:55:18Z","pushed_at":"2015-10-28T03:46:08Z","git_url":"git://github.com/databricks/spark-redshift.git","ssh_url":"git@github.com:databricks/spark-redshift.git","clone_url":"https://github.com/databricks/spark-redshift.git","svn_url":"https://github.com/databricks/spark-redshift","homepage":null,"size":1945,"stargazers_count":102,"watchers_count":102,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":35,"mirror_url":null,"open_issues_count":12,"forks":35,"open_issues":12,"watchers":102,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/114"},"html":{"href":"https://github.com/databricks/spark-redshift/pull/114"},"issue":{"href":"https://api.github.com/repos/databricks/spark-redshift/issues/114"},"comments":{"href":"https://api.github.com/repos/databricks/spark-redshift/issues/114/comments"},"review_comments":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/114/comments"},"review_comment":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/databricks/spark-redshift/pulls/114/commits"},"statuses":{"href":"https://api.github.com/repos/databricks/spark-redshift/statuses/2005df25265bc900dac88236590d37dedda17b07"}}}}