{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/29896733","id":29896733,"diff_hunk":"@@ -51,6 +52,12 @@ private[hive] object SparkSQLEnv extends Logging {\n       sparkContext.addSparkListener(new StatsReportListener())\n       hiveContext = new HiveContext(sparkContext)\n \n+      hiveContext.metadataHive.setOut(new PrintStream(System.out, true, \"UTF-8\"))","path":"sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala","position":21,"original_position":21,"commit_id":"5f3945eb3e0f8b12ba58e8d2f5715525b182fd53","original_commit_id":"5f3945eb3e0f8b12ba58e8d2f5715525b182fd53","user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"body":"See https://github.com/apache/hive/blob/release-0.13.1/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L650-653","created_at":"2015-05-07T21:16:35Z","updated_at":"2015-05-07T21:16:35Z","html_url":"https://github.com/apache/spark/pull/5876#discussion_r29896733","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/5876","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/29896733"},"html":{"href":"https://github.com/apache/spark/pull/5876#discussion_r29896733"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/5876"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5876","id":34610455,"html_url":"https://github.com/apache/spark/pull/5876","diff_url":"https://github.com/apache/spark/pull/5876.diff","patch_url":"https://github.com/apache/spark/pull/5876.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/5876","number":5876,"state":"open","locked":false,"title":"[SPARK-6908] [SQL] Use isolated Hive client","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"body":"This PR switches Spark SQL's Hive support to use the isolated hive client interface introduced by #5851, instead of directly interacting with the client.  By using this isolated client we can now allow users to dynamically configure the version of Hive that they are connecting to by setting `spark.sql.hive.metastore.version` without the need recompile.  This also greatly reduces the surface area for our interaction with the hive libraries, hopefully making it easier to support other versions in the future.\r\n\r\nJars for the desired hive version can be configured using `spark.sql.hive.metastore.jars`, which accepts the following options:\r\n - a colon-separated list of jar files or directories for hive and hadoop.\r\n - `builtin` - attempt to discover the jars that were used to load Spark SQL and use those. This\r\n            option is only valid when using the execution version of Hive.\r\n - `maven` - download the correct version of hive on demand from maven.\r\n\r\nBy default, `builtin` is used for Hive 13.\r\n\r\nThis PR also removes the test step for building against Hive 12, as this will no longer be required to talk to Hive 12 metastores.  However, the full removal of the Shim is deferred until a later PR.\r\n\r\nRemaining TODOs:\r\n - Remove the Hive Shims and inline code for Hive 13.\r\n - Several HiveCompatibility tests are not yet passing. \r\n  - `nullformatCTAS` - As detailed below, we now are handling CTAS parsing ourselves instead of hacking into the Hive semantic analyzer.  However, we currently only handle the common cases and not things like CTAS where the null format is specified.\r\n  - `combine1` now leaks state about compression somehow, breaking all subsequent tests.  As such we currently add it to the blacklist\r\n  - `part_inherit_tbl_props` and `part_inherit_tbl_props_with_star` do not work anymore.  We are correctly propagating the information \r\n  - \"load_dyn_part14.*\" - These tests pass when run on their own, but fail when run with all other tests.  It seems our `RESET` mechanism may not be as robust as it used to be?\r\n\r\nOther required changes:\r\n -  `CreateTableAsSelect` no longer carries parts of the HiveQL AST with it through the query execution pipeline.  Instead, we parse CTAS during the HiveQL conversion and construct a `HiveTable`.  The full parsing here is not yet complete as detailed above in the remaining TODOs.  Since the operator is Hive specific, it is moved to the hive package.\r\n - `Command` is simplified to be a trait that simply acts as a marker for a LogicalPlan that should be eagerly evaluated.","created_at":"2015-05-04T00:11:34Z","updated_at":"2015-05-07T21:16:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"4c08bee08addd86f5e88c94ad0b682dadbcd65bc","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/5876/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/5876/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/5876/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/5f3945eb3e0f8b12ba58e8d2f5715525b182fd53","head":{"label":"marmbrus:useIsolatedClient","ref":"useIsolatedClient","sha":"5f3945eb3e0f8b12ba58e8d2f5715525b182fd53","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"repo":{"id":17230869,"name":"spark","full_name":"marmbrus/spark","owner":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/marmbrus/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/marmbrus/spark","forks_url":"https://api.github.com/repos/marmbrus/spark/forks","keys_url":"https://api.github.com/repos/marmbrus/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/marmbrus/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/marmbrus/spark/teams","hooks_url":"https://api.github.com/repos/marmbrus/spark/hooks","issue_events_url":"https://api.github.com/repos/marmbrus/spark/issues/events{/number}","events_url":"https://api.github.com/repos/marmbrus/spark/events","assignees_url":"https://api.github.com/repos/marmbrus/spark/assignees{/user}","branches_url":"https://api.github.com/repos/marmbrus/spark/branches{/branch}","tags_url":"https://api.github.com/repos/marmbrus/spark/tags","blobs_url":"https://api.github.com/repos/marmbrus/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/marmbrus/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/marmbrus/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/marmbrus/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/marmbrus/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/marmbrus/spark/languages","stargazers_url":"https://api.github.com/repos/marmbrus/spark/stargazers","contributors_url":"https://api.github.com/repos/marmbrus/spark/contributors","subscribers_url":"https://api.github.com/repos/marmbrus/spark/subscribers","subscription_url":"https://api.github.com/repos/marmbrus/spark/subscription","commits_url":"https://api.github.com/repos/marmbrus/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/marmbrus/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/marmbrus/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/marmbrus/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/marmbrus/spark/contents/{+path}","compare_url":"https://api.github.com/repos/marmbrus/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/marmbrus/spark/merges","archive_url":"https://api.github.com/repos/marmbrus/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/marmbrus/spark/downloads","issues_url":"https://api.github.com/repos/marmbrus/spark/issues{/number}","pulls_url":"https://api.github.com/repos/marmbrus/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/marmbrus/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/marmbrus/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/marmbrus/spark/labels{/name}","releases_url":"https://api.github.com/repos/marmbrus/spark/releases{/id}","created_at":"2014-02-27T00:22:54Z","updated_at":"2014-11-27T06:21:59Z","pushed_at":"2015-05-07T20:28:48Z","git_url":"git://github.com/marmbrus/spark.git","ssh_url":"git@github.com:marmbrus/spark.git","clone_url":"https://github.com/marmbrus/spark.git","svn_url":"https://github.com/marmbrus/spark","homepage":null,"size":145715,"stargazers_count":2,"watchers_count":2,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":1,"forks":2,"open_issues":1,"watchers":2,"default_branch":"catalyst"}},"base":{"label":"apache:master","ref":"master","sha":"88717ee4e7542ac8d5d2e5756c912dd390b37e88","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-05-07T21:02:26Z","pushed_at":"2015-05-07T21:13:14Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1606476,"stargazers_count":3887,"watchers_count":3887,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":3406,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":315,"forks":3406,"open_issues":315,"watchers":3887,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/5876"},"html":{"href":"https://github.com/apache/spark/pull/5876"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/5876"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/5876/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/5876/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/5876/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/5f3945eb3e0f8b12ba58e8d2f5715525b182fd53"}}}}