{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5851","labels_url":"https://api.github.com/repos/apache/spark/issues/5851/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5851/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5851/events","html_url":"https://github.com/apache/spark/pull/5851","id":72540324,"number":5851,"title":"[SPARK-6907][SQL] Isolated client for HiveMetastore","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2015-05-01T22:36:49Z","updated_at":"2015-05-02T22:02:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5851","html_url":"https://github.com/apache/spark/pull/5851","diff_url":"https://github.com/apache/spark/pull/5851.diff","patch_url":"https://github.com/apache/spark/pull/5851.patch"},"body":"This PR adds initial support for loading multiple versions of Hive in a single JVM and provides a common interface for extracting metadata from the `HiveMetastoreClient` for a given version.  This is accomplished by creating an isolated `ClassLoader` that operates according to the following rules:\r\n\r\n - __Shared Classes__: Java, Scala, logging, and Spark classes are delegated to `baseClassLoader`\r\n  allowing the results of calls to the `ClientInterface` to be visible externally.\r\n - __Hive Classes__: new instances are loaded from `execJars`.  These classes are not\r\n  accessible externally due to their custom loading.\r\n - __Barrier Classes__: Classes such as `ClientWrapper` are defined in Spark but must link to a specific version of Hive.  As a result, the bytecode is acquired from the Spark `ClassLoader` but a new copy is created for each instance of `IsolatedClientLoader`.\r\n  This new instance is able to see a specific version of hive without using reflection where ever hive is consistent across versions. Since\r\n  this is a unique instance, it is not visible externally other than as a generic\r\n  `ClientInterface`, unless `isolationOn` is set to `false`.\r\n\r\nIn addition to the unit tests, I have also tested this locally against mysql instances of the Hive Metastore.  I've also successfully ported Spark SQL to run with this client, but due to the size of the changes, that will come in a follow-up PR.\r\n\r\nBy default, Hive jars are currently downloaded from Maven automatically for a given version to ease packaging and testing.  However, there is also support for specifying their location manually for deployments without internet."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/98407957","html_url":"https://github.com/apache/spark/pull/5851#issuecomment-98407957","issue_url":"https://api.github.com/repos/apache/spark/issues/5851","id":98407957,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-05-02T22:02:19Z","updated_at":"2015-05-02T22:02:19Z","body":"Merged build started."}}