{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4107","labels_url":"https://api.github.com/repos/apache/spark/issues/4107/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4107/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4107/events","html_url":"https://github.com/apache/spark/pull/4107","id":54818516,"number":4107,"title":"[SPARK-5325] [SQL] Shrink the Hive shim layer","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-01-19T22:08:52Z","updated_at":"2015-01-20T07:07:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4107","html_url":"https://github.com/apache/spark/pull/4107","diff_url":"https://github.com/apache/spark/pull/4107.diff","patch_url":"https://github.com/apache/spark/pull/4107.patch"},"body":"This PR shrinks the Hive shim layer significantly to reduce maintenance cost.  Now the shim layer only contains decimal related stuff and Hive UDF wrapper (which can be further simplified, but I'd like to do it in a separate PR).  Methods/classes are moved out of the shim layer via two means:\r\n\r\n1. Re-implement some functions in a backwards compatible way, e.g. constant object inspector methods in `WritableConstantObjectInspectors`, and the `FileSinkDescWrapper` class\r\n2. For some simple functions that are not performance critical, re-implement them with reflection tricks.\r\n\r\nMost functions are moved from `HiveShim` to `HiveCompat`. Constant object inspector related methods are moved to `o.a.h.h.s.o.p.WritableConstantObjectInspectors` because we need to call package private constructors.\r\n\r\nSeveral Hive 0.13.1 specific test cases are blacklisted for Hive 0.12.0, so that future Hive 0.12.0 dedicated Jenkins builder won't complain.\r\n\r\nReflection utility function `callWithAlternativies` is introduced to simplify reflection tricks. It tries to invoke all provided reflected constructor calls, method calls, and static method calls one by one until one of them succeeds. For example, `HiveDecimal` objects are created via normal constructor in Hive 0.12.0, while `HiveDecimal.create` must be used in Hive 0.13.1. We can fix this by:\r\n\r\n```scala\r\n  def createDecimal(bd: JBigDecimal) = callWithAlternatives(\r\n    // For Hive 0.13.1\r\n    InvokeStatic[HiveDecimal](classOf[HiveDecimal], \"create\",\r\n      classOf[JBigDecimal] -> bd),\r\n\r\n    // For Hive 0.12.0\r\n    Construct(classOf[HiveDecimal],\r\n      classOf[JBigDecimal] -> bd))\r\n```\r\n\r\nWe should move `ReflectionUtils` from `sql/hive-thriftserver` to `sql/hive` or a more general location and merge the facility above into it. Would like to do it in another PR so that this one doesn't get too large.\r\n\r\n<!-- Reviewable:start -->\r\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4107)\r\n<!-- Reviewable:end -->\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/70612792","html_url":"https://github.com/apache/spark/pull/4107#issuecomment-70612792","issue_url":"https://api.github.com/repos/apache/spark/issues/4107","id":70612792,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T07:07:40Z","updated_at":"2015-01-20T07:07:40Z","body":"  [Test build #25800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25800/consoleFull) for   PR 4107 at commit [`cb2edfb`](https://github.com/apache/spark/commit/cb2edfb4abd70aac289abbb9d713cbe51da821c2).\n * This patch merges cleanly."}}