{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9019","labels_url":"https://api.github.com/repos/apache/spark/issues/9019/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9019/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9019/events","html_url":"https://github.com/apache/spark/pull/9019","id":110357851,"number":9019,"title":"[SPARK-10993] [SQL] Inital code generated encoder for product types","user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-08T02:11:00Z","updated_at":"2015-10-08T21:27:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9019","html_url":"https://github.com/apache/spark/pull/9019","diff_url":"https://github.com/apache/spark/pull/9019.diff","patch_url":"https://github.com/apache/spark/pull/9019.patch"},"body":"This PR is a first cut at code generating an encoder that takes a Scala `Product` type and converts it directly into the tungsten binary format.  This is done through the addition of a new set of expression that can be used to invoke methods on raw JVM objects, extracting fields and converting the result into the required format.  These can then be used directly in an `UnsafeProjection` allowing us to leverage the existing encoding logic.\r\n\r\nAccording to some simple benchmarks, this can significantly speed up conversion (~4x).  However, replacing CatalystConverters is deferred to a later PR to keep this PR at a reasonable size.\r\n\r\n```scala\r\ncase class SomeInts(a: Int, b: Int, c: Int, d: Int, e: Int)\r\n\r\nval data = SomeInts(1, 2, 3, 4, 5)\r\nval encoder = ProductEncoder[SomeInts]\r\nval converter = CatalystTypeConverters.createToCatalystConverter(ScalaReflection.schemaFor[SomeInts].dataType)\r\n\r\n\r\n(1 to 5).foreach {iter =>\r\n  benchmark(s\"converter $iter\") {\r\n    var i = 100000000\r\n    while (i > 0) {\r\n      val res = converter(data).asInstanceOf[InternalRow]\r\n      assert(res.getInt(0) == 1)\r\n      assert(res.getInt(1) == 2)\r\n      i -= 1\r\n    }\r\n  }\r\n\r\n  benchmark(s\"encoder $iter\") {\r\n    var i = 100000000\r\n    while (i > 0) {\r\n      val res = encoder.toRow(data)\r\n      assert(res.getInt(0) == 1)\r\n      assert(res.getInt(1) == 2)\r\n      i -= 1\r\n    }\r\n  }\r\n}\r\n```\r\n\r\nResults:\r\n```\r\n[info] converter 1: 7170ms\r\n[info] encoder 1: 1888ms\r\n[info] converter 2: 6763ms\r\n[info] encoder 2: 1824ms\r\n[info] converter 3: 6912ms\r\n[info] encoder 3: 1802ms\r\n[info] converter 4: 7131ms\r\n[info] encoder 4: 1798ms\r\n[info] converter 5: 7350ms\r\n[info] encoder 5: 1912ms\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/146690958","html_url":"https://github.com/apache/spark/pull/9019#issuecomment-146690958","issue_url":"https://api.github.com/repos/apache/spark/issues/9019","id":146690958,"user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"created_at":"2015-10-08T21:27:42Z","updated_at":"2015-10-08T21:27:42Z","body":"Since this is the infrastructural work of encoder/decoder, let's merge this and use a follow-up pr to address the comments. So, other people can start to play with it."}}