{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4938","labels_url":"https://api.github.com/repos/apache/spark/issues/4938/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4938/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4938/events","html_url":"https://github.com/apache/spark/pull/4938","id":60210559,"number":4938,"title":"[SPARK-6195] [SQL] Adds in-memory column type for fixed-precision decimals","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-03-07T15:25:25Z","updated_at":"2015-03-14T08:26:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4938","html_url":"https://github.com/apache/spark/pull/4938","diff_url":"https://github.com/apache/spark/pull/4938.diff","patch_url":"https://github.com/apache/spark/pull/4938.patch"},"body":"This PR adds a specialized in-memory column type for fixed-precision decimals.\r\n\r\nFor all other column types, a single integer column type ID is enough to determine which column type to use. However, this doesn't apply to fixed-precision decimal types with different precision and scale parameters. Moreover, according to the previous design, there seems no trivial way to encode precision and scale information into the columnar byte buffer. On the other hand, considering we always know the data type of the column to be built / scanned ahead of time. This PR no longer use column type ID to construct `ColumnBuilder`s and `ColumnAccessor`s, but resorts to the actual column data type. In this way, we can pass precision / scale information along the way.\r\n\r\nThe column type ID is now not used anymore and can be removed in a future PR.\r\n\r\n### Micro benchmark result\r\n\r\nThe following micro benchmark builds a simple table with 2 million decimals (precision = 10, scale = 0), cache it in memory, then count all the rows. Code (simply paste it into Spark shell):\r\n\r\n```scala\r\nimport sc._\r\nimport sqlContext._\r\nimport sqlContext.implicits._\r\nimport org.apache.spark.sql.types._\r\nimport com.google.common.base.Stopwatch\r\n\r\ndef benchmark(n: Int)(f: => Long) {\r\n  val stopwatch = new Stopwatch()\r\n\r\n  def run() = {\r\n    stopwatch.reset()\r\n    stopwatch.start()\r\n    f\r\n    stopwatch.stop()\r\n    stopwatch.elapsedMillis()\r\n  }\r\n\r\n  val records = (0 until n).map(_ => run())\r\n\r\n  (0 until n).foreach(i => println(s\"Round $i: ${records(i)} ms\"))\r\n  println(s\"Average: ${records.sum / n.toDouble} ms\")\r\n}\r\n\r\n// Explicit casting is required because ScalaReflection can't inspect decimal precision\r\nparallelize(1 to 2000000)\r\n  .map(i => Tuple1(Decimal(i, 10, 0)))\r\n  .toDF(\"dec\")\r\n  .select($\"dec\" cast DecimalType(10, 0))\r\n  .registerTempTable(\"dec\")\r\n\r\nsql(\"CACHE TABLE dec\")\r\nval df = table(\"dec\")\r\n\r\n// Warm up\r\ndf.count()\r\ndf.count()\r\n\r\nbenchmark(5) {\r\n  df.count()\r\n}\r\n```\r\n\r\nWith `FIXED_DECIMAL` column type:\r\n\r\n- Round 0: 75 ms\r\n- Round 1: 97 ms\r\n- Round 2: 75 ms\r\n- Round 3: 70 ms\r\n- Round 4: 72 ms\r\n- Average: 77.8 ms\r\n\r\nWithout `FIXED_DECIMAL` column type:\r\n\r\n- Round 0: 1233 ms\r\n- Round 1: 1170 ms\r\n- Round 2: 1171 ms\r\n- Round 3: 1141 ms\r\n- Round 4: 1141 ms\r\n- Average: 1171.2 ms\r\n\r\n<!-- Reviewable:start -->\r\n[<img src=\"https://reviewable.io/review_button.png\" height=40 alt=\"Review on Reviewable\"/>](https://reviewable.io/reviews/apache/spark/4938)\r\n<!-- Reviewable:end -->\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/80137777","html_url":"https://github.com/apache/spark/pull/4938#issuecomment-80137777","issue_url":"https://api.github.com/repos/apache/spark/issues/4938","id":80137777,"user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"created_at":"2015-03-14T08:26:00Z","updated_at":"2015-03-14T08:26:00Z","body":"Added micro benchmark result. Notice that explicit casting is required because `ScalaReflection` can't capture precision information. This doesn't affect use cases where schema are specified explicitly (Hive tables, parquet tables, etc.)."}}