{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6796","labels_url":"https://api.github.com/repos/apache/spark/issues/6796/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6796/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6796/events","html_url":"https://github.com/apache/spark/pull/6796","id":88004853,"number":6796,"title":"[SPARK-4176][WIP] Support decimal types with precision > 18 in parquet","user":{"login":"rtreffer","id":61306,"avatar_url":"https://avatars.githubusercontent.com/u/61306?v=3","gravatar_id":"","url":"https://api.github.com/users/rtreffer","html_url":"https://github.com/rtreffer","followers_url":"https://api.github.com/users/rtreffer/followers","following_url":"https://api.github.com/users/rtreffer/following{/other_user}","gists_url":"https://api.github.com/users/rtreffer/gists{/gist_id}","starred_url":"https://api.github.com/users/rtreffer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rtreffer/subscriptions","organizations_url":"https://api.github.com/users/rtreffer/orgs","repos_url":"https://api.github.com/users/rtreffer/repos","events_url":"https://api.github.com/users/rtreffer/events{/privacy}","received_events_url":"https://api.github.com/users/rtreffer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-06-13T13:26:59Z","updated_at":"2015-06-18T20:04:54Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6796","html_url":"https://github.com/apache/spark/pull/6796","diff_url":"https://github.com/apache/spark/pull/6796.diff","patch_url":"https://github.com/apache/spark/pull/6796.patch"},"body":"This is my current WIP on SPARK-4176. It should be compatible with other implementations of parquet.\r\n\r\nhttps://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md#decimal\r\n```\r\nFor byte arrays, binary and fixed, the unscaled number must be encoded as\r\ntwo's complement using big-endian byte order\r\n(the most significant byte is the zeroth element)\r\n```\r\nThis is the default encoding on bigint. It should thus be compatible with other implementations, although it would be great if s.o. could test this.\r\n\r\nI've tested this locally with powers of 2 up to 2^200 in the spark shell, without errors but\r\n\r\nCode I've used for (local) testing (on spark shell):\r\n```scala\r\nimport org.apache.spark.sql.types._\r\nimport org.apache.spark.sql.Row\r\nimport java.io.File\r\n\r\ndef decimalList(n : Int) = {\r\n  require(n > 0)\r\n  val one = Decimal(1)\r\n  val two = Decimal(2)\r\n  one.changePrecision(n,0)\r\n  two.changePrecision(n,0)\r\n  val list = scala.collection.mutable.ArrayBuffer(one)\r\n  while (list.length < n) {\r\n    val v = list(list.length - 1)\r\n    list += Decimal(v.toJavaBigDecimal.multiply(two.toJavaBigDecimal),n,0)\r\n  }\r\n  list.toList\r\n}\r\n\r\ndef decimalRows(l : List[Decimal]) = l.zipWithIndex.map(e => Row(e._2,e._1))\r\n\r\ndef decimalRowRdd(l : List[Decimal]) = sc.parallelize(decimalRows(l))\r\n\r\ndef df(n : Int) = {\r\n  val data = decimalList(n)\r\n  val len = data.lastOption.get.toString.length\r\n  val schema = StructType(Array(\r\n    StructField(\"id\", IntegerType, true),\r\n    StructField(\"value\", DecimalType(len,0), true)))\r\n  sqlContext.createDataFrame(decimalRowRdd(data), schema)\r\n}\r\n\r\ndef delete(filename : String) : Unit = {\r\n  val f = new File(filename)\r\n  if (!f.exists) return\r\n  if (f.isDirectory) f.listFiles.foreach(_.delete)\r\n  f.delete\r\n}\r\n\r\ndef test(n : Int) = {\r\n  val src = df(n)\r\n  delete(\"/tmp/typetest\")\r\n  src.save(\"/tmp/typetest\")\r\n  val copy = sqlContext.load(\"/tmp/typetest\")\r\n  src.collect().sortBy(_.getInt(0)).zip(copy.collect().sortBy(_.getInt(0))).foreach(e => {\r\n    if (e._1 != e._2) println(s\"${e._1} != ${e._2}\")\r\n  })\r\n  delete(\"/tmp/typetest\")\r\n  println(s\"Tested 1..2^${n - 1}\")\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/113274539","html_url":"https://github.com/apache/spark/pull/6796#issuecomment-113274539","issue_url":"https://api.github.com/repos/apache/spark/issues/6796","id":113274539,"user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"created_at":"2015-06-18T20:04:54Z","updated_at":"2015-06-18T20:04:54Z","body":"Ah cool, it might even be a good idea to check in small files that are created with other systems for these kinds of tests."}}