{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10400","labels_url":"https://api.github.com/repos/apache/spark/issues/10400/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10400/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10400/events","html_url":"https://github.com/apache/spark/pull/10400","id":123147833,"number":10400,"title":"[SPARK-12444][SQL] A lightweight Scala DSL for schema construction","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-12-20T09:23:29Z","updated_at":"2015-12-20T15:55:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10400","html_url":"https://github.com/apache/spark/pull/10400","diff_url":"https://github.com/apache/spark/pull/10400.diff","patch_url":"https://github.com/apache/spark/pull/10400.patch"},"body":"This PR introduces a lightweight Scala DSL for constructing complex Spark SQL schema.\r\n\r\nTwo DSL methods `!` and `?` are added to `DataType` to help indicating nullability of struct field, array element type, and map value type.\r\n\r\n- `!` means non-nullable (or required), while\r\n- `?` means nullable (or optional)\r\n\r\n  An implicit conversion from any `DataType` `t` to `t -> true` is added, so that the `?` call can be optional in most cases.  This behavior is consistent to existing constructors of complex types, where `nullable`, `containsNull`, and `valueContainsNull` are all `true` by default.\r\n\r\nWith the help of these two methods, and three more constructors, we can now construct schema like this:\r\n\r\n```scala\r\nStructType(\r\n  \"f0\" -> IntegerType.!,\r\n  \"f1\" -> ArrayType(IntegerType).!,\r\n  \"f2\" -> MapType(\r\n    IntegerType,\r\n    StructType(\r\n      \"f20\" -> DoubleType.!,\r\n      \"f21\" -> StringType\r\n    ).!\r\n  )\r\n)\r\n```\r\n\r\nwhich is conciser and arguably more readable than equivalent existing approaches:\r\n\r\n```scala\r\nStructType(Seq(\r\n  StructField(\"f0\", IntegerType, nullable = false),\r\n  StructField(\"f1\", ArrayType(IntegerType, containsNull = true), nullable = false),\r\n  StructField(\"f2\", MapType(\r\n    IntegerType,\r\n    StructType(Seq(\r\n      StructField(\"f20\", DoubleType, nullable = false),\r\n      StructField(\"f21\", StringType, nullable = true)\r\n    )),\r\n    valueContainsNull = false\r\n  ), nullable = true)\r\n))\r\n\r\nnew StructType()\r\n  .add(\"f0\", IntegerType, nullable = false)\r\n  .add(\"f1\", ArrayType(IntegerType, containsNull = true), nullable = false)\r\n  .add(\"f2\", MapType(\r\n    IntegerType,\r\n    new StructType()\r\n      .add(\"f20\", DoubleType, nullable = false)\r\n      .add(\"f21\", StringType, nullable = true),\r\n    valueContainsNull = false\r\n  ), nullable = true)\r\n```"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/166131213","html_url":"https://github.com/apache/spark/pull/10400#issuecomment-166131213","issue_url":"https://api.github.com/repos/apache/spark/issues/10400","id":166131213,"user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"created_at":"2015-12-20T15:55:01Z","updated_at":"2015-12-20T15:55:01Z","body":"@pierre-borckmans Decided to add the implicit conversion at last. API consistency seems to be more important :) The implicit conversion is added in `object DataType` so that users don't need to pay the `import` tax."}}