{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/10429","labels_url":"https://api.github.com/repos/apache/spark/issues/10429/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/10429/comments","events_url":"https://api.github.com/repos/apache/spark/issues/10429/events","html_url":"https://github.com/apache/spark/pull/10429","id":123440687,"number":10429,"title":"[SPARK-12477][SQL] - Tungsten projection fails for null values in array fields","user":{"login":"pierre-borckmans","id":5610359,"avatar_url":"https://avatars.githubusercontent.com/u/5610359?v=3","gravatar_id":"","url":"https://api.github.com/users/pierre-borckmans","html_url":"https://github.com/pierre-borckmans","followers_url":"https://api.github.com/users/pierre-borckmans/followers","following_url":"https://api.github.com/users/pierre-borckmans/following{/other_user}","gists_url":"https://api.github.com/users/pierre-borckmans/gists{/gist_id}","starred_url":"https://api.github.com/users/pierre-borckmans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pierre-borckmans/subscriptions","organizations_url":"https://api.github.com/users/pierre-borckmans/orgs","repos_url":"https://api.github.com/users/pierre-borckmans/repos","events_url":"https://api.github.com/users/pierre-borckmans/events{/privacy}","received_events_url":"https://api.github.com/users/pierre-borckmans/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-12-22T08:48:59Z","updated_at":"2015-12-22T21:40:58Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/10429","html_url":"https://github.com/apache/spark/pull/10429","diff_url":"https://github.com/apache/spark/pull/10429.diff","patch_url":"https://github.com/apache/spark/pull/10429.patch"},"body":"Accessing null elements in an array field fails when tungsten is enabled.\r\nIt works in Spark 1.3.1, and in Spark > 1.5 with Tungsten disabled.\r\n\r\nThis PR solves this by checking if the accessed element in the array field is null, in the generated code.\r\n\r\nExample:\r\n```\r\n// Array of String\r\ncase class AS( as: Seq[String] )\r\nval dfAS = sc.parallelize( Seq( AS ( Seq(\"a\",null,\"b\") ) ) ).toDF\r\ndfAS.registerTempTable(\"T_AS\")\r\nfor (i <- 0 to 2) { println(i + \" = \" + sqlContext.sql(s\"select as[$i] from T_AS\").collect.mkString(\",\"))}\r\n```\r\n\r\nWith Tungsten disabled:\r\n```\r\n0 = [a]\r\n1 = [null]\r\n2 = [b]\r\n```\r\n\r\nWith Tungsten enabled:\r\n```\r\n0 = [a]\r\n15/12/22 09:32:50 ERROR Executor: Exception in task 7.0 in stage 1.0 (TID 15)\r\njava.lang.NullPointerException\r\n\tat org.apache.spark.sql.catalyst.expressions.UnsafeRowWriters$UTF8StringWriter.getSize(UnsafeRowWriters.java:90)\r\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)\r\n\tat org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:90)\r\n\tat org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:88)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:328)\r\n\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\r\n\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/166737236","html_url":"https://github.com/apache/spark/pull/10429#issuecomment-166737236","issue_url":"https://api.github.com/repos/apache/spark/issues/10429","id":166737236,"user":{"login":"pierre-borckmans","id":5610359,"avatar_url":"https://avatars.githubusercontent.com/u/5610359?v=3","gravatar_id":"","url":"https://api.github.com/users/pierre-borckmans","html_url":"https://github.com/pierre-borckmans","followers_url":"https://api.github.com/users/pierre-borckmans/followers","following_url":"https://api.github.com/users/pierre-borckmans/following{/other_user}","gists_url":"https://api.github.com/users/pierre-borckmans/gists{/gist_id}","starred_url":"https://api.github.com/users/pierre-borckmans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pierre-borckmans/subscriptions","organizations_url":"https://api.github.com/users/pierre-borckmans/orgs","repos_url":"https://api.github.com/users/pierre-borckmans/repos","events_url":"https://api.github.com/users/pierre-borckmans/events{/privacy}","received_events_url":"https://api.github.com/users/pierre-borckmans/received_events","type":"User","site_admin":false},"created_at":"2015-12-22T21:40:58Z","updated_at":"2015-12-22T21:40:58Z","body":"@rxin I fixed the test title, and the scala style issues.\r\nI ran `dev/scalastyle` successfully."}}