{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4697","labels_url":"https://api.github.com/repos/apache/spark/issues/4697/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4697/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4697/events","html_url":"https://github.com/apache/spark/pull/4697","id":58250732,"number":4697,"title":"[SPARK-5775] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table","user":{"login":"anselmevignon","id":8995101,"avatar_url":"https://avatars.githubusercontent.com/u/8995101?v=3","gravatar_id":"","url":"https://api.github.com/users/anselmevignon","html_url":"https://github.com/anselmevignon","followers_url":"https://api.github.com/users/anselmevignon/followers","following_url":"https://api.github.com/users/anselmevignon/following{/other_user}","gists_url":"https://api.github.com/users/anselmevignon/gists{/gist_id}","starred_url":"https://api.github.com/users/anselmevignon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anselmevignon/subscriptions","organizations_url":"https://api.github.com/users/anselmevignon/orgs","repos_url":"https://api.github.com/users/anselmevignon/repos","events_url":"https://api.github.com/users/anselmevignon/events{/privacy}","received_events_url":"https://api.github.com/users/anselmevignon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-02-19T18:46:54Z","updated_at":"2015-03-23T12:19:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4697","html_url":"https://github.com/apache/spark/pull/4697","diff_url":"https://github.com/apache/spark/pull/4697.diff","patch_url":"https://github.com/apache/spark/pull/4697.patch"},"body":"The Bug solved here was due to a change in PartitionTableScan, when reading a partitioned table. \r\n\r\n- When the Partititon column is requested out of a parquet table, the Table Scan needs to add the column back to the output Rows. \r\n- To update the Row object created by PartitionTableScan, the Row was first casted in SpecificMutableRow, before being updated.\r\n- This casting was unsafe, since there are no guarantee that the newHadoopRDD used internally will instanciate the output Rows as MutableRow. \r\n\r\nParticularly, when reading a Table with complex (e.g. struct or Array) types,  the newHadoopRDD  uses a parquet.io.api.RecordMateralizer, that is produced by the org.apache.spark.sql.parquet.RowReadSupport . This consumer will be created as a org.apache.spark.sql.parquet.CatalystGroupConverter (a) and not a org.apache.spark.sql.parquet.CatalystPrimitiveRowConverter (b), when there are complex types involved (in the org.apache.spark.sql.parquet.CatalystConverter.createRootConverter factory  )      \r\n\r\nThe consumer (a) will output GenericRow, while the consumer (b) produces SpecificMutableRow. \r\n\r\nTherefore any request selecting a partition columns, plus a complex type column, are returned as GenericRows, and fails into an unsafe casting pit (see https://issues.apache.org/jira/browse/SPARK-5775 for an example. ) \r\n\r\nThe fix proposed here originally replaced the unsafe class casting by a case matching on the Row type, updating the Row if it is of a mutable type, and recreating a Row otherwise.\r\n\r\nThis PR now implements the solution updated by @liancheng on aa39460d4bb4c41084d350ccb1c5a56cd61239b7 :\r\n\r\nThe fix checks if every requested requested columns are primitiveType, in a manner symmetrical to the check in org.apache.spark.sql.parquet.CatalystConverter.createRootConverter. \r\n - If all columns are primitive type,  the Row can safely be casted to a MutableRow.\r\n - Otherwise a new GenericRow is created, and the partition column is written this new row structure   \r\n\r\nThis fix is unit-tested in  sql/hive/src/test/scala/org/apache/spark/sql/parquet/parquetSuites.scala"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/84969583","html_url":"https://github.com/apache/spark/pull/4697#issuecomment-84969583","issue_url":"https://api.github.com/repos/apache/spark/issues/4697","id":84969583,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-03-23T12:19:17Z","updated_at":"2015-03-23T12:19:17Z","body":"  [Test build #28994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28994/consoleFull) for   PR 4697 at commit [`6a4c53d`](https://github.com/apache/spark/commit/6a4c53d9491d182cc90c3160c7418b58f3b3062a).\n * This patch **passes all tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}