{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7231","labels_url":"https://api.github.com/repos/apache/spark/issues/7231/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7231/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7231/events","html_url":"https://github.com/apache/spark/pull/7231","id":93170588,"number":7231,"title":"[SPARK-6123] [SPARK-6775] [SPARK-6776] [SQL] Refactors Parquet read path for interoperability and backwards-compatibility","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-07-06T02:14:39Z","updated_at":"2015-07-06T19:16:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7231","html_url":"https://github.com/apache/spark/pull/7231","diff_url":"https://github.com/apache/spark/pull/7231.diff","patch_url":"https://github.com/apache/spark/pull/7231.patch"},"body":"This PR is part of [SPARK-6774] [2], which aims to ensure interoperability and backwards-compatibility for Spark SQL Parquet support.  And this one fixes the read path.  Now Spark SQL is expected to be able to read legacy Parquet data files generated by most (if not all) common libraries/tools like parquet-thrift, parquet-avro, and parquet-hive.\r\n\r\nMajor changes:\r\n\r\n1. `CatalystConverter` class hierarchy refactoring\r\n\r\n   - Replaces `CatalystConverter` trait with a much simpler `ParentContainerUpdater`.\r\n \r\n     Now instead of extending the original `CatalystConverter` trait, every converter class accepts an updater which is responsible to propagate the converted value to some parent container. For example, appending array elements to a parent array buffer, appending a key-value pairs to a parent mutable map, or setting a converted value to some specific field of a parent row. Root converter doesn't have a parent and thus uses a `NoopUpdater`.\r\n \r\n     This simplifies the design since converters don't need to care about details of their parent converters anymore.\r\n \r\n   - Unifies `CatalystRootConverter`, `CatalystGroupConverter` and `CatalystPrimitiveRowConverter` into `CatalystRowConverter`\r\n \r\n     Specifically, now all row objects are represented by `SpecificMutableRow` during conversion.\r\n \r\n   - Refactors `CatalystArrayConverter`, and removes `CatalystArrayContainsNullConverter` and `CatalystNativeArrayConverter`\r\n \r\n     `CatalystNativeArrayConverter` was probably designed with the intention of avoiding boxing costs. However, the way it uses Scala generics actually doesn't achieve this goal.\r\n \r\n     The new `CatalystArrayConverter` handles both nullable and non-nullable array elements in a consistent way.\r\n \r\n   - Implemented backwards-compatibility rules in `CatalystArrayConverter`\r\n \r\n     When Parquet records are being converted, schema of Parquet files should have already been verified. So we only need to care about the structure rather than field names in the Parquet schema. Since all map objects represented in legacy systems have the same structure as the standard one (see [backwards-compatibility rules for MAP] [1]), we only need to deal with LIST (namely array) in `CatalystArrayConverter`.\r\n\r\n2. Requested columns handling\r\n\r\n   When specifying requested columns in `RowReadSupport`, we used to use a Parquet `MessageType` converted from a Catalyst `StructType` which contains all requested columns.  This is not preferable when taking compatibility and interoperability into consideration.  Because the actual Parquet file may have different physical structure from the converted schema.\r\n\r\n   In this PR, the schema for requested columns is constructed using the following method:\r\n\r\n   - For a column that exists in the target Parquet file, we extract the column type by name from the full file schema, and construct a single-field `MessageType` for that column.\r\n   - For a column that doesn't exist in the target Parquet file, we create a single-field `StructType` and convert it to a `MessageType` using `CatalystSchemaConverter`.\r\n   - Unions all single-field `MessageType`s into a full schema containing all requested fields\r\n\r\n   With this change, we also fix [SPARK-6123] [3] by validating the global schema against each individual Parquet part-files.\r\n\r\n[1]: https://github.com/apache/incubator-parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1\r\n[2]: https://issues.apache.org/jira/browse/SPARK-6774\r\n[3]: https://issues.apache.org/jira/browse/SPARK-6123"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/118968058","html_url":"https://github.com/apache/spark/pull/7231#issuecomment-118968058","issue_url":"https://api.github.com/repos/apache/spark/issues/7231","id":118968058,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-07-06T19:16:27Z","updated_at":"2015-07-06T19:16:27Z","body":"  [Test build #36593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36593/consoleFull) for   PR 7231 at commit [`8216e95`](https://github.com/apache/spark/commit/8216e95e9399abae22b1cd198260196a3c9cf14e)."}}