{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5526","labels_url":"https://api.github.com/repos/apache/spark/issues/5526/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5526/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5526/events","html_url":"https://github.com/apache/spark/pull/5526","id":68688828,"number":5526,"title":"[SPARK-5182] [SQL] Partitioning support for the data sources API","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":125,"created_at":"2015-04-15T13:53:02Z","updated_at":"2015-05-07T16:56:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5526","html_url":"https://github.com/apache/spark/pull/5526","diff_url":"https://github.com/apache/spark/pull/5526.diff","patch_url":"https://github.com/apache/spark/pull/5526.patch"},"body":"This PR adds partitioning support for the external data sources API. It aims to simplify development of file system based data sources, and provide first class partitioning support for both read path and write path.  Existing data sources like JSON and Parquet can be simplified with this work.\r\n\r\n## New features provided\r\n\r\n1. Hive compatible partition discovery\r\n\r\n   This actually generalizes the partition discovery strategy used in Parquet data source in Spark 1.3.0.\r\n\r\n1. Generalized partition pruning optimization\r\n\r\n   Now partition pruning is handled during physical planning phase.  Specific data sources don't need to worry about this harness anymore.\r\n\r\n   (This also implies that we can remove `CatalystScan` after migrating the Parquet data source, since now we don't need to pass Catalyst expressions to data source implementations.)\r\n\r\n1. Insertion with dynamic partitions\r\n\r\n   When inserting data to a `FSBasedRelation`, data can be partitioned dynamically by specified partition columns.\r\n\r\n## New structures provided\r\n\r\n### Developer API\r\n\r\n1. `FSBasedRelation`\r\n\r\n   Base abstract class for file system based data sources.\r\n\r\n1. `OutputWriter`\r\n\r\n   Base abstract class for output row writers, responsible for writing a single row object.\r\n\r\n1. `FSBasedRelationProvider`\r\n\r\n   A new relation provider for `FSBasedRelation` subclasses. Note that data sources extending `FSBasedRelation` don't need to extend `RelationProvider` and `SchemaRelationProvider`.\r\n\r\n### User API\r\n\r\nNew overloaded versions of\r\n\r\n1. `DataFrame.save()`\r\n1. `DataFrame.saveAsTable()`\r\n1. `SQLContext.load()`\r\n\r\nare provided to allow users to save/load DataFrames with user defined dynamic partition columns.\r\n\r\n### Spark SQL query planning\r\n\r\n1. `InsertIntoFSBasedRelation`\r\n\r\n   Used to implement write path for `FSBasedRelation`s.\r\n\r\n1. New rules for `FSBasedRelation` in `DataSourceStrategy`\r\n\r\n   These are added to hook `FSBasedRelation` into physical query plan in read path, and perform partition pruning.\r\n\r\n## TODO\r\n\r\n- [ ] Use scratch directories when overwriting a table with data selected from itself.\r\n\r\n      Currently, this is not supported, because the table been overwritten is always deleted before writing any data to it.\r\n\r\n- [ ] When inserting with dynamic partition columns, use external sorter to group the data first.\r\n\r\n      This ensures that we only need to open a single `OutputWriter` at a time.  For data sources like Parquet, `OutputWriter`s can be quite memory consuming.  One issue is that, this approach breaks the row distribution in the original DataFrame.  However, we did't promise to preserve data distribution when writing a DataFrame.\r\n\r\n- [ ] More tests.  Specifically, test cases for\r\n\r\n      - [ ] Self-join\r\n      - [ ] Loading partitioned relations with a subset of partition columns stored in data files.\r\n      - [ ] `SQLContext.load()` with user defined dynamic partition columns.\r\n\r\n## Parquet data source migration\r\n\r\nParquet data source migration is covered in PR https://github.com/liancheng/spark/pull/6, which is against this PR branch. A formal PR need to be made after this one is merged."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/99937682","html_url":"https://github.com/apache/spark/pull/5526#issuecomment-99937682","issue_url":"https://api.github.com/repos/apache/spark/issues/5526","id":99937682,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-05-07T16:56:42Z","updated_at":"2015-05-07T16:56:42Z","body":"  [Test build #32121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32121/consoleFull) for   PR 5526 at commit [`c073ced`](https://github.com/apache/spark/commit/c073cedfe288f5567a69a485a7da787e4f0efbc1).\n * This patch **fails MiMa tests**.\n * This patch **does not merge cleanly**.\n * This patch adds the following public classes _(experimental)_:\n  * `        s\"FileOutputCommitter or its subclass is expected, but got a $`\n  * `trait FSBasedRelationProvider `\n  * `abstract class OutputWriter `\n"}}