{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6194","labels_url":"https://api.github.com/repos/apache/spark/issues/6194/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6194/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6194/events","html_url":"https://github.com/apache/spark/pull/6194","id":76789765,"number":6194,"title":"[SPARK-2883] [SQL] ORC data source for Spark SQL","user":{"login":"liancheng","id":230655,"avatar_url":"https://avatars.githubusercontent.com/u/230655?v=3","gravatar_id":"","url":"https://api.github.com/users/liancheng","html_url":"https://github.com/liancheng","followers_url":"https://api.github.com/users/liancheng/followers","following_url":"https://api.github.com/users/liancheng/following{/other_user}","gists_url":"https://api.github.com/users/liancheng/gists{/gist_id}","starred_url":"https://api.github.com/users/liancheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liancheng/subscriptions","organizations_url":"https://api.github.com/users/liancheng/orgs","repos_url":"https://api.github.com/users/liancheng/repos","events_url":"https://api.github.com/users/liancheng/events{/privacy}","received_events_url":"https://api.github.com/users/liancheng/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-05-15T17:09:40Z","updated_at":"2015-05-15T19:57:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6194","html_url":"https://github.com/apache/spark/pull/6194","diff_url":"https://github.com/apache/spark/pull/6194.diff","patch_url":"https://github.com/apache/spark/pull/6194.patch"},"body":"This PR is an update of #6135 authored by @zhzhan from Hortonworks.\r\n\r\n----\r\n\r\nThis PR implements a Spark SQL data source for accessing ORC files.\r\n\r\n> **NOTE**\r\n>\r\n> Although ORC is now an Apache TLP, the codebase is still tightly coupled with Hive.  That's why the new ORC data source is under `org.apache.spark.sql.hive` package.\r\n\r\n## New Features\r\n\r\n1.  New save/load methods provided:\r\n\r\n    - `df.saveAsOrcFile()`\r\n\r\n      Used to save the table in ORC format.\r\n\r\n    - `sqlContext.orcFile()`\r\n\r\n      Used to import ORC file as a Spark SQL table.\r\n\r\n\r\n    To enable these two methods, please add the following line to enable corresponding implicit conversions:\r\n\r\n    ```scala\r\n    import org.apache.spark.sql.hive.orc._\r\n    ```\r\n\r\n1.  Support for complex data types (i.e. array, map, and struct)\r\n\r\n1.  Aware of common optimizations provided by Spark SQL:\r\n\r\n    - Column pruning\r\n    - Partitioning pruning\r\n    - Filter push-down\r\n\r\n1.  Saving/loading ORC files without contacting Hive metastore\r\n\r\n1.  The orc file is operated in HiveContext, the only reason is due to package issue, and we don’t want to bring in hive dependency into spark sql. Note that orc operations does not relies on Hive metastore.\r\n\r\n## Future Work\r\n\r\n1.  Schema evolution support\r\n1.  Hive metastore table conversion\r\n\r\n## Acknowledgements\r\n\r\nThis PR also include initial work done by @scwf from Huawei (PR #3753)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/102509161","html_url":"https://github.com/apache/spark/pull/6194#issuecomment-102509161","issue_url":"https://api.github.com/repos/apache/spark/issues/6194","id":102509161,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-05-15T19:57:11Z","updated_at":"2015-05-15T19:57:11Z","body":" Merged build triggered."}}