{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8512","labels_url":"https://api.github.com/repos/apache/spark/issues/8512/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8512/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8512/events","html_url":"https://github.com/apache/spark/pull/8512","id":103796766,"number":8512,"title":"[SPARK-9926] [SPARK-10340] [SQL] Use S3 bulk listing for S3-backed Hive tables","user":{"login":"piaozhexiu","id":179618,"avatar_url":"https://avatars.githubusercontent.com/u/179618?v=3","gravatar_id":"","url":"https://api.github.com/users/piaozhexiu","html_url":"https://github.com/piaozhexiu","followers_url":"https://api.github.com/users/piaozhexiu/followers","following_url":"https://api.github.com/users/piaozhexiu/following{/other_user}","gists_url":"https://api.github.com/users/piaozhexiu/gists{/gist_id}","starred_url":"https://api.github.com/users/piaozhexiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piaozhexiu/subscriptions","organizations_url":"https://api.github.com/users/piaozhexiu/orgs","repos_url":"https://api.github.com/users/piaozhexiu/repos","events_url":"https://api.github.com/users/piaozhexiu/events{/privacy}","received_events_url":"https://api.github.com/users/piaozhexiu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":42,"created_at":"2015-08-28T21:15:19Z","updated_at":"2015-10-28T01:18:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8512","html_url":"https://github.com/apache/spark/pull/8512","diff_url":"https://github.com/apache/spark/pull/8512.diff","patch_url":"https://github.com/apache/spark/pull/8512.patch"},"body":"This PR addresses both [SPARK-9926](https://issues.apache.org/jira/browse/SPARK-9926) and [SPARK-10340](https://issues.apache.org/jira/browse/SPARK-10340).\r\n\r\n[SPARK-9926](https://issues.apache.org/jira/browse/SPARK-9926) makes it possible to calculate input splits for multiple partitions all together in `TableReader` and cache them in `HadoopRDD`. This improves performance because file listing can be done in parallel by taking advantage of `mapreduce.input.fileinputformat.list-status.num-threads`. This optimization can be enabled by a new property `spark.sql.hive.parallelFileListing`.\r\n\r\n[SPARK-10340](https://issues.apache.org/jira/browse/SPARK-10340) makes it possible to list S3 files using `AmazonS3Client` instead of using `S3N`. This improves performance particularly when listing a large number of S3 files. Note that since S3 bulk listing may list more objects than what meets predicates, they must be additionally filtered after listed. This is done by comparing prefixes of S3 objects against original input paths in `JobConf`. This optimization can be enabled by a new property ` spark.sql.hive.s3BulkListing`.\r\n\r\nHere is my benchmark results for the following queries-\r\n```\r\n1 partition: select * from nccp_log where dateint=20150601 and hour=0 limit 10;\r\n24 partitions: select * from nccp_log where dateint=20150601 limit 10;\r\n240 partitions: select * from nccp_log where dateint>=20150601 and dateint<=20150610 limit 10;\r\n720 partitions: select * from nccp_log where dateint>=20150601 and dateint<=20150630 limit 10;\r\n```\r\n\r\n\\# of files|\\# of partitions|1.5 rc2 (a) |SPARK-9926 w/ 10 threads (b) |SPARK-10340 (c)|\r\n------------|-------------------|---------|------|-----\r\n972 | 1 | 38s | 36s | 27s\r\n13646 | 24 | 354s | 41s | 30s\r\n136222 | 240 | 1h | 321s | 195s\r\n445377 | 720 | 3h | 1194s | 274s\r\n\r\n![image](https://cloud.githubusercontent.com/assets/179618/9556501/fda27662-4d89-11e5-8ce1-0b5d7bab02d1.png)\r\n\r\nThe performance gains come from two factors-\r\n* S3 bulk listing.\r\n* Skipping data locality hints computation for S3 objects. This is possible because S3 objects have no data locality to begin with.\r\n\r\nThe code for S3 bulk listing is fully  implemented in `SparkS3Util`. My code is heavily based on Presto [S3FileSystem](https://github.com/facebook/presto/blob/master/presto-hive/src/main/java/com/facebook/presto/hive/PrestoS3FileSystem.java) and Hadoop [FileInputFormat](https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/FileInputFormat.html)."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/151689805","html_url":"https://github.com/apache/spark/pull/8512#issuecomment-151689805","issue_url":"https://api.github.com/repos/apache/spark/issues/8512","id":151689805,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-10-28T01:18:40Z","updated_at":"2015-10-28T01:18:40Z","body":"**[Test build #44476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44476/consoleFull)** for PR 8512 at commit [`7150da7`](https://github.com/apache/spark/commit/7150da7259148858c419f71ae5d9d0e24c55f4ea).\n * This patch passes all tests.\n * This patch merges cleanly.\n * This patch adds no public classes."}}