{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/8346","labels_url":"https://api.github.com/repos/apache/spark/issues/8346/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/8346/comments","events_url":"https://api.github.com/repos/apache/spark/issues/8346/events","html_url":"https://github.com/apache/spark/pull/8346","id":102271917,"number":8346,"title":"[SPARK-10143] [SQL] Use parquet's block size (row group size) setting as the min split size if necessary.","user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-08-21T01:06:16Z","updated_at":"2015-08-21T03:25:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/8346","html_url":"https://github.com/apache/spark/pull/8346","diff_url":"https://github.com/apache/spark/pull/8346.diff","patch_url":"https://github.com/apache/spark/pull/8346.patch"},"body":"https://issues.apache.org/jira/browse/SPARK-10143\r\n\r\nWith this PR, we will set min split size to parquet's block size (row group size) set in the conf if the min split size is smaller. So, we can avoid have too many tasks and even useless tasks for reading parquet data.\r\n\r\nI tested it locally. The table I have has 343MB and it is in my local FS. Because I did not set any min/max split size, the default split size was 32MB and the map stage had 11 tasks. But there were only three tasks that actually read data. With my PR, there were only three tasks in the map stage. Here is the difference.\r\n\r\nWithout this PR:\r\n![image](https://cloud.githubusercontent.com/assets/2072857/9399179/8587dba6-4765-11e5-9189-7ebba52a2b6d.png)\r\n\r\nWith this PR:\r\n![image](https://cloud.githubusercontent.com/assets/2072857/9399185/a4735d74-4765-11e5-8848-1f1e361a6b4b.png)\r\n\r\nEven if the block size setting does match the actual block size of parquet file, I think it is still generally good to use parquet's block size setting if min split size is smaller than this block size."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/133266924","html_url":"https://github.com/apache/spark/pull/8346#issuecomment-133266924","issue_url":"https://api.github.com/repos/apache/spark/issues/8346","id":133266924,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-08-21T03:25:24Z","updated_at":"2015-08-21T03:25:24Z","body":"Merged build finished. Test PASSed."}}