{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/5622","labels_url":"https://api.github.com/repos/apache/spark/issues/5622/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/5622/comments","events_url":"https://api.github.com/repos/apache/spark/issues/5622/events","html_url":"https://github.com/apache/spark/pull/5622","id":69975942,"number":5622,"title":"[SPARK-7041] Avoid writing empty files in ExternalSorter","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-04-21T23:20:29Z","updated_at":"2015-04-22T02:52:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/5622","html_url":"https://github.com/apache/spark/pull/5622","diff_url":"https://github.com/apache/spark/pull/5622.diff","patch_url":"https://github.com/apache/spark/pull/5622.patch"},"body":"In ExternalSorter, we may end up opening disk writers files for empty partitions; this occurs because we manually call `open()` after creating the writer, causing serialization and compression input streams to be created; these streams may write headers to the output stream, resulting in non-zero-length files being created for partitions that contain no records. This is unnecessary, though, since the disk object writer will automatically open itself when the first write is performed. Removing this eager open() call and rewriting the consumers to cope with the non-existence of empty files results in a large performance benefit for certain sparse workloads when using sort-based shuffle."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/95008798","html_url":"https://github.com/apache/spark/pull/5622#issuecomment-95008798","issue_url":"https://api.github.com/repos/apache/spark/issues/5622","id":95008798,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-04-22T02:52:55Z","updated_at":"2015-04-22T02:52:55Z","body":"  [Test build #30722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30722/consoleFull) for   PR 5622 at commit [`ad13424`](https://github.com/apache/spark/commit/ad13424c044d7c1c71bb802a5fb08fdf73effc4d)."}}