{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6397","labels_url":"https://api.github.com/repos/apache/spark/issues/6397/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6397/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6397/events","html_url":"https://github.com/apache/spark/pull/6397","id":80616403,"number":6397,"title":"[SPARK-7855] Move bypassMergeSort-handling from ExternalSorter to own component","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":43,"created_at":"2015-05-25T17:58:10Z","updated_at":"2015-05-26T06:24:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6397","html_url":"https://github.com/apache/spark/pull/6397","diff_url":"https://github.com/apache/spark/pull/6397.diff","patch_url":"https://github.com/apache/spark/pull/6397.patch"},"body":"Spark's `ExternalSorter` writes shuffle output files during sort-based shuffle. Sort-shuffle contains a configuration, `spark.shuffle.sort.bypassMergeThreshold`, which causes ExternalSorter to skip sorting and merging and simply write separate files per partition, which are then concatenated together to form the final map output file.\r\n\r\nThe code paths used during this bypass are almost completely separate from ExternalSorter's other code paths, so refactoring them into a separate file can significantly simplify the code.\r\n\r\nIn addition to re-arranging code, this patch deletes hundreds of lines of dead code.  The main entry point into ExternalSorter is `insertAll()` and in SPARK-4479 / #3422 this method was modified to completely bypass in-memory buffering of records when `bypassMergeSort` takes effect. As a result, the spilling / merging code paths will no longer be called when `bypassMergeSort` is used, so we should be able to safely remove that code.\r\n\r\nThere's an open JIRA ([SPARK-6026](https://issues.apache.org/jira/browse/SPARK-6026)) for removing the `bypassMergeThreshold` parameter and code paths; I have not done that here, but the changes in this patch will make removing that parameter significantly easier if we ever decide to do that."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/105414475","html_url":"https://github.com/apache/spark/pull/6397#issuecomment-105414475","issue_url":"https://api.github.com/repos/apache/spark/issues/6397","id":105414475,"user":{"login":"rxin","id":323388,"avatar_url":"https://avatars.githubusercontent.com/u/323388?v=3","gravatar_id":"","url":"https://api.github.com/users/rxin","html_url":"https://github.com/rxin","followers_url":"https://api.github.com/users/rxin/followers","following_url":"https://api.github.com/users/rxin/following{/other_user}","gists_url":"https://api.github.com/users/rxin/gists{/gist_id}","starred_url":"https://api.github.com/users/rxin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rxin/subscriptions","organizations_url":"https://api.github.com/users/rxin/orgs","repos_url":"https://api.github.com/users/rxin/repos","events_url":"https://api.github.com/users/rxin/events{/privacy}","received_events_url":"https://api.github.com/users/rxin/received_events","type":"User","site_admin":false},"created_at":"2015-05-26T06:24:19Z","updated_at":"2015-05-26T06:24:19Z","body":"@zsxwing please review this one as well. Thanks!\r\n"}}