{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7334","labels_url":"https://api.github.com/repos/apache/spark/issues/7334/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7334/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7334/events","html_url":"https://github.com/apache/spark/pull/7334","id":94166538,"number":7334,"title":"[SPARK-8964] [SQL] [WIP] Use Exchange to perform shuffle in Limit","user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":41,"created_at":"2015-07-09T22:28:09Z","updated_at":"2015-07-21T01:51:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7334","html_url":"https://github.com/apache/spark/pull/7334","diff_url":"https://github.com/apache/spark/pull/7334.diff","patch_url":"https://github.com/apache/spark/pull/7334.patch"},"body":"Spark SQL's physical Limit operator currently performs its own shuffle rather than using Exchange to perform the shuffling. This is less efficient since this non-exchange shuffle path won't be able to benefit from SQL-specific shuffling optimizations, such as SQLSerializer2. It also involves additional unnecessary row copying.\r\n\r\nInstead, I think that we should rewrite Limit to expand into three physical operators:\r\n\r\nPerParititonLimit -> Exchange to one partition -> PerPartitionLimit"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/123127447","html_url":"https://github.com/apache/spark/pull/7334#issuecomment-123127447","issue_url":"https://api.github.com/repos/apache/spark/issues/7334","id":123127447,"user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T01:51:47Z","updated_at":"2015-07-21T01:51:47Z","body":"One hacky approach which would not require any planner changes would be to change `Limit` to hold a reference to the original child being limited.  If `executeCollect()` is called on the post-shuffle `Limit`, then it could just fall back to calling `take()` on the pre-shuffle RDD."}}