{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9067","labels_url":"https://api.github.com/repos/apache/spark/issues/9067/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9067/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9067/events","html_url":"https://github.com/apache/spark/pull/9067","id":110873099,"number":9067,"title":"[SPARK-11055][SQL] Use mixing hash-based and sort-based aggregation in TungstenAggregationIterator","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-10-11T18:09:20Z","updated_at":"2015-10-14T22:15:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9067","html_url":"https://github.com/apache/spark/pull/9067","diff_url":"https://github.com/apache/spark/pull/9067.diff","patch_url":"https://github.com/apache/spark/pull/9067.patch"},"body":"JIRA: https://issues.apache.org/jira/browse/SPARK-11055\r\n\r\nIn TungstenAggregationIterator we switch to sort-based aggregation when we can't allocate more memory for hashmap.\r\n\r\nHowever, using external sorter-based aggregation will write too much key-value pairs into disk. We should use mixing hash-based and sort-based aggregation to reduce the key-value pairs needed to write to disk.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/148218738","html_url":"https://github.com/apache/spark/pull/9067#issuecomment-148218738","issue_url":"https://api.github.com/repos/apache/spark/issues/9067","id":148218738,"user":{"login":"JoshRosen","id":50748,"avatar_url":"https://avatars.githubusercontent.com/u/50748?v=3","gravatar_id":"","url":"https://api.github.com/users/JoshRosen","html_url":"https://github.com/JoshRosen","followers_url":"https://api.github.com/users/JoshRosen/followers","following_url":"https://api.github.com/users/JoshRosen/following{/other_user}","gists_url":"https://api.github.com/users/JoshRosen/gists{/gist_id}","starred_url":"https://api.github.com/users/JoshRosen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JoshRosen/subscriptions","organizations_url":"https://api.github.com/users/JoshRosen/orgs","repos_url":"https://api.github.com/users/JoshRosen/repos","events_url":"https://api.github.com/users/JoshRosen/events{/privacy}","received_events_url":"https://api.github.com/users/JoshRosen/received_events","type":"User","site_admin":false},"created_at":"2015-10-14T22:15:14Z","updated_at":"2015-10-14T22:15:14Z","body":"@rxin, my understanding of this patch is that it lets us continue to perform hash-based pre-aggregation on the remainder of the iterator after we've decided to spill and switch to sort-based aggregation. After we've destructed and freed the original hashMap, we'll now loop back around and continue to use a new hash map to aggregate the remainder of the iterator, spilling that map if it also becomes too large.\r\n\r\nAfter this patch, records will have more opportunities to be pre-aggregated before being spilled to disk or sent across the network. I wonder about the case where you're doing map-side pre-aggregation and are experiencing a very low reduction factor: in that case, this patch means that we'll do more work per record, since we'll be hashing and sorting every record, whereas before there was a chance that we'd skip hashing of some records. OTOH, if you have an input consisting of all unique keys then it's best to skip both the hashing AND the sorting and just push the records straight to the reduce side, but that optimization is kind of orthogonal to this one."}}