{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9406","labels_url":"https://api.github.com/repos/apache/spark/issues/9406/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9406/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9406/events","html_url":"https://github.com/apache/spark/pull/9406","id":114560254,"number":9406,"title":"[SPARK-9241][SQL] Supporting multiple DISTINCT columns (2) - Rewriting Rule","user":{"login":"hvanhovell","id":9616802,"avatar_url":"https://avatars.githubusercontent.com/u/9616802?v=3","gravatar_id":"","url":"https://api.github.com/users/hvanhovell","html_url":"https://github.com/hvanhovell","followers_url":"https://api.github.com/users/hvanhovell/followers","following_url":"https://api.github.com/users/hvanhovell/following{/other_user}","gists_url":"https://api.github.com/users/hvanhovell/gists{/gist_id}","starred_url":"https://api.github.com/users/hvanhovell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hvanhovell/subscriptions","organizations_url":"https://api.github.com/users/hvanhovell/orgs","repos_url":"https://api.github.com/users/hvanhovell/repos","events_url":"https://api.github.com/users/hvanhovell/events{/privacy}","received_events_url":"https://api.github.com/users/hvanhovell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-11-02T09:47:06Z","updated_at":"2015-11-07T00:02:25Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9406","html_url":"https://github.com/apache/spark/pull/9406","diff_url":"https://github.com/apache/spark/pull/9406.diff","patch_url":"https://github.com/apache/spark/pull/9406.patch"},"body":"The second PR for SPARK-9241, this adds support for multiple distinct columns to the new aggregation code path.\r\n\r\nThis PR solves the multiple DISTINCT column problem by rewriting these Aggregates into an Expand-Aggregate-Aggregate combination. See the [JIRA ticket](https://issues.apache.org/jira/browse/SPARK-9241) for some information on this. The advantages over the - competing - [first PR](https://github.com/apache/spark/pull/9280) are:\r\n- This can use the faster TungstenAggregate code path.\r\n- It is impossible to OOM due to an ```OpenHashSet``` allocating to much memory. However, this will multiply the number of input rows by the number of distinct clauses (plus one), and puts a lot more memory pressure on the aggregation code path itself.\r\n\r\nThe location of this Rule is a bit funny, and should probably change when the old aggregation path is changed.\r\n\r\ncc @yhuai - Could you also tell me where to add tests for this?"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/154581656","html_url":"https://github.com/apache/spark/pull/9406#issuecomment-154581656","issue_url":"https://api.github.com/repos/apache/spark/issues/9406","id":154581656,"user":{"login":"marmbrus","id":527,"avatar_url":"https://avatars.githubusercontent.com/u/527?v=3","gravatar_id":"","url":"https://api.github.com/users/marmbrus","html_url":"https://github.com/marmbrus","followers_url":"https://api.github.com/users/marmbrus/followers","following_url":"https://api.github.com/users/marmbrus/following{/other_user}","gists_url":"https://api.github.com/users/marmbrus/gists{/gist_id}","starred_url":"https://api.github.com/users/marmbrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marmbrus/subscriptions","organizations_url":"https://api.github.com/users/marmbrus/orgs","repos_url":"https://api.github.com/users/marmbrus/repos","events_url":"https://api.github.com/users/marmbrus/events{/privacy}","received_events_url":"https://api.github.com/users/marmbrus/received_events","type":"User","site_admin":false},"created_at":"2015-11-07T00:02:25Z","updated_at":"2015-11-07T00:02:25Z","body":"Okay, I looked over this pretty quickly and it looks awesome.  We need some tests and we are super close to me cutting a preview release.  That said, I'd really like to include this in 1.6.  Here is my proposal:\r\n - Lets merge this as is.\r\n - Yin will start ripping out the old aggregation path. (with or without #9409).\r\n - Comments and TODOs can be addressed in a follow up."}}