{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7458","labels_url":"https://api.github.com/repos/apache/spark/issues/7458/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7458/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7458/events","html_url":"https://github.com/apache/spark/pull/7458","id":95582049,"number":7458,"title":"[SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-3056] [SQL] Aggregation Improvement","user":{"login":"yhuai","id":2072857,"avatar_url":"https://avatars.githubusercontent.com/u/2072857?v=3","gravatar_id":"","url":"https://api.github.com/users/yhuai","html_url":"https://github.com/yhuai","followers_url":"https://api.github.com/users/yhuai/followers","following_url":"https://api.github.com/users/yhuai/following{/other_user}","gists_url":"https://api.github.com/users/yhuai/gists{/gist_id}","starred_url":"https://api.github.com/users/yhuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yhuai/subscriptions","organizations_url":"https://api.github.com/users/yhuai/orgs","repos_url":"https://api.github.com/users/yhuai/repos","events_url":"https://api.github.com/users/yhuai/events{/privacy}","received_events_url":"https://api.github.com/users/yhuai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":62,"created_at":"2015-07-17T04:16:25Z","updated_at":"2015-07-22T04:53:28Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7458","html_url":"https://github.com/apache/spark/pull/7458","diff_url":"https://github.com/apache/spark/pull/7458.diff","patch_url":"https://github.com/apache/spark/pull/7458.patch"},"body":"This is the first PR for the aggregation improvement, which is tracked by https://issues.apache.org/jira/browse/SPARK-4366 (umbrella JIRA). This PR contains work for its subtasks, SPARK-3056, SPARK-3947, SPARK-4233, and SPARK-4367.\r\n\r\nThis PR introduces a new code path for evaluating aggregate functions. This code path is guarded by `spark.sql.useAggregate2` and by default the value of this flag is true.\r\n\r\nThis new code path contains:\r\n* A new aggregate function interface (`AggregateFunction2`) and 7 built-int aggregate functions based on this new interface (`AVG`, `COUNT`, `FIRST`, `LAST`, `MAX`, `MIN`, `SUM`)\r\n* A UDAF interface (`UserDefinedAggregateFunction`) based on the new code path and two example UDAFs (`MyDoubleAvg` and `MyDoubleSum`).\r\n* A sort-based aggregate operator (`Aggregate2Sort`) for the new aggregate function interface .\r\n* A sort-based aggregate operator (`FinalAndCompleteAggregate2Sort`) for distinct aggregations (for distinct aggregations the query plan will use `Aggregate2Sort` and `FinalAndCompleteAggregate2Sort` together).\r\n\r\nWith this change, `spark.sql.useAggregate2` is `true`, the flow of compiling an aggregation query is:\r\n1. Our analyzer looks up functions and returns aggregate functions built based on the old aggregate function interface.\r\n2. When our planner is compiling the physical plan, it tries try to convert all aggregate functions to the ones built based on the new interface. The planner will fallback to the old code path if any of the following two conditions is true:\r\n* code-gen is disabled.\r\n* there is any function that cannot be converted (right now, Hive UDAFs).\r\n* the schema of grouping expressions contain any complex data type.\r\n* There are multiple distinct columns.\r\n\r\nRight now, the new code path handles a single distinct column in the query (you can have multiple aggregate functions using that distinct column). For a query having a aggregate function with DISTINCT and regular aggregate functions, the generated plan will do partial aggregations for those regular aggregate function."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/123562244","html_url":"https://github.com/apache/spark/pull/7458#issuecomment-123562244","issue_url":"https://api.github.com/repos/apache/spark/issues/7458","id":123562244,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-07-22T04:53:28Z","updated_at":"2015-07-22T04:53:28Z","body":"Merged build finished. Test FAILed."}}