{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3247","labels_url":"https://api.github.com/repos/apache/spark/issues/3247/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3247/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3247/events","html_url":"https://github.com/apache/spark/pull/3247","id":48641744,"number":3247,"title":"[SPARK-4233] [SQL] WIP:Simplify the UDAF API (Interface)","user":{"login":"chenghao-intel","id":2367031,"avatar_url":"https://avatars.githubusercontent.com/u/2367031?v=3","gravatar_id":"","url":"https://api.github.com/users/chenghao-intel","html_url":"https://github.com/chenghao-intel","followers_url":"https://api.github.com/users/chenghao-intel/followers","following_url":"https://api.github.com/users/chenghao-intel/following{/other_user}","gists_url":"https://api.github.com/users/chenghao-intel/gists{/gist_id}","starred_url":"https://api.github.com/users/chenghao-intel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenghao-intel/subscriptions","organizations_url":"https://api.github.com/users/chenghao-intel/orgs","repos_url":"https://api.github.com/users/chenghao-intel/repos","events_url":"https://api.github.com/users/chenghao-intel/events{/privacy}","received_events_url":"https://api.github.com/users/chenghao-intel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":33,"created_at":"2014-11-13T14:29:59Z","updated_at":"2015-01-05T06:29:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3247","html_url":"https://github.com/apache/spark/pull/3247","diff_url":"https://github.com/apache/spark/pull/3247.diff","patch_url":"https://github.com/apache/spark/pull/3247.patch"},"body":"Simplify the UDAF API is the first step of optimization for Aggregation (see https://issues.apache.org/jira/browse/SPARK-4366).\r\n\r\nCurrently UDAF cannot scale up when data volume grows, particularly for the `distinct` aggregation expressions.  This PR doesn't aim for fixing the `distinct` performance, but facilitate \r\n\r\n* More straightforward API for UDAF implementation\r\nDevelopers will not write the distinct expression any more, like `DistinctAverage` is not necessary, since `Average` is provided, the framework will handle the `distinct` internally\r\n* Schema-ed Aggregation Buffer\r\nAggregation Buffer is stored as `MutableRow` with schema, which means the UDAF developers will benefit from the Catalyst Expression framework in UDAF development. And shuffling the aggregation buffers cross the machine boundary is transparently from the UDAF developers, too, that provide us a chance to switch the aggregation algorithm (sort-based / hash-based)\r\n\r\nIt's a WIP PR, hopefully we could discuss the UDAF interface design in details, and a sample implementation for `max`, and `avg` is also provided.\r\n\r\nOnce we got generally agreement for the new UDAF design, I will continue to finish this PR:\r\n- [x] Integrated with Hive UDAF (Generic UDAF)\r\n- [x] Re-implemented the existed UDAF (e.g. `max`, `first`, `last` etc.)\r\n- [ ] Code style issues. \r\n\r\n* Part of it's changes was in #3429, will rebase when it's merged."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/68672838","html_url":"https://github.com/apache/spark/pull/3247#issuecomment-68672838","issue_url":"https://api.github.com/repos/apache/spark/issues/3247","id":68672838,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-01-05T06:29:44Z","updated_at":"2015-01-05T06:29:44Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25047/\nTest PASSed."}}