{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6278","labels_url":"https://api.github.com/repos/apache/spark/issues/6278/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6278/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6278/events","html_url":"https://github.com/apache/spark/pull/6278","id":78419341,"number":6278,"title":"[SPARK-7712] [SQL] Move Window Functions from Hive UDAFS to Spark Native backend [WIP]","user":{"login":"hvanhovell","id":9616802,"avatar_url":"https://avatars.githubusercontent.com/u/9616802?v=3","gravatar_id":"","url":"https://api.github.com/users/hvanhovell","html_url":"https://github.com/hvanhovell","followers_url":"https://api.github.com/users/hvanhovell/followers","following_url":"https://api.github.com/users/hvanhovell/following{/other_user}","gists_url":"https://api.github.com/users/hvanhovell/gists{/gist_id}","starred_url":"https://api.github.com/users/hvanhovell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hvanhovell/subscriptions","organizations_url":"https://api.github.com/users/hvanhovell/orgs","repos_url":"https://api.github.com/users/hvanhovell/repos","events_url":"https://api.github.com/users/hvanhovell/events{/privacy}","received_events_url":"https://api.github.com/users/hvanhovell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-05-20T07:54:02Z","updated_at":"2015-06-10T20:25:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6278","html_url":"https://github.com/apache/spark/pull/6278","diff_url":"https://github.com/apache/spark/pull/6278.diff","patch_url":"https://github.com/apache/spark/pull/6278.patch"},"body":"This PR aims to improve the current window functionality in Spark SQL in the following ways:\r\n- Moving away from Hive backed UDAF's to a more Spark SQL Native implementation. The main advantages are that some overhead can be avoided, that it it easier to extend, and that it opens up the opportunity to do more aggressive code generation (AggregateEvaluation style).\r\n- Improve the processing for the 'between unbounded and current' amd the 'between current and unbounded' cases. This was very expensive, now it performs in linear time.\r\n- Process different frames for the same group by/order by definition in the same window clause. This should save some memory and reduce some of the overhead.\r\n\r\nThis is currently a work in progress. Quite a bit of testing needs to be done. Any feedback will be greatly appreciated. The JIRA ticket can be found here: https://issues.apache.org/jira/browse/SPARK-7712\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/110901092","html_url":"https://github.com/apache/spark/pull/6278#issuecomment-110901092","issue_url":"https://api.github.com/repos/apache/spark/issues/6278","id":110901092,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2015-06-10T20:25:47Z","updated_at":"2015-06-10T20:25:47Z","body":"  [Test build #34634 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34634/console) for   PR 6278 at commit [`36643cf`](https://github.com/apache/spark/commit/36643cf0fcc0d6a2ee9c06f2d83ec36ce2449846).\n * This patch **fails Scala style tests**.\n * This patch merges cleanly.\n * This patch adds the following public classes _(experimental)_:\n  * `abstract class WindowSpec extends Expression `\n  * `case class WindowSpecReference(name: String) extends WindowSpec with LeafNode[Expression] `\n  * `case class WindowExpression(`\n  * `trait ImpliedOrderSpec `\n  * `case class ComposedWindowFunction(child: Expression) extends Expression`\n  * `abstract class RankLikeExpression extends AggregateExpression with ImpliedOrderSpec`\n  * `abstract class RankLikeFunction extends AggregateFunction `\n  * `case class Rank(children: Seq[Expression]) extends RankLikeExpression `\n  * `case class RankFunction(override val children: Seq[Expression], base: AggregateExpression)`\n  * `case class DenseRank(children: Seq[Expression]) extends RankLikeExpression `\n  * `case class DenseRankFunction(override val children: Seq[Expression], base: AggregateExpression)`\n  * `  case class Token(name: String, children: Seq[ASTNode]) extends Node `\n"}}