{"push_id":863905492,"size":1,"distinct_size":1,"ref":"refs/heads/branch-1.6","head":"6c1bf19e84aa08291e6595e6bc54f9e2e375169a","before":"f061d2539931a2feab7229b95d3a38671ec4c0d3","commits":[{"sha":"6c1bf19e84aa08291e6595e6bc54f9e2e375169a","author":{"email":"17b9e1c64588c7fa6419b4d29dc1f4426279ba01@databricks.com","name":"Michael Armbrust"},"message":"[SPARK-11654][SQL] add reduce to GroupedDataset\n\nThis PR adds a new method, `reduce`, to `GroupedDataset`, which allows similar operations to `reduceByKey` on a traditional `PairRDD`.\n\n```scala\nval ds = Seq(\"abc\", \"xyz\", \"hello\").toDS()\nds.groupBy(_.length).reduce(_ + _).collect()  // not actually commutative :P\n\nres0: Array(3 -> \"abcxyz\", 5 -> \"hello\")\n```\n\nWhile implementing this method and its test cases several more deficiencies were found in our encoder handling.  Specifically, in order to support positional resolution, named resolution and tuple composition, it is important to keep the unresolved encoder around and to use it when constructing new `Datasets` with the same object type but different output attributes.  We now divide the encoder lifecycle into three phases (that mirror the lifecycle of standard expressions) and have checks at various boundaries:\n\n - Unresoved Encoders: all users facing encoders (those constructed by implicits, static methods, or tuple composition) are unresolved, meaning they have only `UnresolvedAttributes` for named fields and `BoundReferences` for fields accessed by ordinal.\n - Resolved Encoders: internal to a `[Grouped]Dataset` the encoder is resolved, meaning all input has been resolved to a specific `AttributeReference`.  Any encoders that are placed into a logical plan for use in object construction should be resolved.\n - BoundEncoder: Are constructed by physical plans, right before actual conversion from row -> object is performed.\n\nIt is left to future work to add explicit checks for resolution and provide good error messages when it fails.  We might also consider enforcing the above constraints in the type system (i.e. `fromRow` only exists on a `ResolvedEncoder`), but we should probably wait before spending too much time on this.\n\nAuthor: Michael Armbrust <michael@databricks.com>\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #9673 from marmbrus/pr/9628.\n\n(cherry picked from commit 41bbd2300472501d69ed46f0407d5ed7cbede4a8)\nSigned-off-by: Michael Armbrust <michael@databricks.com>","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/6c1bf19e84aa08291e6595e6bc54f9e2e375169a"}]}