{"push_id":745542225,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"1ebd41b141a95ec264bd2dd50f0fe24cd459035d","before":"98d6d9c7a996f5456eb2653bb96985a1a05f4ce1","commits":[{"sha":"1ebd41b141a95ec264bd2dd50f0fe24cd459035d","author":{"email":"8adc4758e0403b3bd961e4e414291eaa9ba64098@databricks.com","name":"Yin Huai"},"message":"[SPARK-9240] [SQL] Hybrid aggregate operator using unsafe row\n\nThis PR adds a base aggregation iterator `AggregationIterator`, which is used to create `SortBasedAggregationIterator` (for sort-based aggregation) and `UnsafeHybridAggregationIterator` (first it tries hash-based aggregation and falls back to the sort-based aggregation (using external sorter) if we cannot allocate memory for the map). With these two iterators, we will not need existing iterators and I am removing those. Also, we can use a single physical `Aggregate` operator and it internally determines what iterators to used.\n\nhttps://issues.apache.org/jira/browse/SPARK-9240\n\nAuthor: Yin Huai <yhuai@databricks.com>\n\nCloses #7813 from yhuai/AggregateOperator and squashes the following commits:\n\ne317e2b [Yin Huai] Remove unnecessary change.\n74d93c5 [Yin Huai] Merge remote-tracking branch 'upstream/master' into AggregateOperator\nba6afbc [Yin Huai] Add a little bit more comments.\nc9cf3b6 [Yin Huai] update\n0f1b06f [Yin Huai] Remove unnecessary code.\n21fd15f [Yin Huai] Remove unnecessary change.\n964f88b [Yin Huai] Implement fallback strategy.\nb1ea5cf [Yin Huai] wip\n7fcbd87 [Yin Huai] Add a flag to control what iterator to use.\n533d5b2 [Yin Huai] Prepare for fallback!\n33b7022 [Yin Huai] wip\nbd9282b [Yin Huai] UDAFs now supports UnsafeRow.\nf52ee53 [Yin Huai] wip\n3171f44 [Yin Huai] wip\nd2c45a0 [Yin Huai] wip\nf60cc83 [Yin Huai] Also check input schema.\naf32210 [Yin Huai] Check iter.hasNext before we create an iterator because the constructor of the iterato will read at least one row from a non-empty input iter.\n299008c [Yin Huai] First round cleanup.\n3915bac [Yin Huai] Create a base iterator class for aggregation iterators and add the initial version of the hybrid iterator.","distinct":true,"url":"https://api.github.com/repos/apache/spark/commits/1ebd41b141a95ec264bd2dd50f0fe24cd459035d"}]}