Implementing a Spark SQL UserDefinedAggregateFunction that performs multiple passes over a column
<p>I've been experimenting with the <code>UserDefinedAggregateFunction</code> class to write aggregate functions for use in Spark SQL.</p>

<p>It works well for implementing <i>single pass</i> operations like <code>sum()</code>, <code>avg()</code> etc., but is there a trick you can use to perform multiple passes over a column?</p>

<p>For example, Calculating variance using the <i>naive</i> approach. i.e. With a first pass calculating the column mean and then a second pass that uses this value to calculate the variance. I know that there are single pass algorithms for doing this that give good approximations (as in fact implemented by Spark). I was just using this as an example of a two-pass operation.</p>

<p>It would be nice to be able to do the following,</p>

<p><code>spark.sql("SELECT product, MultiPassAgg(price) FROM products GROUP BY product")</code></p>

<p>I appreciate that I can do this kind of thing using <code>Dataset</code> / <code>DataFrame</code> operations in stages etc., but I was just looking <i>clean</i> approach as illustrated in the SQL above.</p>

<p>Any ideas or suggestions?</p>