Use collect_list and collect_set in Spark SQL
<p>According to the <a href="http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$" rel="noreferrer">docs</a>, the <code>collect_set</code> and <code>collect_list</code> functions should be available in Spark SQL. However, I cannot get it to work. I'm running Spark 1.6.0 using a <a href="https://github.com/sequenceiq/docker-spark" rel="noreferrer">Docker image</a>. </p>

<p>I'm trying to do this in Scala: </p>

<pre><code>import org.apache.spark.sql.functions._ 

df.groupBy("column1") 
  .agg(collect_set("column2")) 
  .show() 
</code></pre>

<p>And receive the following error at runtime: </p>

<pre><code>Exception in thread "main" org.apache.spark.sql.AnalysisException: undefined function collect_set; 
</code></pre>

<p>Also tried it using <code>pyspark</code>, but it also fails. The docs state these functions are aliases of Hive UDAFs, but I can't figure out to enable these functions.</p>

<p>How to fix this? Thanx!</p>