How to coalesce array columns in Spark dataframe
<p>Suppose I have the following dataframe:</p>

<pre><code>id | myStruct
___________________
1  | [val1, val2]
___________________
2  | [val3, val4]
___________________
1  | [val5, val6]
</code></pre>

<p>I would like to group all myStructs that share the same id into an array column of myStructs. So, the above dataframe should become</p>

<pre><code>id | myStruct
__________________________________
1  | [[val1, val2], [val5, val6]]
__________________________________
2  | [[val3, val4]]
</code></pre>

<p>I know there is an array function, but that only converts each column into an array of size 1. How do I coalesce the resulting arrays?</p>

<p>I am using Spark 1.5.2 in a Scala shell.</p>

<p>Given that I am using Spark 1.5.2, I cannot use collect_list or collect_set.</p>