Joining two Spark DataFrame according to size of intersection of two array columns
<p>I have two <code>DataFrame</code> in my spark (v1.5.0) code: </p>

<pre><code>aDF = [user_id : Int, user_purchases: array&lt;int&gt; ]
bDF = [user_id : Int, user_purchases: array&lt;int&gt; ]
</code></pre>

<p>What I want to do is to join these two dataframes, but I only need the lines where the intersection between <code>aDF.user_purchases</code> and <code>bDF.user_purchases</code> has more than 2 elements (intersection > 2).</p>

<p>Do I have to use RDD API or is it possible to use some function from org.apache.sql.functions ? </p>