Spark: Removing rows which occur less than N times
<p>Lets say that I am having the following rdd:</p>



<pre class="lang-py prettyprint-override"><code>a = [('a',1),('a',2),('a',3),('b',1),('b',4),('c',3)]
anRDD = sc.parallelize(a)
</code></pre>

<p>And I want to get the rows that they key occurs more than N times(for this example lets say more or equal than 2). And in another rdd the ones that where excluded.</p>

<p>What I do is the following:</p>

<pre class="lang-py prettyprint-override"><code>threshold = 2
anRDD.persist()
grouped_counts = anRDD
                    .toDF(['letter','number'])
                    .groupBy('letter')
                    .count()

downFromLimit = grouped_counts.filter(grouped_counts['count']&lt;threshold).select("letter").map(lambda x:x.letter).collect()  
upTheLimit = grouped_counts.filter(grouped_counts['count']&gt;=threshold).select("letter").map(lambda x:x.letter).collect()

upData = anRDD.filter(lambda x:x[0] in upTheLimit)
downData = anRDD.filter(lambda x:x[0] in downFromLimit)
anRDD.unpersist()
</code></pre>

<p>It does what I want but it should something more clear,easier and more efficient than this.</p>

<p>If I use <code>reduceByKey</code> and count the length of the values would it be more beneficial? </p>

<p>Any other idea?</p>