Counting distinct texts in a Spark RDD with array objects
<p>I have a spark rdd (<code>words</code>) which consists of arrays of texts. For an example, </p>

<pre><code>words.take(3)
</code></pre>

<p>would return something like.</p>

<pre><code>[ ["A", "B"], ["B", "C"], ["C", "A", "D"] ]
</code></pre>

<p>Now, I want to find out the total number of texts as well as the unique number of texts. If the RDD had only above 3 records, </p>

<pre><code>total_words = 7
unique_words = 4 (only A, B,C,D)
</code></pre>

<p>Now in to order to get the total, I did something similar like this</p>

<pre><code>text_count_rdd = words.map(lambda x: len(x))
text_count_rdd.sum()
</code></pre>

<p>But I'm stuck on how to retrieve the unique count. </p>