Pyspark Compare the iterable list of all keys and return the count of same elements
<p>I am working with Pyspark on a data set with key value pair that looks like the following:</p>

<pre><code>[(u'1', u'10'), (u'1', u'15'), (u'1', u'5'), (u'2', u'11'), (u'2', u'15'), (u'2', u'30'),  (u'3', u'10'), (u'3', u'5'), (u'3', u'11')]
</code></pre>

<p>The data could be interpreted as</p>

<pre><code>1 =&gt; 10, 15, 5;  2 =&gt; 11, 15, 30;  3 =&gt; 10, 5, 11;
</code></pre>

<p>I am trying to compare the values of all the keys and find the count the same values. In the sample data, key 1 and 2 both have value 15, the output should return 1; key 1 and 3 both have value 10 and 5, the output should be 2... etc.</p>

<p>Expected output:</p>

<pre><code>1, 2 =&gt; 1;  1, 3 =&gt; 2;  2, 3 =&gt; 1;
</code></pre>

<p>My thought is to group the data by key to get the iterable list and compare individual elements inside the list. </p>

<pre><code>data1 = data.groupByKey()

data1.map(lambda x: (x[0], list(x[1]))).collect()
</code></pre>

<p>The output of data1:</p>

<pre><code>[(u'1', [u'10', u'15', u'15']), (u'2', [u'11', u'15', u'30']), (u'3', [u'10', u'5', u'11'])]
</code></pre>

<p>I am having trouble coming up with a way to iterate through the list and comparing the elements for all the keys. I would appreciate if anyone would know a way to do the iteration or have a suggestion on how to approach the problem in general. Thanks in advance!</p>