Find related rows from a dataframe
<p>I have a dataframe type (name, id, Dept, DeptId):</p>

<pre><code>john doe  | J30 | NSL | 01
max payne | M23 | ATL | 22
doe john  | D02 | NSL | 01
joyee     | J04 | NSL | 01
payne max | P04 | ATL | 22
</code></pre>

<p>Some data was wrongly entered and new ID's where created. I'm trying to get a list </p>

<pre><code>J30 -&gt; [ J30,D02 ]
M23 -&gt; [ M23,P04 ]
D02 -&gt; [ J30,D02 ]
J04 -&gt; [ J04 ]
P04 -&gt; [ M23,P04 ]
</code></pre>

<p>Ultimately remove duplicates</p>

<pre><code>U1 -&gt; [ J30,D02 ]  
U2 -&gt; [ M23,P04 ]
U3 -&gt; [ J04 ]
</code></pre>

<p>Dataset I have is huge, I'm unable to find anyways through spark to get the job done. Any pointers would be great.</p>