How to delete duplicates from a file when other fields are varying. We have to just remove the duplicates based on one column
<p>Input:</p>

<pre><code>abc 1001 3452 2463 2343 2176 7654 3452 8765 5643 3452
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
aer 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
abc 1002 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
aer 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
</code></pre>

<p>Output (any one value with abc, tas, aer):</p>

<pre><code>abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 2123 1343 2176 7654 3252 8765 5643 3452
aer 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
</code></pre>

<p>I got this working with scala. Is there any spark equivalent to perform this operation.</p>

<p>Scala code:</p>

<pre><code>import scala.io.Source
import java.io._
import scala.util.control._
val filename = sc.textFile("filenamelocation")
val new_file = new PrintWriter(new File("filenamelocation" ))
val new_file1 = new PrintWriter(new File("filenamelocation" ))
// for (line &lt;- Source.fromFile(filename).getLines)
for (line &lt;- filename.map(x +&gt; x.split(" ").slice(0,3)))
    {
       val first_row=line
       val firstWords = first_row.split(" ")(0)
       // println(firstWords)
       //println(line)
       // val list2Add = List(4,5,6)
       new_file.write(firstWords+"\n")
     }
// println(list2Add)
new_file.close
val list = scala.io.Source.fromFile("filenamelocation", "utf-8").getLines.toList
val l2 = list.removeDuplicates
for (e &lt;- l2)
    {
       val loop = new Breaks;
       loop.breakable
            {
                for (line1 &lt;- Source.fromFile(filename).getLines)
                    {
                        val first_row1=line1
                        val firstWords1 = first_row1.split(" ")(0)
                        if ( e == firstWords1)
                            {
                               new_file1.write(line1+"\n")
                               loop.break;
                             }
                     }
              }
    }
new_file1.close
</code></pre>

<p>Spark code am working on:</p>

<pre><code>import scala.io.Source
import java.io._
import scala.util.control._
val filename = sc.textFile("filelocation")
val test = filename.map(line =&gt; line.split(" ")(0))
val distinct = test.distinct().saveAsTextFile("/user/asugured/test1.txt")
// distinct.foreach(println)
val filename1 = sc.textFile("/user/asugured/test1.txt")
</code></pre>

<p>I've stucked at how to iterate both the files in a loop as I did it in Scala.</p>

<p>Could you please suggest how to handle this.</p>