How to make the first row as header when reading a file in PySpark and converting it to Pandas Dataframe
<p>I am reading a file in <code>PySpark</code> and forming the <code>rdd</code> of it. I then convert it to a normal <code>dataframe</code> and then to <code>pandas dataframe</code>. The issue that I am having is that there is header row in my input file and I want to make this as the header of dataframe columns as well but they are read in as an additional row and not as header. This is my current code:</p>

<pre><code>def extract(line):
    return line


input_file = sc.textFile('file1.txt').zipWithIndex().filter(lambda (line,rownum): rownum&gt;=0).map(lambda (line, rownum): line)

input_data = (input_file
    .map(lambda line: line.split(";"))
    .filter(lambda line: len(line) &gt;=0 )
    .map(extract)) # Map to tuples

df_normal = input_data.toDF()
df= df_normal.toPandas()
</code></pre>

<p>Now when I look at the <code>df</code> then the header row of text file becomes the first row of <code>dataframe</code> and there is additional header in <code>df</code> with <code>0,1,2...</code> as header. How can I make the first row as header?</p>