Merge Spark output CSV files with a single header
<p>I want to create a data processing pipeline in AWS to eventually use the processed data for Machine Learning.</p>

<p>I have a Scala script that takes raw data from S3, processes it and writes it to HDFS or even S3 with <strong>Spark-CSV</strong>. I think I can use multiple files as input if I want to use <strong>AWS Machine Learning</strong> tool for training a prediction model. But if I want to use something else, I presume it is best if I receive a single CSV output file.</p>

<p>Currently, as I do not want to use <strong>repartition(1)</strong> nor <strong>coalesce(1)</strong> for performance purposes, I have used <strong>hadoop fs -getmerge</strong> for manual testing, but as it just merges the contents of the job output files, I am running into a small problem. I need <strong>a single row of headers</strong> in the data file for training the prediction model.</p>

<p>If I use <code>.option("header","true")</code> for the spark-csv, then it writes the headers to every output file and after merging I have as many lines of headers in the data as there were output files. But if the header option is false, then it does not add any headers.</p>

<p>Now I found an option to merge the files inside the Scala script with Hadoop API <a href="https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileUtil.html#copyMerge(org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path,%20boolean,%20org.apache.hadoop.conf.Configuration,%20java.lang.String)" rel="noreferrer"><code>FileUtil.copyMerge</code></a>. I tried this in <code>spark-shell</code> with the code below. </p>

<pre><code>import org.apache.hadoop.fs.FileUtil
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
val configuration = new Configuration();
val fs = FileSystem.get(configuration);
FileUtil.copyMerge(fs, new Path("smallheaders"), fs, new Path("/home/hadoop/smallheaders2"), false, configuration, "")
</code></pre>

<p>But this solution still just concatenates the files on top of each other and does not handle headers. <strong>How can I get an output file with only one row of headers?</strong></p>

<p>I even tried adding <code>df.columns.mkString(",")</code> as the last argument for <code>copyMerge</code>, but this added the headers still multiple times, not once.</p>