Processing several files one-by-one separately by Spark
<p>I need help with implementation one workflow with Apache Spark. My task is in next:</p>

<ol>
<li>I have several CSV files as source data. Note: these files could has different layout</li>
<li>I have metadata with info how I need parse each file (this is not problem)</li>
<li>Main goal: result is source file with several additional columns. I have to update each source file without joining to one output range. For example: source 10 files -> 10 result files and each result file have data only from corresponding source file.</li>
</ol>

<p>As I know Spark can open many files by mask: </p>

<pre><code>var source = sc.textFile("/source/data*.gz");
</code></pre>

<p>But in this case I can't recognize which line of a file. If I get list of source files and try to process by following scenario:</p>

<pre><code>JavaSparkContext sc = new JavaSparkContext(...);
List&lt;String&gt; files = new ArrayList() //list of source files full name's
for(String f : files)
{
   JavaRDD&lt;String&gt; data = sc.textFile(f);
   //process this file with Spark
   outRdd.coalesce(1, true).saveAsTextFile(f + "_out"); 
}
</code></pre>

<p>But in this case I will processed all files in sequential mode.</p>

<p><strong>My question is next</strong>: how I can processed many files in parallel mode?. For example: one file - one executor?</p>

<p>Thank very much for help me!</p>