Efficient union of multiple small RDDs
<p>I have a sequence of multiple small files (~1-8KB) and I want to calculate the word count of these files. Specifically, the Sequence I have is <code>files: Seq[String]</code> where each string of the sequence is the path to each file. What I do to calculate the total word count based on that sequence is this:</p>

<pre><code>val totalWordCount = sc.union(
      files.map(path =&gt; sc.textFile(path))
    ).flatMap(line =&gt; line.split(" "))
      .map((_,1))
      // I use a hash partitioner for better performance of reduceByKey
      .partitionBy(new HashPartitioner(numPartitions))
      .reduceByKey(_ + _)
</code></pre>

<p>The problem I am having is that even when I have more than 10000 small files and use the above technique the execution time increases when I increase the partitions. Why is that?</p>

<p>Note that I cannot merge these small files into one from the beginning and as an input I need to have the Sequence of strings.</p>