Most efficient way to load many files in spark in parallel?
<pre><code>[Disclaimer: While this question is somewhat specific, I think it circles a very generic issue with Hadoop/Spark.]
</code></pre>

<p>I need to process a large dataset (~14TB) in Spark. Not doing aggregations, mostly filtering.  Given ~30k files (250 part files, per month for 10 years, each part being ~ 200MB), I would like to load them into a RDD/DataFrame and filter out items based on some arbitrary filters.  </p>

<p>To make the listing of the files efficient (I'm on google dataproc/cloud storage, so the driver doing a wildcard glob was very serial and very slow), I precalculate an RDD of the file names, then load them into an RDD (I'm using avro, but file type shouldn't be relevant), e.g.</p>

<pre><code>#returns an array of files to load
files = sc.textFile('/list/of/files/').collect()  

#load the files into a dataframe
documents = sqlContext.read.format('com.databricks.spark.avro').load(files)
</code></pre>

<p>When I do this, even on a 50-worker cluster, it seems that only one executor is doing the work of reading the files.  I've experimented with broadcasting the files list and read a dozen different approaches but I can't seem to crack the issue.</p>

<p>So, is there an efficient way to create a very large dataframe from multiple files?  How do I best take advantage of all the potential computing power when creating this RDD? </p>

<p>This approach works very well on smaller sets but, at this size, I see a large number of symptoms like long-running processes with no feedback.  Is there some treasure trove of knowledge -- besides @zero323 :-) -- on optimizing spark at this scale?</p>