Spark job in Java: how to access files from 'resources' when run on a cluster
<p>I wrote a Spark job in Java. The job is packaged as a shaded jar and executed:</p>

<pre><code>spark-submit my-jar.jar
</code></pre>

<p>In the code, there are some files (Freemarker templates) that reside in <code>src/main/resources/templates</code>. When run locally, I'm able access the files:</p>

<pre><code>File[] files = new File("src/main/resources/templates/").listFiles();
</code></pre>

<p>When the job is run on a cluster, a null-pointer exception is returned when the previous line is executed.</p>

<p>If I run <code>jar tf my-jar.jar</code> I can see that the files are packaged in a <code>templates/</code> folder:</p>

<pre><code> [...]
 templates/
 templates/my_template.ftl
 [...]
</code></pre>

<p>I'm just unable to read them; I suspect that <code>.listFiles()</code> tries to access the local filesystem on the cluster node, and the files aren't there.</p>

<p>I'm curious to know how I should package files to be used within a self-contained Spark job. I'd rather not copy them to HDFS outside of the job because it becomes messy to maintain.</p>