Using typesafe config with Spark on Yarn
<p>I have a Spark job that reads data from a configuration file. This file is a typesafe config file.</p>

<p>The code that reads the config looks like that:</p>

<pre><code>ConfigFactory.load().getConfig("com.mycompany")
</code></pre>

<p>Now I don't assemble the application.conf as part of my uber jar since I want to pass the file as an external file</p>

<p>The content of the external application.conf I want to use looks like this:</p>

<pre><code>com.mycompany {
  //configurations my program needs
}
</code></pre>

<p>This application.conf file exists on my local machine file system (and not on HDFS)</p>

<p>I'm using Spark 1.6.1 with Yarn</p>

<p>This is how my spark-submit command looks like:</p>

<pre><code>LOG4J_FULL_PATH=/log4j-path
ROOT_DIR=/application.conf-path

    /opt/deploy/spark/bin/spark-submit \
    --class com.mycompany.Main \
    --master yarn \
    --deploy-mode cluster \
    --files $ROOT_DIR/application.conf \
    --files $LOG4J_FULL_PATH/log4j.xml \
    --conf spark.executor.extraClassPath="-Dconfig.file=file:application.conf" \
    --driver-class-path $ROOT_DIR/application.conf \
    --verbose \
    /opt/deploy/lal-ml.jar
</code></pre>

<p>The exception I receive is:</p>

<pre><code>2016-11-09 12:32:14 ERROR ApplicationMaster:95 - User class threw exception: com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'com'
com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'com'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
    at com.typesafe.config.impl.SimpleConfig.getObject(SimpleConfig.java:218)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:224)
    at com.typesafe.config.impl.SimpleConfig.getConfig(SimpleConfig.java:33)
    at com.mycompany.Main$.main(Main.scala:36)
    at com.mycompany.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
</code></pre>

<p>And so my question is: does anybody know how I can load an external typesafe application.conf file that sit on my local machine with spark-submit and yarn?</p>

<p>I tried following some of the solutions in <a href="https://stackoverflow.com/questions/35509510/how-to-add-a-typesafe-config-file-which-is-located-on-hdfs-to-spark-submit-clus">How to add a typesafe config file which is located on HDFS to spark-submit (cluster-mode)?</a> and in <a href="https://stackoverflow.com/questions/38068927/typesafe-config-in-spark">Typesafe Config in Spark</a> and also in <a href="https://stackoverflow.com/questions/28166667/how-to-pass-d-parameter-or-environment-variable-to-spark-job">How to pass -D parameter or environment variable to Spark job?</a> and nothing worked</p>

<p>I'll appreciate any direction to solving this</p>

<p>Thanks in advance</p>