Spark Yarn integration on Hortonworks Sandbox 2.4
<p>I'm running the Hortonworks sandbox 2.4 which comes with Spark 1.6.0. Trying to run a sample spark program and I'm able to submit it successfully with spark-submit. As I need to use this spark jar file in an external application, trying to use the "java -jar" command to achieve the same output as in spark-submit.</p>

<p>I'm using maven to build and used the maven shade plugin to build a fat jar as I was facing issues with the class not found exception earlier for the dependencies "spark-core_2.10" &amp; "spark-yarn_2.10" artifacts. Now those issues are resolved.</p>

<p>However, the jar file is referring to the yarn-default.xml that comes with the dependencies inside the fat jar and not using the yarn-site.xml present in the Hortonworks sandbox. This is causing issues as below as it is not able to copy the file into hdfs.</p>

<p>java.io.FileNotFoundException: File file:/tmp/spark-1b318406-c7a1-4a94-9605-d6a46f0170d4/__spark_conf__5199620712586647591.zip does not exist</p>

<p>How can I make this jar file point to the Hortonworks sandbox settings from this default ones? If I don't build the fat jar, it was throwing the exception as below.</p>

<p>Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function</p>