oozie spark action - how to specify spark-opts
<p>I am running spark job in yarn-client mode via oozie spark action. I need to specify driver and application master related settings. I tried configuring spark-opts as documented by oozie but its not working.</p>

<p>Here's from oozie doc:</p>

<pre><code>Example:

&lt;workflow-app name="sample-wf" xmlns="uri:oozie:workflow:0.1"&gt;
    ...
    &lt;action name="myfirstsparkjob"&gt;
        &lt;spark xmlns="uri:oozie:spark-action:0.1"&gt;
            &lt;job-tracker&gt;foo:8021&lt;/job-tracker&gt;
            &lt;name-node&gt;bar:8020&lt;/name-node&gt;
            &lt;prepare&gt;
                &lt;delete path="${jobOutput}"/&gt;
            &lt;/prepare&gt;
            &lt;configuration&gt;
                &lt;property&gt;
                    &lt;name&gt;mapred.compress.map.output&lt;/name&gt;
                    &lt;value&gt;true&lt;/value&gt;
                &lt;/property&gt;
            &lt;/configuration&gt;
            &lt;master&gt;local[*]&lt;/master&gt;
            &lt;mode&gt;client&lt;mode&gt;
            &lt;name&gt;Spark Example&lt;/name&gt;
            &lt;class&gt;org.apache.spark.examples.mllib.JavaALS&lt;/class&gt;
            &lt;jar&gt;/lib/spark-examples_2.10-1.1.0.jar&lt;/jar&gt;
            &lt;spark-opts&gt;--executor-memory 20G --num-executors 50&lt;/spark-opts&gt;
            &lt;arg&gt;inputpath=hdfs://localhost/input/file.txt&lt;/arg&gt;
            &lt;arg&gt;value=2&lt;/arg&gt;
        &lt;/spark&gt;
        &lt;ok to="myotherjob"/&gt;
        &lt;error to="errorcleanup"/&gt;
    &lt;/action&gt;
    ...
&lt;/workflow-app&gt;
</code></pre>

<p>In above spark-opts are specified as <code>--executor-memory 20G --num-executors 50</code></p>

<p>while on the same page in description it says:</p>

<p>"The spark-opts element if present, contains a list of spark options that can be passed to spark driver. Spark configuration options can be passed by specifying '--conf key=value' here"</p>

<p>so according to document it should be <code>--conf executor-memory=20G</code> </p>

<p>which one is right here then? I tried both but it's not seem working. I am running on yarn-client mode so mainly want to setup driver related settings. I think this is the only place I can setup driver settings. </p>

<pre><code>&lt;spark-opts&gt;--driver-memory 10g --driver-java-options "-XX:+UseCompressedOops -verbose:gc" --conf spark.driver.memory=10g --conf spark.yarn.am.memory=2g --conf spark.driver.maxResultSize=10g&lt;/spark-opts&gt;

&lt;spark-opts&gt;--driver-memory 10g&lt;/spark-opts&gt;
</code></pre>

<p>None of the above driver related settings getting set in actual driver jvm. I verified it on linux process info.</p>

<p>reference: <a href="https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html" rel="nofollow">https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html</a></p>