How do I get Spark installed on Amazon's EMR core/worker nodes/instances while creating the cluster?
<p>I am trying to launch a EMR cluster with Spark (1.6.0) and Hadoop (Distribution: Amazon 2.7.1) applications. The release label is <strong>emr-4.4.0</strong>.
The cluster gets setup as needed but it doesn't run Spark master (in the master instances) as a daemon process and also I cannot find Spark being installed in the worker (core) instances (the Spark dir under <strong><em>/usr/lib/</em></strong> has just <strong><em>lib</em></strong> and <strong><em>yarn</em></strong> directories). </p>

<p>I'd like to run the Spark master and worker nodes as soon as the cluster has been setup. (i.e., workers connect to the master automatically and become a part of the Spark cluster). </p>

<p>How do I achieve this? Or am I missing something?</p>

<p>Thanks in advance!</p>