Apache Spark: Differences between client and cluster deploy modes
<p><strong>TL;DR:</strong> In a Spark Standalone cluster, what are the differences between client and cluster deploy modes? How do I set which mode my application is going to run on?</p>

<hr>

<p>We have a Spark Standalone cluster with three machines, all of them with Spark 1.6.1:</p>

<ul>
<li>A master machine, which also is where our application is run using <code>spark-submit</code></li>
<li>2 identical worker machines</li>
</ul>

<p>From the <a href="http://spark.apache.org/docs/latest/spark-standalone.html" rel="noreferrer" title="Spark Standalone Mode">Spark Documentation</a>, I read:</p>

<blockquote>
  <p>(...) For standalone clusters, Spark currently supports two deploy modes. In client mode, the driver is launched in the same process as the client that submits the application. In cluster mode, however, the driver is launched from one of the Worker processes inside the cluster, and the client process exits as soon as it fulfills its responsibility of submitting the application without waiting for the application to finish.</p>
</blockquote>

<p>However, I don't really understand the practical differences by reading this, and I don't get what are the advantages and disadvantages of the different deploy modes. </p>

<p>Additionally, when I start my application using start-submit, even if I set the property <code>spark.submit.deployMode</code> to "cluster", the Spark UI for my context shows the following entry:</p>

<p><a href="https://i.stack.imgur.com/VpP3d.png" rel="noreferrer"><img src="https://i.stack.imgur.com/VpP3d.png" alt="Context UI"></a></p>

<p>So I am not able to test both modes to see the practical differences. That being said, my questions are:</p>

<p>1) What are the practical differences between Spark Standalone <em>client</em> deploy mode and <em>cluster</em> deploy mode? What are the pro's and con's of using each one?</p>

<p>2) How to I choose which one my application is going to be running on, using <code>spark-submit</code>?</p>