How does spark.python.worker.memory relate to spark.executor.memory?
<p><a href="http://0x0fff.com/wp-content/uploads/2015/03/Spark-Architecture-On-YARN.png" rel="noreferrer">This diagram</a> is quite clear on the relationship between the different YARN and Spark memory related settings, except when it comes to <code>spark.python.worker.memory</code>.</p>

<p>How does <code>spark.python.worker.memory</code> fit into this memory model?</p>

<p>Are the Python processes governed by <code>spark.executor.memory</code> or <code>yarn.nodemanager.resource.memory-mb</code>?</p>

<p><strong>Update</strong></p>

<p><a href="https://stackoverflow.com/questions/31565497/what-is-spark-python-worker-memory">This question</a> explains what the setting does, but doesn't answer the question concerning the memory governance, or how it relates to other memory settings.</p>