Why there are more python processes than the number of cores in Spark?
<p>I am running a pyspark job in ipython console. I set spark master as <code>local[4]</code>. So I expect one core for the driver, which should be a Java process, and the other three cores each runs a Python process. However, this the screen shot of my <code>top</code>,</p>
<p><a href="https://i.stack.imgur.com/b7qQj.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b7qQj.png" alt="enter image description here" /></a></p>
<p>Why there are 16 python processes? Why isn't there only 3 python processes? If I remember right, 16 is the total number of cores on this server.</p>