How does Spark achieve parallelism within one task on multi-core or hyper-threaded machines
<p>I have been reading and trying to understand how does Spark framework use its cores in <strong>Standalone</strong> mode. According to Spark documentation, the parameter  "<strong>spark.task.cpus</strong>"'s value is set to be 1 by default, which means number of cores to allocate for each task. </p>

<p><strong>Question 1:</strong>
For a multi-core machine (e.g., 4 cores in total, 8 hardware threads), when "spark.task.cpus = 4", will Spark use 4 cores (1 thread per core) or 2 cores with hyper-thread? </p>

<p>What will it happen if I set "spark.task.cpus = 16", more than the number of available hardware threads on this machine?</p>

<p><strong>Question 2:</strong> 
How is this type of hardware parallelism achieved? I tried to look into the code but couldn't find anything that communicates with the hardware or JVM for core-level parallelism. For example, if the task is "filter" function, how is a single filter task spitted to multiple cores or threads? </p>

<p>Maybe I am missing something. Is this related to the Scala language? </p>