Using Python multiprocessing on an HPC cluster
<p>I am running a Python script on a Windows HPC cluster. A function in the script uses <code>starmap</code> from the <code>multiprocessing</code> package to parallelize a certain computationally intensive process.</p>

<p>When I run the script on a single non-cluster machine, I obtain the expected speed boost. When I log into a node and run the script locally, I obtain the expected speed boost. However, when the job manager runs the script, the speed boost from <code>multiprocessing</code> is either completely mitigated or, sometimes, even 2x slower. We have noticed that memory paging occurs when the <code>starmap</code> function is called. We believe that this has something to do with the nature of Python's <code>multiprocessing</code>, i.e. the fact that a separate Python interpreter is kicked off for each core.</p>

<p>Since we had success running from the console from a single node, we tried to run the script with <code>HPC_CREATECONSOLE=True</code>, to no avail.</p>

<p>Is there some kind of setting within the job manager that we should use when running Python scripts that use <code>multiprocessing</code>? Is <code>multiprocessing</code> just not appropriate for an HPC cluster?</p>