How to profile multiple subprocesses using Python multiprocessing and memory_profiler?
<p>I have a utility that spawns multiple workers using the Python <a href="https://docs.python.org/3.5/library/multiprocessing.html" rel="noreferrer"><code>multiprocessing</code></a> module, and I'd like to be able to track their memory usage via the excellent <a href="https://pypi.python.org/pypi/memory_profiler" rel="noreferrer"><code>memory_profiler</code></a> utility, which does everything I want - particularly sampling memory usage over time and plotting the final result (I'm not concerned with the line-by-line memory profiling for this question). </p>

<p>In order to setup this question, I have created a simpler version of the script, that has a worker function which allocates memory similar to the <a href="https://github.com/fabianp/memory_profiler/blob/master/examples/example.py" rel="noreferrer">example</a> given in the <code>memory_profiler</code> library. The worker is as follows:</p>

<pre><code>import time

X6 = 10 ** 6
X7 = 10 ** 7

def worker(num, wait, amt=X6):
    """
    A function that allocates memory over time.
    """
    frame = []

    for idx in range(num):
        frame.extend([1] * amt)
        time.sleep(wait)

    del frame
</code></pre>

<p>Given a sequential workload of 4 workers as follows: </p>

<pre><code>if __name__ == '__main__':
    worker(5, 5, X6)
    worker(5, 2, X7)
    worker(5, 5, X6)
    worker(5, 2, X7)
</code></pre>

<p>Running the <code>mprof</code> executable to profile my script takes 70 seconds having each worker run one after the other. The script, run as follows: </p>

<pre><code>$ mprof run python myscript.py
</code></pre>

<p>Produces the following memory usage graph:</p>

<p><a href="https://i.stack.imgur.com/Yq7My.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Yq7My.png" alt="Sequential Memory Generating Workers"></a></p>

<p>Having these workers go in parallel with <code>multiprocessing</code> means that the script will finish as slow as the slowest worker (25 seconds). That script is as follows:</p>

<pre><code>import multiprocessing as mp

if __name__ == '__main__':
    pool    = mp.Pool(processes=4)
    tasks   = [
        pool.apply_async(worker, args) for args in
        [(5, 5, X6), (5, 2, X7), (5, 5, X6), (5, 2, X7)]
    ]

    results = [p.get() for p in tasks]
</code></pre>

<p>Memory profiler does indeed work, or at least there are no errors when using <code>mprof</code> but the results are a bit strange:</p>

<p><a href="https://i.stack.imgur.com/6Lb7l.png" rel="noreferrer"><img src="https://i.stack.imgur.com/6Lb7l.png" alt="enter image description here"></a></p>

<p>A quick look at Activity Monitor shows that in fact there are 6 Python processes, one for <code>mprof</code> one for <code>python myscript.py</code> and then one for each worker subprocess. It appears that <code>mprof</code> is only measuring the memory usage for the <code>python myscript.py</code> process. </p>

<p><a href="https://i.stack.imgur.com/6R13j.png" rel="noreferrer"><img src="https://i.stack.imgur.com/6R13j.png" alt="Python Processes in Activity Monitor"></a></p>

<p>The <code>memory_profiler</code> library is highly customizable, and I'm pretty confident that I should be able to capture the memory of each process and possibly write them out to separate log files by using the library itself. I'm just not sure where to begin or how to approach that level of customization.</p>

<p><strong>EDIT</strong> </p>

<p>After reading through the <code>mprof</code> script I did discover the <code>-C</code> flag which sums up the memory usage of all child (forked) processes. This leads to a (much improved) graph as follows:</p>

<p><a href="https://i.stack.imgur.com/FwBUn.png" rel="noreferrer"><img src="https://i.stack.imgur.com/FwBUn.png" alt="Multiprocessing Workers with Include Children Flag"></a></p>

<p>But what I'm looking for is the memory usage of each individual subprocess over time so that I can plot all workers (and the master) on the same graph. My idea is to have each subprocess <code>memory_usage</code> written to a different log file, which I can then visualize. </p>