Python multiprocessing unexpected high memory usage
<p>In our system we have started to experience problems with a task that was working ok, but now it seems to be hanging and uses a high amount of memory (so other tasks fail and raise MemoryError).</p>

<p>Context: example code below had no problem, but for a new database <em>huge_dataframe</em> became a lot bigger. The question is that if I run both parts separately it works. But running all together in <em>process_data_task</em> will lead to the problem. Running under Python 2.7 on Linux.</p>

<p>I suspect that is something with fork(), but how could each sub-process take so much memory? <em>huge_dataframe</em> is deleted before starting multiprocessing. Also is strange that <em>do_recalculations</em> hangs only when is called inside <em>process_data_task</em> (child not joining?), but it doesn't throw any exception. Any explanation or idea for troubleshooting? </p>

<pre><code>def process_data_task():
    # part 1, high memory usage
    huge_dataframe = retrieve_data()
    process_table(huge_dataframe)

    # added this trying to fix the problem but it didn't help
    del huge_dataframe
    gc.collect()

    # part 2, heavy CPU usage, multiprocessing
    do_recalculations()  # recalculate items in parallel

# Multiprocessing done here
def do_recalculations():
     processes = cpu_count()
     items_to_update = [...]  # query database
     work_total_list = chunkify(items_to_update, processes)
     p = Pool(processes)
     result = p.map(sub_process_func, work_total_list)
     p.close()
     p.join()
</code></pre>