Using Concurrent Futures without running out of RAM
<p>I'm doing some file parsing that is a CPU bound task. No matter how many files I throw at the process it uses no more than about 50MB of RAM.
The task is parrallelisable, and I've set it up to use concurrent futures below to parse each file as a separate process:</p>

<pre><code>    from concurrent import futures
    with futures.ProcessPoolExecutor(max_workers=6) as executor:
        # A dictionary which will contain a list the future info in the key, and the filename in the value
        jobs = {}

        # Loop through the files, and run the parse function for each file, sending the file-name to it.
        # The results of can come back in any order.
        for this_file in files_list:
            job = executor.submit(parse_function, this_file, **parser_variables)
            jobs[job] = this_file

        # Get the completed jobs whenever they are done
        for job in futures.as_completed(jobs):

            # Send the result of the file the job is based on (jobs[job]) and the job (job.result)
            results_list = job.result()
            this_file = jobs[job]

            # delete the result from the dict as we don't need to store it.
            del jobs[job]

            # post-processing (putting the results into a database)
            post_process(this_file, results_list)
</code></pre>

<p>The problem is that when I run this using futures, RAM usage rockets and before long I've run out and Python has crashed. This is probably in large part because the results from parse_function are several MB in size. Once the results have been through <code>post_processing</code>, the application has no further need of them. As you can see, I'm trying <code>del jobs[job]</code> to clear items out of <code>jobs</code>, but this has made no difference, memory usage remains unchanged, and seems to increase at the same rate. </p>

<p>I've also confirmed it's not because it's waiting on the <code>post_process</code> function by only using a single process, plus throwing in a <code>time.sleep(1)</code>.</p>

<p>There's nothing in the futures docs about memory management, and while a brief search indicates it has come up before in real-world applications of futures (<a href="https://stackoverflow.com/questions/31720674/clear-memory-in-python-loop">Clear memory in python loop</a> and <a href="http://grokbase.com/t/python/python-list/1458ss5etz/real-world-use-of-concurrent-futures" rel="noreferrer">http://grokbase.com/t/python/python-list/1458ss5etz/real-world-use-of-concurrent-futures</a>) - the answers don't translate to my use-case (they're all concerned with timeouts and the like).</p>

<p>So, how do you use Concurrent futures without running out of RAM?
(Python 3.5)</p>