inserting millions of documents - mongo / pymongo - insert_many
<p>New to mongo/pymongo.
Currently using the latest - v3.2.2</p>

<p>It looks as if insert_many is not performing as intended?  I've noticed that even when supplying a generator to db.col.insert_many, memory usage still spikes (which makes inserting millions of documents difficult - though I do realize that sys.mem should be > collection size for best performance, so in reality perhaps this is nothing I should worry about?</p>

<p>I was under the impression that if you pass a generator to insert_many that pymongo will 'buffer' the insert into 16 or 32mb 'chunks'?</p>

<p>Performing this buffering/chunking manually solves the issue...</p>

<p>See below:</p>

<p>Example1 = straight insert_many (high memory usage - 2.625 GB)</p>

<p>Example2 = 'buffered' insert_many (expected [low] memory usage - ~300 MB)</p>

<pre><code>import itertools
from itertools import chain,islice
import pymongo

client = pymongo.MongoClient()
db=client['test']

def generate_kv(N):
    for i in range(N):
        yield {'x': i}

print "example 1"
db.testcol.drop()
db.testcol.insert_many(generate_kv(5000000))

def chunks(iterable, size=10000):
        iterator = iter(iterable)
        for first in iterator:
            yield chain([first], islice(iterator, size - 1))

print "example 2"
db.testcol.drop()
for c in chunks(generate_kv(5000000)):
        db.testcol.insert_many(c)
</code></pre>

<p>Any ideas?  Bug? Am I using this wrong?</p>