Mongodb Out of Memory while total DB size < available RAM
<p>I have read what I could find on memory consumption for MongoDB, but the gist of what I understood was that everything was handled by the OS, and if no memory is available the data will be read from disk, then replace something else in memory.</p>

<p>I have a pretty small database </p>

<pre><code>    &gt; db.stats()
    {
            "db" : "prod",
            "collections" : 11,
            "objects" : 2022,
            "avgObjSize" : 43469.34915924827,
            "dataSize" : 87895024,
            "storageSize" : 113283072,
            "numExtents" : 30,
            "indexes" : 10,
            "indexSize" : 4840192,
            "fileSize" : 201326592,
            "nsSizeMB" : 16,
            "extentFreeList" : {
                    "num" : 0,
                    "totalSize" : 0
            },
            "dataFileVersion" : {
                    "major" : 4,
                    "minor" : 22
            },
            "ok" : 1
    }
</code></pre>

<p>with a small server with 1GB of RAM. Seeing the size of the DB (~100MB), I would assume 1GB of RAM should be plenty.</p>

<p>I have however been having Out of Memory errors for some time now, first infrequently (one every 2-3 weeks), and almost twice a day now.</p>

<p>I'm at a loss as to what could cause these issues, and thought that I may be missing something completely.</p>

<p>I ran what diagnosis I found on the net:</p>

<ul>
<li><p>ulimit   </p>

<pre><code>$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7826
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7826
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
</code></pre></li>
<li><p>mongod version is 3.0.12</p></li>
<li><p>OS info:</p>

<pre><code>NAME="Amazon Linux AMI"
VERSION="2015.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2015.03"
PRETTY_NAME="Amazon Linux AMI 2015.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2015.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
Amazon Linux AMI release 2015.03
</code></pre></li>
<li><p>db.serverStatus() is on <a href="http://pastebin.com/6698KJbW" rel="nofollow">pastebin</a>. Edit: looking at <a href="https://docs.mongodb.com/manual/reference/command/serverStatus/#memory-status" rel="nofollow">https://docs.mongodb.com/manual/reference/command/serverStatus/#memory-status</a> </p>

<blockquote>
  <p>If mem.virtual value is significantly larger than mem.mapped (e.g. 3 or more times), this may indicate a memory leak. </p>
</blockquote>

<p>So maybe something to look at</p></li>
<li><p>Swap is enabled</p></li>
<li><p><code>free -m</code> execution before launching MongoDB</p>

<pre><code>$ free -m
             total       used       free     shared    buffers     cached
Mem:           996         60        935          0          5         19
-/+ buffers/cache:         36        959
Swap:         4095          9       4086
</code></pre>

<p>and right after launch (launching + running the command immediately)</p>

<pre><code>$ free -m
             total       used       free     shared    buffers     cached
Mem:           996        925         71          0          5        834
-/+ buffers/cache:         84        911
Swap:         4095          9       4086
</code></pre></li>
<li><p><code>mongostat</code> run</p>

<pre><code>$ mongostat
insert query update delete getmore command flushes mapped vsize    res faults qr|qw ar|aw netIn netOut conn     time
    *0    *0     *0     *0       0     1|0       0 688.0M  7.6G 764.0M      0   0|0   0|0   79b    10k    9 07:04:55
    *0    *0     *0     *0       0     1|0       0 688.0M  7.6G 764.0M      0   0|0   0|0   79b    10k    9 07:04:56
    *0    *0     *0     *0       0     3|0       0 688.0M  7.6G 764.0M      0   0|0   0|0  196b    11k    9 07:04:57
    *0    *0     *0     *0       0     1|0       0 688.0M  7.6G 764.0M      0   0|0   0|0   79b    10k    9 07:04:58
    *0    *0     *0     *0       0     2|0       0 688.0M  7.6G 764.0M      0   0|0   0|0  133b    10k    9 07:04:59
    *0    *0     *0     *0       0     1|0       0 688.0M  7.6G 764.0M      0   0|0   0|0   79b    10k    9 07:05:00
</code></pre></li>
<li><p>Running <code>mongostat</code> a few hours later show an increase in res memory (Edit: re-running serverStatus() <a href="http://pastebin.com/jvWyAS1j" rel="nofollow">shows no increase in mem.resident</a> though)</p>

<pre><code>$ mongostat
insert query update delete getmore command flushes mapped vsize    res faults qr|qw ar|aw netIn netOut conn     time
    *0    *0     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      8   0|0   0|0   79b    10k    8 10:39:50
    *0    *0     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      0   0|0   0|0   79b    10k    8 10:39:51
    *0    *0     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      0   0|0   0|0   79b    10k    8 10:39:52
    *0    *0     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      0   0|0   0|0   79b    10k    8 10:39:53
    *0    *0     *0     *0       0     4|0       0 688.0M  7.7G 856.0M      0   0|0   0|0  250b    11k    8 10:39:54
    *0     2     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      0   0|0   0|0  183b    11k    8 10:39:55
    *0     1     *0     *0       0     1|0       0 688.0M  7.7G 856.0M      0   0|0   0|0  131b    11k    8 10:39:56
</code></pre></li>
<li><p><code>swapon -s</code></p>

<pre><code>$ swapon -s
Filename                                Type            Size    Used    Priority
/swapfile                               file    4194300 10344   -1
</code></pre></li>
</ul>

<p>Edit: I've setup MongoDB Cloud Monitoring, and the issue just reoccurred. <a href="http://imgur.com/LWscx29" rel="nofollow">This is the report</a>, and the mongo process was killed at 02:29</p>

<p>Do you have any idea what may be causing the issue? Or hints into where I should look?</p>

<p>Thanks for your help!</p>

<p>Seb</p>