{"action":"created","issue":{"url":"https://api.github.com/repos/deis/deis/issues/4618","labels_url":"https://api.github.com/repos/deis/deis/issues/4618/labels{/name}","comments_url":"https://api.github.com/repos/deis/deis/issues/4618/comments","events_url":"https://api.github.com/repos/deis/deis/issues/4618/events","html_url":"https://github.com/deis/deis/issues/4618","id":111571848,"number":4618,"title":"HELP! deis-store-daemon high memory utilisation","user":{"login":"synaptics","id":1358381,"avatar_url":"https://avatars.githubusercontent.com/u/1358381?v=3","gravatar_id":"","url":"https://api.github.com/users/synaptics","html_url":"https://github.com/synaptics","followers_url":"https://api.github.com/users/synaptics/followers","following_url":"https://api.github.com/users/synaptics/following{/other_user}","gists_url":"https://api.github.com/users/synaptics/gists{/gist_id}","starred_url":"https://api.github.com/users/synaptics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/synaptics/subscriptions","organizations_url":"https://api.github.com/users/synaptics/orgs","repos_url":"https://api.github.com/users/synaptics/repos","events_url":"https://api.github.com/users/synaptics/events{/privacy}","received_events_url":"https://api.github.com/users/synaptics/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-15T08:52:41Z","updated_at":"2015-10-15T15:23:21Z","closed_at":null,"body":"Hi,\r\n\r\nI have a 3 node cluster (deis 1.9) launched through deis.com.  Everything is mostly good, apart from about once every 3 weeks or so one of the nodes seems to die and generally has trouble coming back up without a bit of manual intervention.  Unfortunately I don't generally get time to retrospectively debug the issue, but I've been suspecting that it's a memory issue so I've been keeping a closer eye on it recently...\r\n\r\nToday one of my nodes is showing this (the box is an AWS M3 Large):\r\n\r\nCONTAINER           CPU %               MEM USAGE/LIMIT       MEM %               NET I/O\r\ndeis-logspout       0.09%               10.06 MiB/7.308 GiB   0.13%               89.95 MiB/86.96 MiB\r\ndeis-publisher      0.11%               10.73 MiB/7.308 GiB   0.14%               4.359 KiB/738 B\r\ndeis-router         0.00%               29.77 MiB/7.308 GiB   0.40%               3.442 GiB/1.036 GiB\r\ndeis-store-admin    0.08%               13.02 MiB/7.308 GiB   0.17%               769.2 MiB/192.9 MiB\r\ndeis-store-daemon   0.03%               5.007 GiB/7.308 GiB   68.52%              0 B/0 B\r\nlogstash_v6.cmd.1   0.06%               370.5 MiB/7.308 GiB   4.95%               59.32 MiB/64.57 MiB\r\n\r\nAs you can see the box is only running the Deis platform and a logstash container, but deis-store-daemon is using 5GB of 7GB available?!  Why is this so high on this one node?  Can I just restart the daemon and this will sort itself out?  Can i somehow limit this behaviour?  Can I just restart the service and it will resolve itself?\r\n\r\nFYI the other 2 nodes in the cluster are running at this (much more reasonable) level:\r\n\r\ndeis-store-daemon         0.00%               520.8 MiB/7.308 GiB   6.96%               0 B/0 B\r\ndeis-store-daemon         0.06%               577.4 MiB/7.308 GiB   7.72%               0 B/0 B\r\n\r\nThanks,\r\nJohn"},"comment":{"url":"https://api.github.com/repos/deis/deis/issues/comments/148420574","html_url":"https://github.com/deis/deis/issues/4618#issuecomment-148420574","issue_url":"https://api.github.com/repos/deis/deis/issues/4618","id":148420574,"user":{"login":"bacongobbler","id":1360539,"avatar_url":"https://avatars.githubusercontent.com/u/1360539?v=3","gravatar_id":"","url":"https://api.github.com/users/bacongobbler","html_url":"https://github.com/bacongobbler","followers_url":"https://api.github.com/users/bacongobbler/followers","following_url":"https://api.github.com/users/bacongobbler/following{/other_user}","gists_url":"https://api.github.com/users/bacongobbler/gists{/gist_id}","starred_url":"https://api.github.com/users/bacongobbler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bacongobbler/subscriptions","organizations_url":"https://api.github.com/users/bacongobbler/orgs","repos_url":"https://api.github.com/users/bacongobbler/repos","events_url":"https://api.github.com/users/bacongobbler/events{/privacy}","received_events_url":"https://api.github.com/users/bacongobbler/received_events","type":"User","site_admin":false},"created_at":"2015-10-15T15:23:21Z","updated_at":"2015-10-15T15:23:21Z","body":"@synaptics would you mind getting some logs from the deis-store-monitor container running on that host? That certainly is a high amount of memory usage. You could try restarting the daemon but it'd be better to get to the bottom of this for educational purposes."}}