{"action":"created","issue":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/4079","labels_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/4079/labels{/name}","comments_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/4079/comments","events_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/4079/events","html_url":"https://github.com/HabitRPG/habitrpg/issues/4079","id":43856006,"number":4079,"title":"Memory Leak","user":{"login":"lefnire","id":195202,"avatar_url":"https://avatars.githubusercontent.com/u/195202?v=3","gravatar_id":"","url":"https://api.github.com/users/lefnire","html_url":"https://github.com/lefnire","followers_url":"https://api.github.com/users/lefnire/followers","following_url":"https://api.github.com/users/lefnire/following{/other_user}","gists_url":"https://api.github.com/users/lefnire/gists{/gist_id}","starred_url":"https://api.github.com/users/lefnire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lefnire/subscriptions","organizations_url":"https://api.github.com/users/lefnire/orgs","repos_url":"https://api.github.com/users/lefnire/repos","events_url":"https://api.github.com/users/lefnire/events{/privacy}","received_events_url":"https://api.github.com/users/lefnire/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/bug","name":"bug","color":"e102d8"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/hammer","name":"hammer","color":"e11d21"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/Memorable","name":"Memorable","color":"207de5"},{"url":"https://api.github.com/repos/HabitRPG/habitrpg/labels/priority%20-%20critical","name":"priority - critical","color":"e10c02"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/milestones/4","html_url":"https://github.com/HabitRPG/habitrpg/milestones/Backend%20Perf/Scale","labels_url":"https://api.github.com/repos/HabitRPG/habitrpg/milestones/4/labels","id":967636,"number":4,"title":"Backend Perf/Scale","description":"Per Alys:\r\n\r\n>I am getting the impression that the frequency and maybe severity of weird glitches in people's data are increasing. I'm seeing more people with duplicated tasks, and just recently unexplained jumps in level or XP, and of course the \"2 versions\" bug https://github.com/HabitRPG/habitrpg/issues/4313 . Perhaps it's just because we have more users now, but I am worried that the site or database might be becoming less stable. I'm not sure where to start investigating this, and in fact haven't even had time to look into the localStorage problem properly.\r\n\r\nThe site's architecture is poor, causing frequent Dyno upscales and constant request queueing. This milestone will track the critical issues for improving backend scalability","creator":{"login":"lefnire","id":195202,"avatar_url":"https://avatars.githubusercontent.com/u/195202?v=3","gravatar_id":"","url":"https://api.github.com/users/lefnire","html_url":"https://github.com/lefnire","followers_url":"https://api.github.com/users/lefnire/followers","following_url":"https://api.github.com/users/lefnire/following{/other_user}","gists_url":"https://api.github.com/users/lefnire/gists{/gist_id}","starred_url":"https://api.github.com/users/lefnire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lefnire/subscriptions","organizations_url":"https://api.github.com/users/lefnire/orgs","repos_url":"https://api.github.com/users/lefnire/repos","events_url":"https://api.github.com/users/lefnire/events{/privacy}","received_events_url":"https://api.github.com/users/lefnire/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":1,"state":"open","created_at":"2015-02-05T19:57:02Z","updated_at":"2015-07-19T02:21:08Z","due_on":"2015-02-28T07:00:00Z","closed_at":null},"comments":21,"created_at":"2014-09-25T04:45:41Z","updated_at":"2015-07-19T02:21:08Z","closed_at":"2015-07-19T02:21:08Z","body":"After investigating recent server craziness, I've come to a couple points:\r\n\r\n1. Memory leak. Per [StrongLoop](http://strongloop.com/strongblog/node-js-performance-tip-of-the-week-memory-leak-diagnosis/), a steady incline in mem usage (vs the \"comb\" pattern) indicates a memleak. Viewing our [mem usage](https://dashboard-next.heroku.com/apps/habitrpg/metrics/web) (anyone who has heroku access, [screenshot for others](http://gyazo.com/315fac5dc9d33325018bbf94ea2ca619)) shows inclines before each crash/restart, a *very* likely candidate. Compare that with [NewRelic](https://rpm.newrelic.com/accounts/565514/applications/3386694) will give us deets about which txns / errors might be correlated. My eye is on spell-casting and all things Challenges. However, I haven't been able to find any concrete memleak evidence on local experiments. [Here are tools / resources for getting started](https://workflowy.com/s/14RdBB75p5). No leads yet - though I can almost guarantee we have a leak (per Heroku) - and those are the tools for anyone so inclined.\r\n2. We need to cut back on application code resource usage. NewRelic shows most (like 95%) of our perf bottlenecks on the application side, and Mongo seriously underutilized. To show what I mean, check out https://github.com/HabitRPG/habitrpg/commit/632726fe0273ad507138ae9c30a8a890659e8a0f which pushes all the filtering from the server to MongoDB (saving the server resources). The comments give details on why/what, but this is tip-of-the-iceberg regarding what we can fix-up in Habit's code by moving everything mongo-side. The biggest pain-point here is Challenges, where way too much is handled on the server instead of MongoDB; but there are other locations.\r\n\r\nAnyway, this is simply a brain-dump, I'll be in it deep in coming days, just posting here in case someone has insights\r\n\r\n\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/4755911-memory-leak?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"},"comment":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/comments/122615612","html_url":"https://github.com/HabitRPG/habitrpg/issues/4079#issuecomment-122615612","issue_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/4079","id":122615612,"user":{"login":"crookedneighbor","id":2916945,"avatar_url":"https://avatars.githubusercontent.com/u/2916945?v=3","gravatar_id":"","url":"https://api.github.com/users/crookedneighbor","html_url":"https://github.com/crookedneighbor","followers_url":"https://api.github.com/users/crookedneighbor/followers","following_url":"https://api.github.com/users/crookedneighbor/following{/other_user}","gists_url":"https://api.github.com/users/crookedneighbor/gists{/gist_id}","starred_url":"https://api.github.com/users/crookedneighbor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crookedneighbor/subscriptions","organizations_url":"https://api.github.com/users/crookedneighbor/orgs","repos_url":"https://api.github.com/users/crookedneighbor/repos","events_url":"https://api.github.com/users/crookedneighbor/events{/privacy}","received_events_url":"https://api.github.com/users/crookedneighbor/received_events","type":"User","site_admin":false},"created_at":"2015-07-19T02:21:08Z","updated_at":"2015-07-19T02:21:08Z","body":"I believe @gisikw fixed this. New Relic interacting with node was causing the memory leak"}}