{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3565","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3565/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3565/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3565/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/3565","id":54378325,"number":3565,"title":"Detect low memory conditions and raise cluster issue","user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/41","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/41/labels","id":367201,"number":41,"title":"subsequent","description":"","creator":{"login":"coffeemug","id":48436,"avatar_url":"https://avatars.githubusercontent.com/u/48436?v=3","gravatar_id":"","url":"https://api.github.com/users/coffeemug","html_url":"https://github.com/coffeemug","followers_url":"https://api.github.com/users/coffeemug/followers","following_url":"https://api.github.com/users/coffeemug/following{/other_user}","gists_url":"https://api.github.com/users/coffeemug/gists{/gist_id}","starred_url":"https://api.github.com/users/coffeemug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coffeemug/subscriptions","organizations_url":"https://api.github.com/users/coffeemug/orgs","repos_url":"https://api.github.com/users/coffeemug/repos","events_url":"https://api.github.com/users/coffeemug/events{/privacy}","received_events_url":"https://api.github.com/users/coffeemug/received_events","type":"User","site_admin":false},"open_issues":305,"closed_issues":1,"state":"open","created_at":"2013-06-30T07:32:52Z","updated_at":"2015-01-14T21:21:43Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-01-14T21:21:43Z","updated_at":"2015-01-14T23:36:08Z","closed_at":null,"body":"This had been briefly discussed in https://github.com/rethinkdb/rethinkdb/issues/1114 before.\r\n\r\nWe're still getting reports from users who think RethinkDB is crashing, but it's actually running out of memory and gets terminated by the OOM killer.\r\n\r\nWhile we cannot solve the problem completely, we can do some things to make users better aware of the issue:\r\n* Monitor the amount of free RAM on the server (ignoring i/o caches etc)\r\n* If it goes below a certain threshold:\r\n  * create a cluster issue that shows up on the web UI\r\n  * print a message to the log\r\n\r\nOf course there can be situations where memory runs out before the threshold condition gets triggered. However this method would detect cases where memory usage grows relatively steadily and inform users about what's going on."},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/70014652","html_url":"https://github.com/rethinkdb/rethinkdb/issues/3565#issuecomment-70014652","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/3565","id":70014652,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2015-01-14T23:36:08Z","updated_at":"2015-01-14T23:36:08Z","body":"Also I think auto-resizing the cache can be a possible second step, but not a replacement for showing a warning."}}