{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4474","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4474/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4474/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4474/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4474","id":91970009,"number":4474,"title":"Limit backfill memory usage when loading large values","user":{"login":"timmaxw","id":316661,"avatar_url":"https://avatars.githubusercontent.com/u/316661?v=3","gravatar_id":"","url":"https://api.github.com/users/timmaxw","html_url":"https://github.com/timmaxw","followers_url":"https://api.github.com/users/timmaxw/followers","following_url":"https://api.github.com/users/timmaxw/following{/other_user}","gists_url":"https://api.github.com/users/timmaxw/gists{/gist_id}","starred_url":"https://api.github.com/users/timmaxw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timmaxw/subscriptions","organizations_url":"https://api.github.com/users/timmaxw/orgs","repos_url":"https://api.github.com/users/timmaxw/repos","events_url":"https://api.github.com/users/timmaxw/events{/privacy}","received_events_url":"https://api.github.com/users/timmaxw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/st:review","name":"st:review","color":"fbda04"}],"state":"open","locked":false,"assignee":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/95","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/95/labels","id":1006506,"number":95,"title":"2.1","description":"To-do list before shipping auto-failover","creator":{"login":"timmaxw","id":316661,"avatar_url":"https://avatars.githubusercontent.com/u/316661?v=3","gravatar_id":"","url":"https://api.github.com/users/timmaxw","html_url":"https://github.com/timmaxw","followers_url":"https://api.github.com/users/timmaxw/followers","following_url":"https://api.github.com/users/timmaxw/following{/other_user}","gists_url":"https://api.github.com/users/timmaxw/gists{/gist_id}","starred_url":"https://api.github.com/users/timmaxw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timmaxw/subscriptions","organizations_url":"https://api.github.com/users/timmaxw/orgs","repos_url":"https://api.github.com/users/timmaxw/repos","events_url":"https://api.github.com/users/timmaxw/events{/privacy}","received_events_url":"https://api.github.com/users/timmaxw/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":177,"state":"open","created_at":"2015-03-04T21:09:42Z","updated_at":"2015-07-29T23:56:53Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-06-30T03:48:17Z","updated_at":"2015-07-30T03:25:04Z","closed_at":null,"body":"The new backfilling algorithm loads up to 16 leaf nodes at a time from disk. With 16MB values, that would be multiple tens of gigabytes of RAM usage per backfill. The solution is twofold:\r\n\r\n1. Apply the item chunk size limit as we load values from disk. That probably means that `backfill_item_consumer_t` should have a separate method like `acquire_semaphore()` that gets called before `on_item()`. Before loading each value, the B-tree traversal code would try `acquire_semaphore()`.\r\n\r\n2. If `acquire_semaphore()` returns `false` while we're loading a multi-key backfill item, then we should stop loading the item at that point and set its right bound to just before the key we were about to load, then pass on the incomplete item normally. This allows us to break large items into smaller chunks.\r\n\r\nIf this is done right, the memory usage per-backfill should be limited to approximately the value of `item_queue_mem_size` plus the size of the largest value in the B-tree."},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/126169912","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4474#issuecomment-126169912","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4474","id":126169912,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2015-07-30T03:25:04Z","updated_at":"2015-07-30T03:25:04Z","body":"In CR 3108 by @timmaxw "}}