{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4808","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4808/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4808/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4808/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4808","id":105438044,"number":4808,"title":"Crash while building secondary index","user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/pr:high","name":"pr:high","color":"e10c02"},{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"open","locked":false,"assignee":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1.x","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104/labels","id":1249551,"number":104,"title":"2.1.x","description":null,"creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":42,"closed_issues":0,"state":"open","created_at":"2015-08-11T07:18:11Z","updated_at":"2015-09-16T00:17:04Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2015-09-08T18:04:42Z","updated_at":"2015-09-16T22:23:20Z","closed_at":null,"body":"Brad Gardner reported this on the mailing list: https://groups.google.com/forum/#!topic/rethinkdb/-BgbC473KgI\r\n\r\nHe tried to migrate indexes for a table, but at some point during the index construction one of his two servers got stuck and wouldn't reply to queries anymore. He had to kill and restart it.\r\nThis seems to happen every time for the table.\r\n\r\nThere are some indications of memory exhaustion. However\r\n- There shouldn't be a reason for index construction to use a lot of memory (unless there are a lot of writes to the table going on while construction is running maybe?)\r\n- The mode of crash, with the server just hanging, isn't typical for an out-of-memory issue. Unless maybe it went into swap...\r\n\r\nI'm somewhat concerned that there might have been a table corruption at some point in the past, that's now showing up (https://github.com/rethinkdb/rethinkdb/issues/4769 maybe?).\r\n\r\nI'll ask Brad to provide some additional info on this issue."},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/140911379","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4808#issuecomment-140911379","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4808","id":140911379,"user":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T22:23:20Z","updated_at":"2015-09-16T22:23:20Z","body":"Quick update for other readers on this issue: We've worked together to track down this issue, and so far it looks a lot like memory consumption is simply too high. @bgardner87 reduced the cache size on the servers which appears to have helped.\r\n\r\nPart of the memory overhead stems from https://github.com/rethinkdb/rethinkdb/issues/1951 , which we are working on fixing."}}