{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4678","id":100663966,"number":4678,"title":"Changefeed Crashing Bugs","user":{"login":"mlucy","id":1777134,"avatar_url":"https://avatars.githubusercontent.com/u/1777134?v=3","gravatar_id":"","url":"https://api.github.com/users/mlucy","html_url":"https://github.com/mlucy","followers_url":"https://api.github.com/users/mlucy/followers","following_url":"https://api.github.com/users/mlucy/following{/other_user}","gists_url":"https://api.github.com/users/mlucy/gists{/gist_id}","starred_url":"https://api.github.com/users/mlucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlucy/subscriptions","organizations_url":"https://api.github.com/users/mlucy/orgs","repos_url":"https://api.github.com/users/mlucy/repos","events_url":"https://api.github.com/users/mlucy/events{/privacy}","received_events_url":"https://api.github.com/users/mlucy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/pr:high","name":"pr:high","color":"e10c02"},{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"open","locked":false,"assignee":{"login":"mlucy","id":1777134,"avatar_url":"https://avatars.githubusercontent.com/u/1777134?v=3","gravatar_id":"","url":"https://api.github.com/users/mlucy","html_url":"https://github.com/mlucy","followers_url":"https://api.github.com/users/mlucy/followers","following_url":"https://api.github.com/users/mlucy/following{/other_user}","gists_url":"https://api.github.com/users/mlucy/gists{/gist_id}","starred_url":"https://api.github.com/users/mlucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlucy/subscriptions","organizations_url":"https://api.github.com/users/mlucy/orgs","repos_url":"https://api.github.com/users/mlucy/repos","events_url":"https://api.github.com/users/mlucy/events{/privacy}","received_events_url":"https://api.github.com/users/mlucy/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1.x","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104/labels","id":1249551,"number":104,"title":"2.1.x","description":null,"creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":1,"state":"open","created_at":"2015-08-11T07:18:11Z","updated_at":"2015-08-13T04:38:20Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2015-08-13T00:26:48Z","updated_at":"2015-08-13T04:53:11Z","closed_at":null,"body":"This was reported by SageMath on the mailing list.\r\n\r\nThe first bug is a failed sanity check:\r\n```\r\nFEED 30288716-0d35-41e0-acbf-a81be2ae9527 is broken, so canceling -- {\"message\":\"SANITY CHECK FAILED: [batch.size() != 0 || (parent->coro_env->return_empty_normal_batches == return_empty_normal_batches_t::YES) || (bs.get_batch_type() == batch_type_t::NORMAL_FIRST)] at `src/rdb_protocol/datum_stream.cc:1286` (server is buggy).  Backtrace:\r\nWed Aug 12 21:11:16 2015\r\n\r\n1: backtrace_t::backtrace_t() at 0xa4acb0 (/usr/bin/rethinkdb)\r\n2: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at 0xa4afae (/usr/bin/rethinkdb)\r\n3: ql::runtime_sanity_check_failed(char const*, int, char const*, std::string const&) at 0x789736 (/usr/bin/rethinkdb)\r\n4: ql::coro_stream_t::cb(auto_drainer_t::lock_t) at 0x7cdd57 (/usr/bin/rethinkdb)\r\n5: callable_action_instance_t<ql::coro_stream_t::maybe_launch_read()::{lambda()#2}>::run_action() at 0x7ce120 (/usr/bin/rethinkdb)\r\n6: coro_t::run() at 0x954418 (/usr/bin/rethinkdb)\r\n in:\r\nr.db(\"smc\").table(\"file_use\").getAll(\"0ab77792-c73f-4173-b3f8-3ee9b62b265e\", ...\r\n```\r\n\r\nThe second is a crash:\r\n```\r\n2015-08-12T22:21:30.086420124 47.641620s notice: Connected to server \"rethink1\" 5c0f5431-e72d-4e30-bda3-a4804f19d281\r\n2015-08-12T22:23:19.046464266 156.601664s error: Error in src/rdb_protocol/changefeed.cc at line 1739:\r\n2015-08-12T22:23:19.046499149 156.601698s error: Guarantee failed: [active()]\r\n2015-08-12T22:23:19.046508241 156.601707s error: Backtrace:\r\n2015-08-12T22:23:19.200879225 156.756080s error: Error in src/rdb_protocol/changefeed.cc at line 1739:\r\n2015-08-12T22:23:19.200918701 156.756118s error: Guarantee failed: [active()]\r\n2015-08-12T22:23:19.200928371 156.756127s error: Backtrace:\r\n2015-08-12T22:23:19.286724209 156.841926s error: Wed Aug 12 22:23:19 2015\\n\\n1: backtrace_t::backtrace_t() at ??:?\\n2: format_backtrace(bool) at ??:?\\n3: report_fatal_error\r\n(char const*, int, char const*, ...) at ??:?\r\n4: ql::changefeed::range_sub_t::apply_ops(ql::datum_t) at ??:?\r\n5: ql::changefeed::msg_visitor_t::operator()(ql::changefeed::msg_t::change_t const&) const::{lambda(ql::changefeed::range_sub_t*)#1}::operator()(ql::changefeed::range_sub_t*) const at ??:?\r\n6: void ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int) at ??:?\r\n7: callable_action_instance_t<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)>, long> >::run_action() at ??:?\r\n8: coro_t::run() at ??:?\r\n```"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/130535256","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4678#issuecomment-130535256","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678","id":130535256,"user":{"login":"williamstein","id":1276278,"avatar_url":"https://avatars.githubusercontent.com/u/1276278?v=3","gravatar_id":"","url":"https://api.github.com/users/williamstein","html_url":"https://github.com/williamstein","followers_url":"https://api.github.com/users/williamstein/followers","following_url":"https://api.github.com/users/williamstein/following{/other_user}","gists_url":"https://api.github.com/users/williamstein/gists{/gist_id}","starred_url":"https://api.github.com/users/williamstein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/williamstein/subscriptions","organizations_url":"https://api.github.com/users/williamstein/orgs","repos_url":"https://api.github.com/users/williamstein/repos","events_url":"https://api.github.com/users/williamstein/events{/privacy}","received_events_url":"https://api.github.com/users/williamstein/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T04:53:11Z","updated_at":"2015-08-13T04:53:11Z","body":"I just dumped all my data using rethinkdb's dump, then restored into 2.0.4 *without* using indexes (since I guess the dump format is otherwise not compatible), then recreated my indexes (which is easy for me, since I define them all in code).    I'm now running that combination live in production and it seems to be working very, very well.\r\n\r\nSomething I didn't mention above is that many of my Node.js clients for rethinkdb were pegged at 100% cpu, as rethinkdb itself was often at about 500% (or more CPU), when I was running 2.1, and moreover when it did sort of work for requests, it was slow.   Now with 2.0.4, the rethinkdb server is at 20% cpu and the clients are at about 3%.     Also all the Reql* errors that were in the log with 2.1 are gone. \r\n\r\nI'm going to try to write a stress test in node.js tomorrow to simulate the load that my node.js program parts on rethinkdb, and use that to test out anything you guys provide.   Thanks for being so responsive and helpful!!"}}