{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4678","id":100663966,"number":4678,"title":"Changefeed Crashing Bugs","user":{"login":"mlucy","id":1777134,"avatar_url":"https://avatars.githubusercontent.com/u/1777134?v=3","gravatar_id":"","url":"https://api.github.com/users/mlucy","html_url":"https://github.com/mlucy","followers_url":"https://api.github.com/users/mlucy/followers","following_url":"https://api.github.com/users/mlucy/following{/other_user}","gists_url":"https://api.github.com/users/mlucy/gists{/gist_id}","starred_url":"https://api.github.com/users/mlucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlucy/subscriptions","organizations_url":"https://api.github.com/users/mlucy/orgs","repos_url":"https://api.github.com/users/mlucy/repos","events_url":"https://api.github.com/users/mlucy/events{/privacy}","received_events_url":"https://api.github.com/users/mlucy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/pr:high","name":"pr:high","color":"e10c02"},{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"closed","locked":false,"assignee":{"login":"mlucy","id":1777134,"avatar_url":"https://avatars.githubusercontent.com/u/1777134?v=3","gravatar_id":"","url":"https://api.github.com/users/mlucy","html_url":"https://github.com/mlucy","followers_url":"https://api.github.com/users/mlucy/followers","following_url":"https://api.github.com/users/mlucy/following{/other_user}","gists_url":"https://api.github.com/users/mlucy/gists{/gist_id}","starred_url":"https://api.github.com/users/mlucy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mlucy/subscriptions","organizations_url":"https://api.github.com/users/mlucy/orgs","repos_url":"https://api.github.com/users/mlucy/repos","events_url":"https://api.github.com/users/mlucy/events{/privacy}","received_events_url":"https://api.github.com/users/mlucy/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/106","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1.2","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/106/labels","id":1267005,"number":106,"title":"2.1.2","description":"","creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":18,"state":"closed","created_at":"2015-08-21T22:34:48Z","updated_at":"2015-11-11T01:16:34Z","due_on":null,"closed_at":"2015-08-31T17:52:19Z"},"comments":33,"created_at":"2015-08-13T00:26:48Z","updated_at":"2015-11-15T15:59:42Z","closed_at":"2015-08-25T00:40:58Z","body":"This was reported by SageMath on the mailing list.\r\n\r\nThe first bug is a failed sanity check:\r\n```\r\nFEED 30288716-0d35-41e0-acbf-a81be2ae9527 is broken, so canceling -- {\"message\":\"SANITY CHECK FAILED: [batch.size() != 0 || (parent->coro_env->return_empty_normal_batches == return_empty_normal_batches_t::YES) || (bs.get_batch_type() == batch_type_t::NORMAL_FIRST)] at `src/rdb_protocol/datum_stream.cc:1286` (server is buggy).  Backtrace:\r\nWed Aug 12 21:11:16 2015\r\n\r\n1: backtrace_t::backtrace_t() at 0xa4acb0 (/usr/bin/rethinkdb)\r\n2: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at 0xa4afae (/usr/bin/rethinkdb)\r\n3: ql::runtime_sanity_check_failed(char const*, int, char const*, std::string const&) at 0x789736 (/usr/bin/rethinkdb)\r\n4: ql::coro_stream_t::cb(auto_drainer_t::lock_t) at 0x7cdd57 (/usr/bin/rethinkdb)\r\n5: callable_action_instance_t<ql::coro_stream_t::maybe_launch_read()::{lambda()#2}>::run_action() at 0x7ce120 (/usr/bin/rethinkdb)\r\n6: coro_t::run() at 0x954418 (/usr/bin/rethinkdb)\r\n in:\r\nr.db(\"smc\").table(\"file_use\").getAll(\"0ab77792-c73f-4173-b3f8-3ee9b62b265e\", ...\r\n```\r\n\r\nThe second is a crash:\r\n```\r\n2015-08-12T22:21:30.086420124 47.641620s notice: Connected to server \"rethink1\" 5c0f5431-e72d-4e30-bda3-a4804f19d281\r\n2015-08-12T22:23:19.046464266 156.601664s error: Error in src/rdb_protocol/changefeed.cc at line 1739:\r\n2015-08-12T22:23:19.046499149 156.601698s error: Guarantee failed: [active()]\r\n2015-08-12T22:23:19.046508241 156.601707s error: Backtrace:\r\n2015-08-12T22:23:19.200879225 156.756080s error: Error in src/rdb_protocol/changefeed.cc at line 1739:\r\n2015-08-12T22:23:19.200918701 156.756118s error: Guarantee failed: [active()]\r\n2015-08-12T22:23:19.200928371 156.756127s error: Backtrace:\r\n2015-08-12T22:23:19.286724209 156.841926s error: Wed Aug 12 22:23:19 2015\\n\\n1: backtrace_t::backtrace_t() at ??:?\\n2: format_backtrace(bool) at ??:?\\n3: report_fatal_error\r\n(char const*, int, char const*, ...) at ??:?\r\n4: ql::changefeed::range_sub_t::apply_ops(ql::datum_t) at ??:?\r\n5: ql::changefeed::msg_visitor_t::operator()(ql::changefeed::msg_t::change_t const&) const::{lambda(ql::changefeed::range_sub_t*)#1}::operator()(ql::changefeed::range_sub_t*) const at ??:?\r\n6: void ql::changefeed::feed_t::each_sub_in_vec_cb<ql::changefeed::range_sub_t>(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int) at ??:?\r\n7: callable_action_instance_t<pmap_runner_one_arg_t<std::_Bind<std::_Mem_fn<void (ql::changefeed::feed_t::*)(std::function<void (ql::changefeed::range_sub_t*)> const&, std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const&, std::vector<int, std::allocator<int> > const&, int)> (ql::changefeed::feed_t*, std::reference_wrapper<std::function<void (ql::changefeed::range_sub_t*)> const>, std::reference_wrapper<std::vector<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> >, std::allocator<std::set<ql::changefeed::range_sub_t*, std::less<ql::changefeed::range_sub_t*>, std::allocator<ql::changefeed::range_sub_t*> > > > const>, std::reference_wrapper<std::vector<int, std::allocator<int> > const>, std::_Placeholder<1>)>, long> >::run_action() at ??:?\r\n8: coro_t::run() at ??:?\r\n```"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/156822344","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4678#issuecomment-156822344","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4678","id":156822344,"user":{"login":"williamstein","id":1276278,"avatar_url":"https://avatars.githubusercontent.com/u/1276278?v=3","gravatar_id":"","url":"https://api.github.com/users/williamstein","html_url":"https://github.com/williamstein","followers_url":"https://api.github.com/users/williamstein/followers","following_url":"https://api.github.com/users/williamstein/following{/other_user}","gists_url":"https://api.github.com/users/williamstein/gists{/gist_id}","starred_url":"https://api.github.com/users/williamstein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/williamstein/subscriptions","organizations_url":"https://api.github.com/users/williamstein/orgs","repos_url":"https://api.github.com/users/williamstein/repos","events_url":"https://api.github.com/users/williamstein/events{/privacy}","received_events_url":"https://api.github.com/users/williamstein/received_events","type":"User","site_admin":false},"created_at":"2015-11-15T15:59:42Z","updated_at":"2015-11-15T15:59:42Z","body":"I upgraded to RethinkDB 2.2, and I'm getting this constantly:\r\n\r\n```\r\n2015-11-15T15:52:55.896809798 5334.337322s error: Error in src/rdb_protocol/datum_stream.cc at line 855:\r\n2015-11-15T15:52:55.896843712 5334.337355s error: Guarantee failed: [found_hash_pair]\r\n2015-11-15T15:52:55.896851427 5334.337363s error: Backtrace:\r\n2015-11-15T15:52:56.101595162 5334.542109s error: Sun Nov 15 15:52:55 2015\\n\\n1 [0xa99990]: backtrace_t::backtrace_t() at ??:?\\n2 [0xa99d5c]: format_backtrace(bool) at ??:?\\n3 [0xd12ea3]: report_fatal_error(char const*, int, char c\r\nonst*, ...) at ??:?\\n4 [0x7ef4a9]: ql::primary_readgen_t::restrict_active_ranges(sorting_t, ql::active_ranges_t*) const at ??:?\\n5 [0x7f5ba6]: ql::rget_response_reader_t::unshard(sorting_t, rget_read_response_t&&) at ??:?\\n6 [0x7f8\r\ndd2]: ql::rget_reader_t::do_range_read(ql::env_t*, read_t const&) at ??:?\\n7 [0x7f932a]: ql::rget_reader_t::load_items(ql::env_t*, ql::batchspec_t const&) at ??:?\\n8 [0x7e9d2a]: ql::rget_response_reader_t::next_batch(ql::env_t*, ql\r\n::batchspec_t const&) at ??:?\\n9 [0x7e7345]: ql::lazy_datum_stream_t::next_batch_impl(ql::env_t*, ql::batchspec_t const&) at ??:?\\n10 [0x7e8907]: ql::datum_stream_t::next_batch(ql::env_t*, ql::batchspec_t const&) at ??:?\\n11 [0x889\r\n239]: ql::replace_term_t::eval_impl(ql::scope_env_t*, ql::args_t*, ql::eval_flags_t) const at ??:?\\n12 [0x7ce492]: ql::op_term_t::term_eval(ql::scope_env_t*, ql::eval_flags_t) const at ??:?\\n13 [0x8f07aa]: ql::runtime_term_t::eval_\r\non_current_stack(ql::scope_env_t*, ql::eval_flags_t) const at ??:?\\n14 [0x8f0adc]: ql::runtime_term_t::eval(ql::scope_env_t*, ql::eval_flags_t) const at ??:?\\n15 [0x6f6cb9]: ql::query_cache_t::ref_t::run(ql::env_t*, ql::response_t*\r\n) at ??:?\\n16 [0x6f8676]: ql::query_cache_t::ref_t::fill_response(ql::response_t*) at ??:?\\n17 [0x7c8d5b]: rdb_query_server_t::run_query(ql::query_params_t*, ql::response_t*, signal_t*) at ??:?\\n18 [0xa963ba]: void save_exception<v\r\noid query_server_t::connection_loop<json_protocol_t>(linux_tcp_conn_t*, unsigned long, ql::query_cache_t*, signal_t*)::{lambda()#1}::operator()() const::{lambda()#1}>(std::__exception_ptr::exception_ptr*, std::string*, cond_t*, voi\r\nd query_server_t::connection_loop<json_protocol_t>(linux_tcp_conn_t*, unsigned long, ql::query_cache_t*, signal_t*)::{lambda()#1}::operator()() const::{lambda()#1}&&) at ??:?\\n19 [0xa96a2d]: void query_server_t::connection_loop<jso\r\nn_protocol_t>(linux_tcp_conn_t*, unsigned long, ql::query_cache_t*, signal_t*)::{lambda()#1}::operator()() const at ??:?\\n20 [0x9a4f68]: coro_t::run() at ??:?\r\n2015-11-15T15:52:56.101699892 5334.542211s error: Exiting.\r\n2015-11-15T15:54:01.444564463 0.232422s notice: Running rethinkdb 2.2.0~0vivid (GCC 4.9.2)...\r\n2015-11-15T15:54:01.448363754 0.236220s notice: Running on Linux 3.19.0-31-generic x86_64\r\n2015-11-15T15:54:01.448399911 0.236255s notice: Loading data from directory /var/lib/rethinkdb/default/data\r\n2015-11-15T15:54:01.492573965 0.280430s info: Cache size is set to 9000 MB\r\n```"}}