{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4979","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4979/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4979/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4979/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4979","id":112714265,"number":4979,"title":"Crash during table fuzzing, Guarantee failed: [last >= first - 1]","user":{"login":"Tryneus","id":552910,"avatar_url":"https://avatars.githubusercontent.com/u/552910?v=3","gravatar_id":"","url":"https://api.github.com/users/Tryneus","html_url":"https://github.com/Tryneus","followers_url":"https://api.github.com/users/Tryneus/followers","following_url":"https://api.github.com/users/Tryneus/following{/other_user}","gists_url":"https://api.github.com/users/Tryneus/gists{/gist_id}","starred_url":"https://api.github.com/users/Tryneus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tryneus/subscriptions","organizations_url":"https://api.github.com/users/Tryneus/orgs","repos_url":"https://api.github.com/users/Tryneus/repos","events_url":"https://api.github.com/users/Tryneus/events{/privacy}","received_events_url":"https://api.github.com/users/Tryneus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/cp:clustering","name":"cp:clustering","color":"207de5"},{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"open","locked":false,"assignee":{"login":"VeXocide","id":167416,"avatar_url":"https://avatars.githubusercontent.com/u/167416?v=3","gravatar_id":"","url":"https://api.github.com/users/VeXocide","html_url":"https://github.com/VeXocide","followers_url":"https://api.github.com/users/VeXocide/followers","following_url":"https://api.github.com/users/VeXocide/following{/other_user}","gists_url":"https://api.github.com/users/VeXocide/gists{/gist_id}","starred_url":"https://api.github.com/users/VeXocide/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VeXocide/subscriptions","organizations_url":"https://api.github.com/users/VeXocide/orgs","repos_url":"https://api.github.com/users/VeXocide/repos","events_url":"https://api.github.com/users/VeXocide/events{/privacy}","received_events_url":"https://api.github.com/users/VeXocide/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1.x","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104/labels","id":1249551,"number":104,"title":"2.1.x","description":null,"creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":34,"closed_issues":8,"state":"open","created_at":"2015-08-11T07:18:11Z","updated_at":"2015-10-23T00:51:48Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-10-22T01:56:26Z","updated_at":"2015-10-23T01:13:34Z","closed_at":null,"body":"This was running on `v2.1.x` (a007331), using the invocation: `table_fuzzer.py --servers 7 --threads 48 --kill`.  Unfortunately the core file was overwritten by another crash (Branch history is missing pieces).  \r\n\r\n```cpp\r\nerror: Error in ../src/clustering/generic/raft_core.tcc at line 1040:\r\nerror: Guarantee failed: [last >= first - 1]\r\nerror: Backtrace:\r\n2015-10-21T18:49:47.365139506 39.897259s error: Wed Oct 21 18:49:46 2015\r\n\r\n1 [0x1bb88ac]: rethinkdb_backtrace(void**, int) at rethinkdb_backtrace.cc:101\r\n2 [0x17dd771]: backtrace_t::backtrace_t() at backtrace.cc:203\r\n3 [0x17ddddf]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:283\r\n4 [0x17dd6bf]: format_backtrace(bool) at backtrace.cc:198\r\n5 [0x17d15a2]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:83\r\n6 [0x1d9c36f]: raft_member_t<table_raft_state_t>::apply_log_entries(raft_member_t<table_raft_state_t>::state_and_config_t*, raft_log_t<table_raft_state_t> const&, unsigned long, unsigned long) at raft_core.tcc:1040\r\n7 [0x1da8488]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x1da8488] at 0x1da8488 ()\r\n8 [0x1de317f]: std::_Function_handler<bool (raft_member_t<table_raft_state_t>::state_and_config_t*), raft_member_t<table_raft_state_t>::on_append_entries_rpc(raft_rpc_request_t<table_raft_state_t>::append_entries_t const&, raft_rpc_reply_t::append_entries_t*)::{lambda(raft_member_t<table_raft_state_t>::state_and_config_t*)#1}>::_M_invoke(std::_Any_data const&, raft_member_t<table_raft_state_t>::state_and_config_t*) at functional:1765\r\n9 [0x1d9c60f]: std::function<bool (raft_member_t<table_raft_state_t>::state_and_config_t*)>::operator()(raft_member_t<table_raft_state_t>::state_and_config_t*) const at functional:2162\r\n10 [0x1d9b8da]: watchable_variable_t<raft_member_t<table_raft_state_t>::state_and_config_t>::apply_atomic_op(std::function<bool (raft_member_t<table_raft_state_t>::state_and_config_t*)> const&) at watchable.hpp:221\r\n11 [0x1dcaa99]: raft_member_t<table_raft_state_t>::on_append_entries_rpc(raft_rpc_request_t<table_raft_state_t>::append_entries_t const&, raft_rpc_reply_t::append_entries_t*) at raft_core.tcc:677\r\n12 [0x1dbe475]: raft_member_t<table_raft_state_t>::on_rpc(raft_rpc_request_t<table_raft_state_t> const&, raft_rpc_reply_t*) at raft_core.tcc:302\r\n13 [0x1db391c]: raft_networked_member_t<table_raft_state_t>::on_rpc(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&) at raft_network.tcc:81\r\n14 [0x1de49cf]: std::_Mem_fn<void (raft_networked_member_t<table_raft_state_t>::*)(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&)>::operator()(raft_networked_member_t<table_raft_state_t>*, signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&) const at functional:551\r\n15 [0x1dd86ea]: void std::_Bind<std::_Mem_fn<void (raft_networked_member_t<table_raft_state_t>::*)(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&)> (raft_networked_member_t<table_raft_state_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>)>::__call<void, signal_t*&&, raft_rpc_request_t<table_raft_state_t>&&, mailbox_addr_t<void (raft_rpc_reply_t)>&&, 0, 1, 2, 3>(std::tuple<signal_t*&&, raft_rpc_request_t<table_raft_state_t>&&, mailbox_addr_t<void (raft_rpc_reply_t)>&&>&&, std::_Index_tuple<0, 1, 2, 3>) at functional:1147\r\n16 [0x1dcbad9]: void std::_Bind<std::_Mem_fn<void (raft_networked_member_t<table_raft_state_t>::*)(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&)> (raft_networked_member_t<table_raft_state_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>)>::operator()<signal_t*, raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>, void>(signal_t*&&, raft_rpc_request_t<table_raft_state_t>&&, mailbox_addr_t<void (raft_rpc_reply_t)>&&) at functional:1206\r\n17 [0x1dbf3e5]: std::_Function_handler<void (signal_t*, raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>), std::_Bind<std::_Mem_fn<void (raft_networked_member_t<table_raft_state_t>::*)(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&)> (raft_networked_member_t<table_raft_state_t>*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>)> >::_M_invoke(std::_Any_data const&, signal_t*, raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>) at functional:1780\r\n18 [0x1e0c956]: std::function<void (signal_t*, raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>)>::operator()(signal_t*, raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>) const at functional:2161\r\n19 [0x1e0a679]: mailbox_t<void (raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>)>::read_impl_t::read(read_stream_t*, signal_t*) at typed.hpp:245\r\n20 [0x1783ae7]: mailbox_manager_t::mailbox_read_coroutine(connectivity_cluster_t::connection_t*, auto_drainer_t::lock_t, threadnum_t, unsigned long, std::vector<char, std::allocator<char> >*, long, mailbox_manager_t::force_yield_t) at mailbox.cc:277\r\n21 [0x1783957]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x1783957] at 0x1783957 ()\r\n22 [0x1784492]: /home/ssd2/grey/rethinkdb/build/debug/rethinkdb() [0x1784492] at 0x1784492 ()\r\n23 [0x15ffa04]: callable_action_wrapper_t::run() at runtime_utils.cc:43\r\n24 [0x1607c0b]: coro_t::run() at coroutines.cc:207\r\n```"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/150399475","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4979#issuecomment-150399475","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4979","id":150399475,"user":{"login":"Tryneus","id":552910,"avatar_url":"https://avatars.githubusercontent.com/u/552910?v=3","gravatar_id":"","url":"https://api.github.com/users/Tryneus","html_url":"https://github.com/Tryneus","followers_url":"https://api.github.com/users/Tryneus/followers","following_url":"https://api.github.com/users/Tryneus/following{/other_user}","gists_url":"https://api.github.com/users/Tryneus/gists{/gist_id}","starred_url":"https://api.github.com/users/Tryneus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tryneus/subscriptions","organizations_url":"https://api.github.com/users/Tryneus/orgs","repos_url":"https://api.github.com/users/Tryneus/repos","events_url":"https://api.github.com/users/Tryneus/events{/privacy}","received_events_url":"https://api.github.com/users/Tryneus/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T01:13:34Z","updated_at":"2015-10-23T01:13:34Z","body":"I ran into this again (on `next` this time).  Following this crash, further attempts to relaunch the server result in a different crash.  Core files suspiciously missing.  Data files available on newton in `~grey/issue_4979_2/`.\r\n\r\n```\r\nerror: Error in ../src/clustering/generic/raft_core.hpp at line 271:\r\nerror: Guarantee failed: [index <= get_latest_index()] the log doesn't go forward this far\r\nerror: Backtrace:\r\nerror: Thu Oct 22 18:09:45 2015\r\n\r\n       1 [0x1d22d28]: rethinkdb_backtrace(void**, int) at rethinkdb_backtrace.cc:101\r\n       2 [0x190af01]: backtrace_t::backtrace_t() at backtrace.cc:203\r\n       3 [0x190b56f]: lazy_backtrace_formatter_t::lazy_backtrace_formatter_t() at backtrace.cc:283\r\n       4 [0x190ae4f]: format_backtrace(bool) at backtrace.cc:198\r\n       5 [0x18fed0e]: report_fatal_error(char const*, int, char const*, ...) at errors.cc:83\r\n       6 [0x1e38ded]: raft_log_t<table_raft_state_t>::get_entry_ref(unsigned long) const at raft_core.hpp:271\r\n       7 [0x1f216e3]: raft_member_t<table_raft_state_t>::apply_log_entries(raft_member_t<table_raft_state_t>::state_and_config_t*, raft_log_t<table_raft_state_t> const&, unsigned long, unsigned long) at raft_core.tcc:1043\r\n       8 [0x1f2b044]: /home/grey/rethinkdb/build/debug/rethinkdb() [0x1f2b044] at 0x1f2b044 ()\r\n       9 [0x1f5250b]: std::_Function_handler<bool (raft_member_t<table_raft_state_t>::state_and_config_t*), raft_member_t<table_raft_state_t>::state_and_config_t(raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*, raft_network_interface_t<table_raft_state_t>*, std::string const&)::{lambda(raft_member_t<table_raft_state_t>::state_and_config_t*)#3}>::_M_invoke(std::_Any_data const&, raft_member_t<table_raft_state_t>::state_and_config_t*) at functional:1765\r\n       10 [0x1f21937]: std::function<bool (raft_member_t<table_raft_state_t>::state_and_config_t*)>::operator()(raft_member_t<table_raft_state_t>::state_and_config_t*) const at functional:2162\r\n       11 [0x1f20c02]: watchable_variable_t<raft_member_t<table_raft_state_t>::state_and_config_t>::apply_atomic_op(std::function<bool (raft_member_t<table_raft_state_t>::state_and_config_t*)> const&) at watchable.hpp:221\r\n       12 [0x1f39928]: raft_member_t<table_raft_state_t>::raft_member_t(raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*, raft_network_interface_t<table_raft_state_t>*, std::string const&) at raft_core.tcc:108\r\n       13 [0x1f3419f]: raft_networked_member_t<table_raft_state_t>::raft_networked_member_t(raft_member_id_t const&, mailbox_manager_t*, watchable_map_t<raft_member_id_t, raft_business_card_t<table_raft_state_t> >*, raft_storage_interface_t<table_raft_state_t>*, std::string const&) at raft_network.tcc:24\r\n       14 [0x1f27518]: table_manager_t::table_manager_t(uuid_u const&, mailbox_manager_t*, server_config_client_t*, watchable_map_t<std::pair<peer_id_t, uuid_u>, table_manager_bcard_t>*, backfill_throttler_t*, watchable_map_t<std::pair<uuid_u, uuid_u>, empty_value_t>*, base_path_t const&, io_backender_t*, uuid_u const&, multi_table_manager_timestamp_t::epoch_t const&, raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*, multistore_ptr_t*, perfmon_collection_t*) at table_manager.cc:67\r\n       15 [0x1fcfb94]: multi_table_manager_t::active_table_t::active_table_t(multi_table_manager_t*, multi_table_manager_t::table_t*, uuid_u const&, multi_table_manager_timestamp_t::epoch_t const&, raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*, multistore_ptr_t*, perfmon_collection_t*) at multi_table_manager.cc:136\r\n       16 [0x1fd7289]: scoped_ptr_t<multi_table_manager_t::active_table_t> make_scoped<multi_table_manager_t::active_table_t, multi_table_manager_t*, multi_table_manager_t::table_t*&, uuid_u const&, multi_table_manager_timestamp_t::epoch_t const&, raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*&, multistore_ptr_t*, perfmon_collection_t*>(multi_table_manager_t*&&, multi_table_manager_t::table_t*&, uuid_u const&, multi_table_manager_timestamp_t::epoch_t const&, raft_member_id_t const&, raft_storage_interface_t<table_raft_state_t>*&, multistore_ptr_t*&&, perfmon_collection_t*&&) at scoped.hpp:122\r\n       17 [0x1fcec58]: /home/grey/rethinkdb/build/debug/rethinkdb() [0x1fcec58] at 0x1fcec58 ()\r\n       18 [0x1fd3d17]: /home/grey/rethinkdb/build/debug/rethinkdb() [0x1fd3d17] at 0x1fd3d17 ()\r\n       19 [0x1e44de1]: std::function<void (uuid_u const&, table_active_persistent_state_t const&, raft_storage_interface_t<table_raft_state_t>*, metadata_file_t::read_txn_t*)>::operator()(uuid_u const&, table_active_persistent_state_t const&, raft_storage_interface_t<table_raft_state_t>*, metadata_file_t::read_txn_t*) const at functional:2162\r\n       20 [0x1e41644]: real_table_persistence_interface_t::read_all_metadata(std::function<void (uuid_u const&, table_active_persistent_state_t const&, raft_storage_interface_t<table_raft_state_t>*, metadata_file_t::read_txn_t*)> const&, std::function<void (uuid_u const&, table_inactive_persistent_state_t const&, metadata_file_t::read_txn_t*)> const&, signal_t*) at table_interface.cc:203\r\n       21 [0x1fcf173]: multi_table_manager_t::multi_table_manager_t(uuid_u const&, mailbox_manager_t*, server_config_client_t*, watchable_map_t<peer_id_t, multi_table_manager_bcard_t>*, watchable_map_t<std::pair<peer_id_t, uuid_u>, table_manager_bcard_t>*, watchable_map_t<std::pair<uuid_u, uuid_u>, empty_value_t>*, table_persistence_interface_t*, base_path_t const&, io_backender_t*, perfmon_collection_repo_t*) at multi_table_manager.cc:66\r\n       22 [0x1d4a1ce]: do_serve(io_backender_t*, bool, base_path_t const&, metadata_file_t*, serve_info_t const&, os_signal_cond_t*) at serve.cc:295\r\n```"}}