{"action":"created","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4949","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4949/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4949/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4949/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4949","id":111221890,"number":4949,"title":"Error: [request.last_included_term == ps().log.prev_term]","user":{"login":"mbroadst","id":312966,"avatar_url":"https://avatars.githubusercontent.com/u/312966?v=3","gravatar_id":"","url":"https://api.github.com/users/mbroadst","html_url":"https://github.com/mbroadst","followers_url":"https://api.github.com/users/mbroadst/followers","following_url":"https://api.github.com/users/mbroadst/following{/other_user}","gists_url":"https://api.github.com/users/mbroadst/gists{/gist_id}","starred_url":"https://api.github.com/users/mbroadst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mbroadst/subscriptions","organizations_url":"https://api.github.com/users/mbroadst/orgs","repos_url":"https://api.github.com/users/mbroadst/repos","events_url":"https://api.github.com/users/mbroadst/events{/privacy}","received_events_url":"https://api.github.com/users/mbroadst/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/labels/tp:bug","name":"tp:bug","color":"e102d8"}],"state":"open","locked":false,"assignee":{"login":"VeXocide","id":167416,"avatar_url":"https://avatars.githubusercontent.com/u/167416?v=3","gravatar_id":"","url":"https://api.github.com/users/VeXocide","html_url":"https://github.com/VeXocide","followers_url":"https://api.github.com/users/VeXocide/followers","following_url":"https://api.github.com/users/VeXocide/following{/other_user}","gists_url":"https://api.github.com/users/VeXocide/gists{/gist_id}","starred_url":"https://api.github.com/users/VeXocide/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VeXocide/subscriptions","organizations_url":"https://api.github.com/users/VeXocide/orgs","repos_url":"https://api.github.com/users/VeXocide/repos","events_url":"https://api.github.com/users/VeXocide/events{/privacy}","received_events_url":"https://api.github.com/users/VeXocide/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104","html_url":"https://github.com/rethinkdb/rethinkdb/milestones/2.1.x","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/milestones/104/labels","id":1249551,"number":104,"title":"2.1.x","description":null,"creator":{"login":"danielmewes","id":505365,"avatar_url":"https://avatars.githubusercontent.com/u/505365?v=3","gravatar_id":"","url":"https://api.github.com/users/danielmewes","html_url":"https://github.com/danielmewes","followers_url":"https://api.github.com/users/danielmewes/followers","following_url":"https://api.github.com/users/danielmewes/following{/other_user}","gists_url":"https://api.github.com/users/danielmewes/gists{/gist_id}","starred_url":"https://api.github.com/users/danielmewes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielmewes/subscriptions","organizations_url":"https://api.github.com/users/danielmewes/orgs","repos_url":"https://api.github.com/users/danielmewes/repos","events_url":"https://api.github.com/users/danielmewes/events{/privacy}","received_events_url":"https://api.github.com/users/danielmewes/received_events","type":"User","site_admin":false},"open_issues":31,"closed_issues":3,"state":"open","created_at":"2015-08-11T07:18:11Z","updated_at":"2015-10-13T18:29:06Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-10-13T16:48:31Z","updated_at":"2015-10-13T18:58:41Z","closed_at":null,"body":"Looks like we've run into another crash on a small cluster (three nodes), I've manually expanded the backtrace:\r\n\r\n```\r\n2015-10-13T11:34:37.047697588 96.109902s error: Error in ./src/clustering/generic/raft_core.tcc at line 552:\r\n2015-10-13T11:34:37.047763158 96.109967s error: Guarantee failed: [request.last_included_term == ps().log.prev_term] The entry we shapshotted at was committed, so we shouldn't be receiving a different committed entry at the same index.\r\n2015-10-13T11:34:37.047803579 96.110007s error: Backtrace:\r\n2015-10-13T11:34:37.926402621 96.988611s error: Tue Oct 13 11:34:37 2015\r\n1 [0xa0f920]: backtrace_t::backtrace_t() at ??:?\r\n2 [0xa0fcb3]: format_backtrace(bool) at ??:?\r\n3 [0xc440b5]: report_fatal_error(char const*, int, char const*, ...) at ??:?\r\n4 [0xa5c871]: raft_member_t<table_raft_state_t>::on_install_snapshot_rpc(raft_rpc_request_t<table_raft_state_t>::install_snapshot_t const&, raft_rpc_reply_t::install_snapshot_t*) at ??:?\r\n5 [0xa5c9c8]: raft_member_t<table_raft_state_t>::on_rpc(raft_rpc_request_t<table_raft_state_t> const&, raft_rpc_reply_t*) at ??:?\r\n6 [0xa5cb91]: raft_networked_member_t<table_raft_state_t>::on_rpc(signal_t*, raft_rpc_request_t<table_raft_state_t> const&, mailbox_addr_t<void (raft_rpc_reply_t)> const&) at ??:?\r\n7 [0xa652a4]: mailbox_t<void (raft_rpc_request_t<table_raft_state_t>, mailbox_addr_t<void (raft_rpc_reply_t)>)>::read_impl_t::read(read_stream_t*, signal_t*) at ??:?\r\n8 [0x953f72]: /usr/bin/rethinkdb() [0x953f72] at 0x953f72 ()\r\n9 [0x933748]: coro_t::run() at ??:?\r\n2015-10-13T11:34:37.926632057 96.988836s error: Exiting.\r\n```"},"comment":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/comments/147814139","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4949#issuecomment-147814139","issue_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4949","id":147814139,"user":{"login":"timmaxw","id":316661,"avatar_url":"https://avatars.githubusercontent.com/u/316661?v=3","gravatar_id":"","url":"https://api.github.com/users/timmaxw","html_url":"https://github.com/timmaxw","followers_url":"https://api.github.com/users/timmaxw/followers","following_url":"https://api.github.com/users/timmaxw/following{/other_user}","gists_url":"https://api.github.com/users/timmaxw/gists{/gist_id}","starred_url":"https://api.github.com/users/timmaxw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timmaxw/subscriptions","organizations_url":"https://api.github.com/users/timmaxw/orgs","repos_url":"https://api.github.com/users/timmaxw/repos","events_url":"https://api.github.com/users/timmaxw/events{/privacy}","received_events_url":"https://api.github.com/users/timmaxw/received_events","type":"User","site_admin":false},"created_at":"2015-10-13T18:58:41Z","updated_at":"2015-10-13T18:58:41Z","body":"Nothing pops out at me as the cause of this bug.\r\n\r\nIf the bug is in the core Raft implementation, then the clock skew is probably unrelated. The core Raft algorithm doesn't ever pass system clock timestamps back and forth across the network. But I wonder if what we're seeing is cross-talk between different Raft instances--perhaps different epochs of the same table. The system clock is used to assign epoch timestamps, so it's possible that the clock skew could affect that somehow.\r\n\r\nThe output of `r.db(\"rethinkdb\").table(\"_debug_table_status\")` might point to the cause. @VeXocide, are you working on reproducing it locally?"}}