{"action":"created","issue":{"url":"https://api.github.com/repos/atomix/copycat/issues/19","labels_url":"https://api.github.com/repos/atomix/copycat/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/atomix/copycat/issues/19/comments","events_url":"https://api.github.com/repos/atomix/copycat/issues/19/events","html_url":"https://github.com/atomix/copycat/pull/19","id":112245556,"number":19,"title":"Cleanup commitFutures that have failed.","user":{"login":"madjam","id":8388825,"avatar_url":"https://avatars.githubusercontent.com/u/8388825?v=3","gravatar_id":"","url":"https://api.github.com/users/madjam","html_url":"https://github.com/madjam","followers_url":"https://api.github.com/users/madjam/followers","following_url":"https://api.github.com/users/madjam/following{/other_user}","gists_url":"https://api.github.com/users/madjam/gists{/gist_id}","starred_url":"https://api.github.com/users/madjam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/madjam/subscriptions","organizations_url":"https://api.github.com/users/madjam/orgs","repos_url":"https://api.github.com/users/madjam/repos","events_url":"https://api.github.com/users/madjam/events{/privacy}","received_events_url":"https://api.github.com/users/madjam/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-19T22:25:08Z","updated_at":"2015-10-20T01:19:53Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/atomix/copycat/pulls/19","html_url":"https://github.com/atomix/copycat/pull/19","diff_url":"https://github.com/atomix/copycat/pull/19.diff","patch_url":"https://github.com/atomix/copycat/pull/19.patch"},"body":"Scenario: Start a 3 node cluster c1, c2 and c3. Let's assume c1 becomes LEADER. Shut down c2 and c3.\r\nExpected: c1 should eventually detect that it is partitioned away and should step down and become a FOLLOWER\r\n\r\nWhat we see is that c1 remains in leader state. Stops heart beating because commitFuture and nextCommitFuture are both non-null and incomplete. We never fail commitFuture when we detect the communication with follower(s) failed.\r\n\r\nLater if we bring back c2 and c3 up, They will start their own polls since they can no longer receive heartbeats from c1. c1 continues to remain in LEADER state and can never rejoin others.\r\n\r\nSolution: Exceptionally complete commitFuture if a quorum of successful commits cannot be guaranteed.\r\nThis change also disables exponential backoff when interacting with down followers as it can interfere with the ability to detect and fail commitFutures that have no chance of success."},"comment":{"url":"https://api.github.com/repos/atomix/copycat/issues/comments/149391749","html_url":"https://github.com/atomix/copycat/pull/19#issuecomment-149391749","issue_url":"https://api.github.com/repos/atomix/copycat/issues/19","id":149391749,"user":{"login":"kuujo","id":823318,"avatar_url":"https://avatars.githubusercontent.com/u/823318?v=3","gravatar_id":"","url":"https://api.github.com/users/kuujo","html_url":"https://github.com/kuujo","followers_url":"https://api.github.com/users/kuujo/followers","following_url":"https://api.github.com/users/kuujo/following{/other_user}","gists_url":"https://api.github.com/users/kuujo/gists{/gist_id}","starred_url":"https://api.github.com/users/kuujo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kuujo/subscriptions","organizations_url":"https://api.github.com/users/kuujo/orgs","repos_url":"https://api.github.com/users/kuujo/repos","events_url":"https://api.github.com/users/kuujo/events{/privacy}","received_events_url":"https://api.github.com/users/kuujo/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T01:19:53Z","updated_at":"2015-10-20T01:19:53Z","body":"That test failure is unrelated."}}