{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/2166","labels_url":"https://api.github.com/repos/coreos/etcd/issues/2166/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2166/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/2166/events","html_url":"https://github.com/coreos/etcd/issues/2166","id":55674658,"number":2166,"title":"Getting tons of `received msgApp rejection`","user":{"login":"mediocregopher","id":933154,"avatar_url":"https://avatars.githubusercontent.com/u/933154?v=3","gravatar_id":"","url":"https://api.github.com/users/mediocregopher","html_url":"https://github.com/mediocregopher","followers_url":"https://api.github.com/users/mediocregopher/followers","following_url":"https://api.github.com/users/mediocregopher/following{/other_user}","gists_url":"https://api.github.com/users/mediocregopher/gists{/gist_id}","starred_url":"https://api.github.com/users/mediocregopher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediocregopher/subscriptions","organizations_url":"https://api.github.com/users/mediocregopher/orgs","repos_url":"https://api.github.com/users/mediocregopher/repos","events_url":"https://api.github.com/users/mediocregopher/events{/privacy}","received_events_url":"https://api.github.com/users/mediocregopher/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-27T21:08:50Z","updated_at":"2015-01-28T01:03:05Z","closed_at":null,"body":"We have a three node etcd cluster (2.0.0-rc.1), and at some point we started receiving tons of:\r\n\r\n```\r\n2015/01/27 16:00:25 raft: 512ef1c906982373 received msgApp rejection from 233a2bfca7c59d9 for index 518557273\r\n2015/01/27 16:00:25 raft: 512ef1c906982373 received msgApp rejection from 233a2bfca7c59d9 for index 518557272\r\n2015/01/27 16:00:25 raft: 512ef1c906982373 received msgApp rejection from 233a2bfca7c59d9 for index 518557271\r\n2015/01/27 16:00:25 raft: 512ef1c906982373 received msgApp rejection from 233a2bfca7c59d9 for index 518557270\r\n```\r\n\r\non one node, the other two being the rejectors. I searched around and tried restarting the whole cluster (which didn't work), as well as removing the problem node from the cluster using `etcdctl member remove 512ef1c906982373` on one of the \"working\" nodes, but that just gets back `context deadline exceeded`.\r\n\r\nWe have this cluster up for testing, so it's not mission critical to get it working right now. I can mess around with things and bring it up and down if that helps diagnose any bugs. If anyone could give me a hand I'd really appreciate it :)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/71762294","html_url":"https://github.com/coreos/etcd/issues/2166#issuecomment-71762294","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2166","id":71762294,"user":{"login":"mediocregopher","id":933154,"avatar_url":"https://avatars.githubusercontent.com/u/933154?v=3","gravatar_id":"","url":"https://api.github.com/users/mediocregopher","html_url":"https://github.com/mediocregopher","followers_url":"https://api.github.com/users/mediocregopher/followers","following_url":"https://api.github.com/users/mediocregopher/following{/other_user}","gists_url":"https://api.github.com/users/mediocregopher/gists{/gist_id}","starred_url":"https://api.github.com/users/mediocregopher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mediocregopher/subscriptions","organizations_url":"https://api.github.com/users/mediocregopher/orgs","repos_url":"https://api.github.com/users/mediocregopher/repos","events_url":"https://api.github.com/users/mediocregopher/events{/privacy}","received_events_url":"https://api.github.com/users/mediocregopher/received_events","type":"User","site_admin":false},"created_at":"2015-01-28T01:03:05Z","updated_at":"2015-01-28T01:03:05Z","body":"I deployed a build off master to all three of the nodes, shut off etcd on all of them, then started it up with the master build. I no longer get the `received msgApp errors`, but the nodes won't full start up. They die with the message:\r\n\r\n    2015/01/27 20:01:06 etcdserver: read wal error: wal: snapshot not found\r\n\r\nAnd aren't responsive to any calls from etcdctl"}}