{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1346","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1346/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1346/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1346/events","html_url":"https://github.com/hashicorp/consul/issues/1346","id":113489825,"number":1346,"title":"Problem with 0.6 automated protocol upshift","user":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-27T02:08:46Z","updated_at":"2015-10-27T04:20:45Z","closed_at":null,"body":"While upgrading the nyc demo cluster I ran into an issue where only one of the three servers was sending its coordinates. After adding some debug logging and restarting it started working, so it's not clear what's going on. There's TLS enabled so I couldn't see in tcpdump if the servers weren't sending or if the servers were rejecting the coordinate for some reason.\r\n\r\nI'm going to add some more instrumentation and see if I can reproduce while upgrading one of the other DCs. All the servers shows up in the WAN coordinates even when the /v1/coordinate/nodes endpoint didn't show anything for a server.\r\n\r\nAdditionally, I realized that if we have a mixed configuration of servers and/or a client who doesn't yet see all the servers, then we could end up in a situation where a client fires off a coordinate which may create an unsupported item in the raft log for the other servers. I need to test this case and possibly add the ignore flag to this type of message so that we don't create problems during an upgrade."},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/151371187","html_url":"https://github.com/hashicorp/consul/issues/1346#issuecomment-151371187","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1346","id":151371187,"user":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"created_at":"2015-10-27T04:20:45Z","updated_at":"2015-10-27T04:20:45Z","body":"Need to prove it but I'm thinking it might be related to these lines in memberlist:\r\n\r\nhttps://github.com/hashicorp/memberlist/blob/master/state.go#L707-L711\r\n\r\nI did a shutdown without a leave, so I'm thinking that memberlist might have skipped updating this node's version as it came back up and announced it (thus the fishy previous value of `1 2 2 2 4 4` for all the versions)."}}