{"action":"created","comment":{"url":"https://api.github.com/repos/coreos/etcd/pulls/comments/31861456","id":31861456,"diff_hunk":"@@ -258,4 +260,43 @@ func (p *peer) pick(m raftpb.Message) (writec chan<- raftpb.Message, picked stri\n \treturn p.pipeline.msgc, pipelineMsg\n }\n \n+type peerStatus struct {\n+\tid         types.ID\n+\tmu         sync.Mutex\n+\tactive     bool\n+\tlastErrMap map[string]error\n+}\n+\n+func newPeerStatus(id types.ID) *peerStatus {\n+\treturn &peerStatus{\n+\t\tid:         id,\n+\t\tlastErrMap: make(map[string]error),\n+\t}\n+}\n+\n+func (s *peerStatus) success() {","path":"rafthttp/peer.go","position":54,"original_position":54,"commit_id":"00a33348c572b9db0572029e16c92839da8da2a0","original_commit_id":"00a33348c572b9db0572029e16c92839da8da2a0","user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"body":"active()","created_at":"2015-06-06T00:18:01Z","updated_at":"2015-06-06T00:18:01Z","html_url":"https://github.com/coreos/etcd/pull/2923#discussion_r31861456","pull_request_url":"https://api.github.com/repos/coreos/etcd/pulls/2923","_links":{"self":{"href":"https://api.github.com/repos/coreos/etcd/pulls/comments/31861456"},"html":{"href":"https://github.com/coreos/etcd/pull/2923#discussion_r31861456"},"pull_request":{"href":"https://api.github.com/repos/coreos/etcd/pulls/2923"}}},"pull_request":{"url":"https://api.github.com/repos/coreos/etcd/pulls/2923","id":37079296,"html_url":"https://github.com/coreos/etcd/pull/2923","diff_url":"https://github.com/coreos/etcd/pull/2923.diff","patch_url":"https://github.com/coreos/etcd/pull/2923.patch","issue_url":"https://api.github.com/repos/coreos/etcd/issues/2923","number":2923,"state":"open","locked":false,"title":"wip rafthttp: pretty print connection error","user":{"login":"yichengq","id":2976232,"avatar_url":"https://avatars.githubusercontent.com/u/2976232?v=3","gravatar_id":"","url":"https://api.github.com/users/yichengq","html_url":"https://github.com/yichengq","followers_url":"https://api.github.com/users/yichengq/followers","following_url":"https://api.github.com/users/yichengq/following{/other_user}","gists_url":"https://api.github.com/users/yichengq/gists{/gist_id}","starred_url":"https://api.github.com/users/yichengq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yichengq/subscriptions","organizations_url":"https://api.github.com/users/yichengq/orgs","repos_url":"https://api.github.com/users/yichengq/repos","events_url":"https://api.github.com/users/yichengq/events{/privacy}","received_events_url":"https://api.github.com/users/yichengq/received_events","type":"User","site_admin":false},"body":"The PR helps to log rafthttp event better. The changes are:\r\n1. print out the status change of connection with peer\r\n2. only print the first error for repeated ones\r\nSo users are aware of the status without being flooded by repeated log lines.\r\n\r\npending on #2911 \r\ninherit from #2402 \r\n\r\nThe log still has verbose log from raft package about unreachable report. My plan is to change it in follow-up PR.\r\n\r\nexample output when restart one member:\r\n\r\n```\r\n16:24:22 etcd2 | Terminating etcd2\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 the connection with 6e3bd23ae5f1eae0 became inactive\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 failed to dial 6e3bd23ae5f1eae0 on stream MsgApp v2 (dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd1 | 2015/06/5 16:24:22 the connection with 6e3bd23ae5f1eae0 became inactive\r\n16:24:22 etcd1 | 2015/06/5 16:24:22 failed to dial 6e3bd23ae5f1eae0 on stream MsgApp v2 (dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 failed to dial 6e3bd23ae5f1eae0 on stream Message (dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd1 | 2015/06/5 16:24:22 failed to dial 6e3bd23ae5f1eae0 on stream Message (dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 etcdserver: failed to reach the peerURL(http://localhost:7002) of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 etcdserver: cannot get the version of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 failed to write 6e3bd23ae5f1eae0 on stream Message (write tcp 127.0.0.1:51375: broken pipe)\r\n16:24:22 etcd3 | 2015/06/5 16:24:22 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 failed to write 6e3bd23ae5f1eae0 on pipeline (dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = true, pendingSnapshot = 0]\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 failed to heartbeat 6e3bd23ae5f1eae0 on stream MsgApp v2 (write tcp 127.0.0.1:51374: broken pipe)\r\n16:24:23 etcd3 | 2015/06/5 16:24:23 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n(repeat)\r\n16:24:27 etcd3 | 2015/06/5 16:24:27 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:27 etcd3 | 2015/06/5 16:24:27 etcdserver: failed to reach the peerURL(http://localhost:7002) of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:27 etcd3 | 2015/06/5 16:24:27 etcdserver: cannot get the version of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:27 etcd3 | 2015/06/5 16:24:27 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n(repeat)\r\n16:24:28 etcd3 | 2015/06/5 16:24:28 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:28 etcd1 | 2015/06/5 16:24:28 failed to heartbeat 6e3bd23ae5f1eae0 on stream MsgApp v2 (write tcp 127.0.0.1:51376: broken pipe)\r\n16:24:28 etcd1 | 2015/06/5 16:24:28 failed to heartbeat 6e3bd23ae5f1eae0 on stream Message (write tcp 127.0.0.1:51377: broken pipe)\r\n16:24:28 etcd3 | 2015/06/5 16:24:28 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n(repeat)\r\n16:24:32 etcd3 | 2015/06/5 16:24:32 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:32 etcd3 | 2015/06/5 16:24:32 etcdserver: failed to reach the peerURL(http://localhost:7002) of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:32 etcd3 | 2015/06/5 16:24:32 etcdserver: cannot get the version of member 6e3bd23ae5f1eae0 (Get http://localhost:7002/version: dial tcp 127.0.0.1:7002: connection refused)\r\n16:24:32 etcd3 | 2015/06/5 16:24:32 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n(repeat)\r\n16:24:37 etcd3 | 2015/06/5 16:24:37 raft: 924e2e83e93f2560 failed to send message to 6e3bd23ae5f1eae0 because it is unreachable [next = 154106, match = 154105, state = ProgressStateProbe, waiting = false, pendingSnapshot = 0]\r\n16:24:37 etcd3 | 2015/06/5 16:24:37 the connection with 6e3bd23ae5f1eae0 became active\r\n16:24:37 etcd1 | 2015/06/5 16:24:37 the connection with 6e3bd23ae5f1eae0 became active\r\n```","created_at":"2015-06-05T23:41:25Z","updated_at":"2015-06-06T00:18:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"da404660ededde407b8815be208342530ce7cca8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/coreos/etcd/pulls/2923/commits","review_comments_url":"https://api.github.com/repos/coreos/etcd/pulls/2923/comments","review_comment_url":"https://api.github.com/repos/coreos/etcd/pulls/comments{/number}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/2923/comments","statuses_url":"https://api.github.com/repos/coreos/etcd/statuses/00a33348c572b9db0572029e16c92839da8da2a0","head":{"label":"yichengq:rafthttp-status","ref":"rafthttp-status","sha":"00a33348c572b9db0572029e16c92839da8da2a0","user":{"login":"yichengq","id":2976232,"avatar_url":"https://avatars.githubusercontent.com/u/2976232?v=3","gravatar_id":"","url":"https://api.github.com/users/yichengq","html_url":"https://github.com/yichengq","followers_url":"https://api.github.com/users/yichengq/followers","following_url":"https://api.github.com/users/yichengq/following{/other_user}","gists_url":"https://api.github.com/users/yichengq/gists{/gist_id}","starred_url":"https://api.github.com/users/yichengq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yichengq/subscriptions","organizations_url":"https://api.github.com/users/yichengq/orgs","repos_url":"https://api.github.com/users/yichengq/repos","events_url":"https://api.github.com/users/yichengq/events{/privacy}","received_events_url":"https://api.github.com/users/yichengq/received_events","type":"User","site_admin":false},"repo":{"id":16744058,"name":"etcd","full_name":"yichengq/etcd","owner":{"login":"yichengq","id":2976232,"avatar_url":"https://avatars.githubusercontent.com/u/2976232?v=3","gravatar_id":"","url":"https://api.github.com/users/yichengq","html_url":"https://github.com/yichengq","followers_url":"https://api.github.com/users/yichengq/followers","following_url":"https://api.github.com/users/yichengq/following{/other_user}","gists_url":"https://api.github.com/users/yichengq/gists{/gist_id}","starred_url":"https://api.github.com/users/yichengq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yichengq/subscriptions","organizations_url":"https://api.github.com/users/yichengq/orgs","repos_url":"https://api.github.com/users/yichengq/repos","events_url":"https://api.github.com/users/yichengq/events{/privacy}","received_events_url":"https://api.github.com/users/yichengq/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/yichengq/etcd","description":"A highly-available key value store for shared configuration and service discovery","fork":true,"url":"https://api.github.com/repos/yichengq/etcd","forks_url":"https://api.github.com/repos/yichengq/etcd/forks","keys_url":"https://api.github.com/repos/yichengq/etcd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/yichengq/etcd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/yichengq/etcd/teams","hooks_url":"https://api.github.com/repos/yichengq/etcd/hooks","issue_events_url":"https://api.github.com/repos/yichengq/etcd/issues/events{/number}","events_url":"https://api.github.com/repos/yichengq/etcd/events","assignees_url":"https://api.github.com/repos/yichengq/etcd/assignees{/user}","branches_url":"https://api.github.com/repos/yichengq/etcd/branches{/branch}","tags_url":"https://api.github.com/repos/yichengq/etcd/tags","blobs_url":"https://api.github.com/repos/yichengq/etcd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/yichengq/etcd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/yichengq/etcd/git/refs{/sha}","trees_url":"https://api.github.com/repos/yichengq/etcd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/yichengq/etcd/statuses/{sha}","languages_url":"https://api.github.com/repos/yichengq/etcd/languages","stargazers_url":"https://api.github.com/repos/yichengq/etcd/stargazers","contributors_url":"https://api.github.com/repos/yichengq/etcd/contributors","subscribers_url":"https://api.github.com/repos/yichengq/etcd/subscribers","subscription_url":"https://api.github.com/repos/yichengq/etcd/subscription","commits_url":"https://api.github.com/repos/yichengq/etcd/commits{/sha}","git_commits_url":"https://api.github.com/repos/yichengq/etcd/git/commits{/sha}","comments_url":"https://api.github.com/repos/yichengq/etcd/comments{/number}","issue_comment_url":"https://api.github.com/repos/yichengq/etcd/issues/comments{/number}","contents_url":"https://api.github.com/repos/yichengq/etcd/contents/{+path}","compare_url":"https://api.github.com/repos/yichengq/etcd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/yichengq/etcd/merges","archive_url":"https://api.github.com/repos/yichengq/etcd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/yichengq/etcd/downloads","issues_url":"https://api.github.com/repos/yichengq/etcd/issues{/number}","pulls_url":"https://api.github.com/repos/yichengq/etcd/pulls{/number}","milestones_url":"https://api.github.com/repos/yichengq/etcd/milestones{/number}","notifications_url":"https://api.github.com/repos/yichengq/etcd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/yichengq/etcd/labels{/name}","releases_url":"https://api.github.com/repos/yichengq/etcd/releases{/id}","created_at":"2014-02-11T20:18:23Z","updated_at":"2015-02-12T19:42:08Z","pushed_at":"2015-06-05T23:40:58Z","git_url":"git://github.com/yichengq/etcd.git","ssh_url":"git@github.com:yichengq/etcd.git","clone_url":"https://github.com/yichengq/etcd.git","svn_url":"https://github.com/yichengq/etcd","homepage":"","size":20206,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"coreos:master","ref":"master","sha":"05b55d9d752bed7bf1879fa1fe1afe3d85e0db6b","user":{"login":"coreos","id":3730757,"avatar_url":"https://avatars.githubusercontent.com/u/3730757?v=3","gravatar_id":"","url":"https://api.github.com/users/coreos","html_url":"https://github.com/coreos","followers_url":"https://api.github.com/users/coreos/followers","following_url":"https://api.github.com/users/coreos/following{/other_user}","gists_url":"https://api.github.com/users/coreos/gists{/gist_id}","starred_url":"https://api.github.com/users/coreos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coreos/subscriptions","organizations_url":"https://api.github.com/users/coreos/orgs","repos_url":"https://api.github.com/users/coreos/repos","events_url":"https://api.github.com/users/coreos/events{/privacy}","received_events_url":"https://api.github.com/users/coreos/received_events","type":"Organization","site_admin":false},"repo":{"id":11225014,"name":"etcd","full_name":"coreos/etcd","owner":{"login":"coreos","id":3730757,"avatar_url":"https://avatars.githubusercontent.com/u/3730757?v=3","gravatar_id":"","url":"https://api.github.com/users/coreos","html_url":"https://github.com/coreos","followers_url":"https://api.github.com/users/coreos/followers","following_url":"https://api.github.com/users/coreos/following{/other_user}","gists_url":"https://api.github.com/users/coreos/gists{/gist_id}","starred_url":"https://api.github.com/users/coreos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coreos/subscriptions","organizations_url":"https://api.github.com/users/coreos/orgs","repos_url":"https://api.github.com/users/coreos/repos","events_url":"https://api.github.com/users/coreos/events{/privacy}","received_events_url":"https://api.github.com/users/coreos/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/coreos/etcd","description":"A highly-available key-value store for shared configuration and service discovery","fork":false,"url":"https://api.github.com/repos/coreos/etcd","forks_url":"https://api.github.com/repos/coreos/etcd/forks","keys_url":"https://api.github.com/repos/coreos/etcd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/coreos/etcd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/coreos/etcd/teams","hooks_url":"https://api.github.com/repos/coreos/etcd/hooks","issue_events_url":"https://api.github.com/repos/coreos/etcd/issues/events{/number}","events_url":"https://api.github.com/repos/coreos/etcd/events","assignees_url":"https://api.github.com/repos/coreos/etcd/assignees{/user}","branches_url":"https://api.github.com/repos/coreos/etcd/branches{/branch}","tags_url":"https://api.github.com/repos/coreos/etcd/tags","blobs_url":"https://api.github.com/repos/coreos/etcd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/coreos/etcd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/coreos/etcd/git/refs{/sha}","trees_url":"https://api.github.com/repos/coreos/etcd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/coreos/etcd/statuses/{sha}","languages_url":"https://api.github.com/repos/coreos/etcd/languages","stargazers_url":"https://api.github.com/repos/coreos/etcd/stargazers","contributors_url":"https://api.github.com/repos/coreos/etcd/contributors","subscribers_url":"https://api.github.com/repos/coreos/etcd/subscribers","subscription_url":"https://api.github.com/repos/coreos/etcd/subscription","commits_url":"https://api.github.com/repos/coreos/etcd/commits{/sha}","git_commits_url":"https://api.github.com/repos/coreos/etcd/git/commits{/sha}","comments_url":"https://api.github.com/repos/coreos/etcd/comments{/number}","issue_comment_url":"https://api.github.com/repos/coreos/etcd/issues/comments{/number}","contents_url":"https://api.github.com/repos/coreos/etcd/contents/{+path}","compare_url":"https://api.github.com/repos/coreos/etcd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/coreos/etcd/merges","archive_url":"https://api.github.com/repos/coreos/etcd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/coreos/etcd/downloads","issues_url":"https://api.github.com/repos/coreos/etcd/issues{/number}","pulls_url":"https://api.github.com/repos/coreos/etcd/pulls{/number}","milestones_url":"https://api.github.com/repos/coreos/etcd/milestones{/number}","notifications_url":"https://api.github.com/repos/coreos/etcd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/coreos/etcd/labels{/name}","releases_url":"https://api.github.com/repos/coreos/etcd/releases{/id}","created_at":"2013-07-06T21:57:21Z","updated_at":"2015-06-05T21:52:00Z","pushed_at":"2015-06-06T00:10:34Z","git_url":"git://github.com/coreos/etcd.git","ssh_url":"git@github.com:coreos/etcd.git","clone_url":"https://github.com/coreos/etcd.git","svn_url":"https://github.com/coreos/etcd","homepage":"","size":36010,"stargazers_count":6594,"watchers_count":6594,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":866,"mirror_url":null,"open_issues_count":167,"forks":866,"open_issues":167,"watchers":6594,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/coreos/etcd/pulls/2923"},"html":{"href":"https://github.com/coreos/etcd/pull/2923"},"issue":{"href":"https://api.github.com/repos/coreos/etcd/issues/2923"},"comments":{"href":"https://api.github.com/repos/coreos/etcd/issues/2923/comments"},"review_comments":{"href":"https://api.github.com/repos/coreos/etcd/pulls/2923/comments"},"review_comment":{"href":"https://api.github.com/repos/coreos/etcd/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/coreos/etcd/pulls/2923/commits"},"statuses":{"href":"https://api.github.com/repos/coreos/etcd/statuses/00a33348c572b9db0572029e16c92839da8da2a0"}}}}