{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/3805","labels_url":"https://api.github.com/repos/coreos/etcd/issues/3805/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/3805/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/3805/events","html_url":"https://github.com/coreos/etcd/issues/3805","id":115057706,"number":3805,"title":"etcd v2.2.1 tries to connect directly to member even when you run local proxy","user":{"login":"kayrus","id":4047483,"avatar_url":"https://avatars.githubusercontent.com/u/4047483?v=3","gravatar_id":"","url":"https://api.github.com/users/kayrus","html_url":"https://github.com/kayrus","followers_url":"https://api.github.com/users/kayrus/followers","following_url":"https://api.github.com/users/kayrus/following{/other_user}","gists_url":"https://api.github.com/users/kayrus/gists{/gist_id}","starred_url":"https://api.github.com/users/kayrus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kayrus/subscriptions","organizations_url":"https://api.github.com/users/kayrus/orgs","repos_url":"https://api.github.com/users/kayrus/repos","events_url":"https://api.github.com/users/kayrus/events{/privacy}","received_events_url":"https://api.github.com/users/kayrus/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-04T14:03:26Z","updated_at":"2015-11-04T15:36:34Z","closed_at":"2015-11-04T15:19:33Z","body":"This issue was reported by tittof from etcd IRC channel. I've tried to reproduce it and here are my results:\r\n\r\n* coreos1 (192.168.122.143) - etcd member\r\n* coreos2 (192.168.122.2) - etcd member\r\n* coreos3 (192.168.122.71) - etcd proxy\r\n\r\nI run etcdctl on coreos3 and look for strace and tcpdump results:\r\n\r\n```\r\ncoreos3 ~ # strace etcdctl --debug ls /128 2>&1 | grep sa_family=AF_INET\r\nbind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, \"::1\", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0\r\nbind(4, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, \"::ffff:127.0.0.1\", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0\r\nconnect(5, {sa_family=AF_INET, sin_port=htons(2379), sin_addr=inet_addr(\"127.0.0.1\")}, 16) = -1 EINPROGRESS (Operation now in progress)\r\ngetsockname(6, {sa_family=AF_INET, sin_port=htons(49934), sin_addr=inet_addr(\"192.168.122.71\")}, [16]) = 0\r\nconnect(6, {sa_family=AF_INET, sin_port=htons(2379), sin_addr=inet_addr(\"192.168.122.143\")}, 16) = -1 EINPROGRESS (Operation now in progress)\r\n```\r\n\r\n```\r\n14:47:12.863382 IP coreos3.49934 > coreos1.2379: Flags [S], seq 3546867370, win 29200, options [mss 1460,sackOK,TS val 898136 ecr 0,nop,wscale 6], length 0\r\nE..<R.@.@.q...zG..z...  K.h........r.vV.........\r\n.X........\r\n14:47:12.863550 IP coreos1.2379 > coreos3.49934: Flags [S.], seq 1258281781, ack 3546867371, win 28960, options [mss 1460,sackOK,TS val 900660 ecr 898136,nop,wscale 6], length 0\r\nE..<..@.@.....z...zG    K..J..5.h....q vV.........\r\n.X....\r\n14:47:12.863611 IP coreos3.49934 > coreos1.2379: Flags [.], ack 1, win 457, options [nop,nop,TS val 898136 ecr 900660], length 0\r\nE..4R.@.@.q...zG..z...  K.h..J..6....vN.....\r\n.4.\r\n14:47:12.864414 IP coreos3.49934 > coreos1.2379: Flags [P.], seq 1:147, ack 1, win 457, options [nop,nop,TS val 898137 ecr 900660], length 146\r\nE...R.@.@.q...zG..z...  K.h..J..6....v......\r\n.4GET /v2/keys/128?quorum=false&recursive=false&sorted=false HTTP/1.1\r\nHost: coreos1:2379\r\nUser-Agent: Go-http-client/1.1\r\nAccept-Encoding: gzip\r\n\r\n\r\n14:47:12.864535 IP coreos1.2379 > coreos3.49934: Flags [.], ack 147, win 470, options [nop,nop,TS val 900662 ecr 898137], length 0\r\nE..4e.@.@._...z...zG    K..J..6.h.=....vN.....\r\n.Y.\r\n14:47:12.864715 IP coreos1.2379 > coreos3.49934: Flags [P.], seq 1:243, ack 147, win 470, options [nop,nop,TS val 900662 ecr 898137], length 242\r\nE..&e.@.@.^...z...zG    K..J..6.h.=....w@.....\r\n.YHTTP/1.1 404 Not Found\r\nContent-Type: application/json\r\nX-Etcd-Cluster-Id: fa15befb7584f0da\r\nX-Etcd-Index: 838\r\nDate: Wed, 04 Nov 2015 13:47:12 GMT\r\nContent-Length: 71\r\n\r\n{\"errorCode\":100,\"message\":\"Key not found\",\"cause\":\"/128\",\"index\":838}\r\n\r\n14:47:12.864765 IP coreos3.49934 > coreos1.2379: Flags [.], ack 243, win 473, options [nop,nop,TS val 898137 ecr 900662], length 0\r\nE..4R.@.@.q...zG..z...  K.h.=J..(....vN.....\r\n.6.\r\n14:47:12.865523 IP coreos3.49934 > coreos1.2379: Flags [F.], seq 147, ack 243, win 473, options [nop,nop,TS val 898138 ecr 900662], length 0\r\nE..4R.@.@.q...zG..z...  K.h.=J..(....vN.....\r\n.6.\r\n14:47:12.865624 IP coreos1.2379 > coreos3.49934: Flags [F.], seq 243, ack 148, win 470, options [nop,nop,TS val 900663 ecr 898138], length 0\r\nE..4e.@.@._...z...zG    K..J..(.h.>....vN.....\r\n.Z.\r\n14:47:12.865680 IP coreos3.49934 > coreos1.2379: Flags [.], ack 244, win 473, options [nop,nop,TS val 898138 ecr 900663], length 0\r\nE..4R.@.@.q...zG..z...  K.h.>J..)....vN.....\r\n.7.\r\n```\r\n\r\nI'm not sure why etcdctl on coreos3 initializes connection directly with coreos1 instead of using local etcd2 proxy. This issue is reproducible from time to time (~1 of 4)"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/153765393","html_url":"https://github.com/coreos/etcd/issues/3805#issuecomment-153765393","issue_url":"https://api.github.com/repos/coreos/etcd/issues/3805","id":153765393,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T15:36:34Z","updated_at":"2015-11-04T15:36:34Z","body":"@kayrus Sync cluster updates the cluster member list."}}