{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18100","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18100/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18100/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18100/events","html_url":"https://github.com/kubernetes/kubernetes/issues/18100","id":120022374,"number":18100,"title":"Service IP NAT Issue","user":{"login":"ddispaltro","id":46037,"avatar_url":"https://avatars.githubusercontent.com/u/46037?v=3","gravatar_id":"","url":"https://api.github.com/users/ddispaltro","html_url":"https://github.com/ddispaltro","followers_url":"https://api.github.com/users/ddispaltro/followers","following_url":"https://api.github.com/users/ddispaltro/following{/other_user}","gists_url":"https://api.github.com/users/ddispaltro/gists{/gist_id}","starred_url":"https://api.github.com/users/ddispaltro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ddispaltro/subscriptions","organizations_url":"https://api.github.com/users/ddispaltro/orgs","repos_url":"https://api.github.com/users/ddispaltro/repos","events_url":"https://api.github.com/users/ddispaltro/events{/privacy}","received_events_url":"https://api.github.com/users/ddispaltro/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2015-12-02T19:53:39Z","updated_at":"2015-12-29T23:45:57Z","closed_at":"2015-12-03T00:27:38Z","body":"I am using kube 1.1.2, and I ran into an issue with a Service IP being unable to NAT to the Pod it's connected to. I am using the `iptables` proxy instead of user mode. I am using AWS through the kube-aws stuff from coreos, however, I don't think it's a problem with the provisioning scripts from them, I've made everything as permissive as possible.  It's also using VPC's natively vs Flannel.  I apologize in advance if this is local to those scripts, there are a lot of moving parts and was hoping this might be the best place to triage it.\r\n\r\nHere's what I've observed:\r\n\r\nget service:\r\n```console\r\nNAMESPACE     NAME                  CLUSTER_IP   EXTERNAL_IP   PORT(S)             SELECTOR                AGE\r\nkube-system   heapster              10.3.0.77    <none>        80/TCP              k8s-app=heapster        7h\r\n```\r\n\r\nget endpoints:\r\n```console\r\nNAMESPACE     NAME                  ENDPOINTS                     AGE\r\nkube-system   heapster              10.2.1.3:8082                 7h\r\n```\r\n\r\nssh to heapster, use service ip\r\n```console\r\n~ ❯❯❯ kubectl --namespace=kube-system exec heapster-v10-gcvri -it bash\r\nroot@heapster-v10-gcvri:/go# curl 10.3.0.77\r\n```\r\nthis times out\r\n\r\nssh to heapster, then use endpoint ip\r\n```console\r\n~ ❯❯❯ kubectl --namespace=kube-system exec heapster-v10-gcvri -it bash\r\nroot@heapster-v10-gcvri:/go# curl 10.2.1.3:8082\r\n<a href=\"/validate/\">Temporary Redirect</a>.\r\n```\r\n\r\nssh to random container on same machine or not, just not in the same pod.\r\n```console\r\n~ ❯❯❯ kubectl exec busybox -it sh\r\nsh: shopt: not found\r\n[ root@busybox:/ ]$ curl 10.3.0.77\r\n<a href=\"/validate/\">Temporary Redirect</a>.\r\n```\r\n\r\nNext step was to dig in:\r\n* I spun up a new container that wasn't doing anything and made a service ip for port `3030` to localize the problem.  I observed the same thing.\r\n* I started tracing the iptables on the host the container is running \r\n\r\n```console\r\n$ sudo iptables -t raw -A PREROUTING -p tcp --dport 3030  -j TRACE\r\n$ sudo iptables -t raw -A OUTPUT -p tcp --dport 3030  -j TRACE\r\n```\r\n* I noticed it was never hitting the FORWARD or POSTROUTING rules\r\n* Then I booted up tcpdump capturing `cbr0` and it started working, I quit tcpdump and it stopped working.\r\n* I explicitly set the `cbr0` in promiscuous mode and it worked.  Below are two stanzas, the first is promisc mode off the second one is promisc on, which you can see includes the masquerade + forward call.  The mac's are different so it makes sense why promisc would work, I am no iptables guru so I am having a hard time pinpointing it.\r\n\r\n```\r\n[74345.384116] TRACE: raw:PREROUTING:policy:2 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307)\r\n[74345.385056] TRACE: nat:PREROUTING:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307)\r\n[74345.385056] TRACE: nat:KUBE-SERVICES:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307)\r\n[74345.385056] TRACE: nat:KUBE-SVC-5R7YSNSPWU55JWWD:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307)\r\n[74345.385056] TRACE: nat:KUBE-SEP-MXULWDIECKL2JEAS:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307)\r\n[74345.385056] TRACE: nat:KUBE-SEP-MXULWDIECKL2JEAS:rule:2 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=27481 DF PROTO=TCP SPT=43592 DPT=3030 SEQ=1144889564 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A0469D7C80000000001030307) MARK=0x4d415351\r\n```\r\n\r\n```\r\n[74355.169651] device cbr0 entered promiscuous mode\r\n[74357.596833] TRACE: raw:PREROUTING:policy:2 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307)\r\n[74357.615212] TRACE: nat:PREROUTING:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307)\r\n[74357.633793] TRACE: nat:KUBE-SERVICES:rule:5 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307)\r\n[74357.652254] TRACE: nat:KUBE-SVC-5R7YSNSPWU55JWWD:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307)\r\n[74357.670057] TRACE: nat:KUBE-SEP-MXULWDIECKL2JEAS:rule:1 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307)\r\n[74357.688025] TRACE: nat:KUBE-SEP-MXULWDIECKL2JEAS:rule:2 IN=cbr0 OUT= PHYSIN=vethe1a452f MAC=6e:b1:3d:c0:01:6b:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.3.0.207 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307) MARK=0x4d415351\r\n[74357.706498] TRACE: filter:FORWARD:policy:1 IN=cbr0 OUT=cbr0 PHYSIN=vethe1a452f MAC=02:42:0a:02:01:04:02:42:0a:02:01:04:08:00 SRC=10.2.1.4 DST=10.2.1.4 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307) MARK=0x4d415351\r\n[74357.725498] TRACE: nat:POSTROUTING:rule:2 IN= OUT=cbr0 PHYSIN=vethe1a452f SRC=10.2.1.4 DST=10.2.1.4 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=38208 DF PROTO=TCP SPT=43612 DPT=3030 SEQ=2065142332 ACK=0 WINDOW=26883 RES=0x00 SYN URGP=0 OPT (020423010402080A046A077C0000000001030307) MARK=0x4d415351\r\n[74360.411875] device cbr0 left promiscuous mode\r\n```\r\n\r\nAny insights on what's causing this?\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/167900961","html_url":"https://github.com/kubernetes/kubernetes/issues/18100#issuecomment-167900961","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/18100","id":167900961,"user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"created_at":"2015-12-29T23:45:57Z","updated_at":"2015-12-29T23:45:57Z","body":"I think the fix here is just to also check for \"noqueue\" during setup and\ntreat the results as if that interface had no data at all.\n\nI'm swamped right now - got time?  Easy fix...\n\nOn Tue, Dec 29, 2015 at 3:06 PM, Rudi C <notifications@github.com> wrote:\n\n> Actually, even on 835.9.0, which has no issues, tc is version ss150210.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kubernetes/kubernetes/issues/18100#issuecomment-167897221>\n> .\n>\n"}}