{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14870","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14870/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14870/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14870/events","html_url":"https://github.com/kubernetes/kubernetes/issues/14870","id":109175776,"number":14870,"title":"kube-proxy keeps adding duplicate iptables rules","user":{"login":"AdrianAbraham","id":13833219,"avatar_url":"https://avatars.githubusercontent.com/u/13833219?v=3","gravatar_id":"","url":"https://api.github.com/users/AdrianAbraham","html_url":"https://github.com/AdrianAbraham","followers_url":"https://api.github.com/users/AdrianAbraham/followers","following_url":"https://api.github.com/users/AdrianAbraham/following{/other_user}","gists_url":"https://api.github.com/users/AdrianAbraham/gists{/gist_id}","starred_url":"https://api.github.com/users/AdrianAbraham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AdrianAbraham/subscriptions","organizations_url":"https://api.github.com/users/AdrianAbraham/orgs","repos_url":"https://api.github.com/users/AdrianAbraham/repos","events_url":"https://api.github.com/users/AdrianAbraham/events{/privacy}","received_events_url":"https://api.github.com/users/AdrianAbraham/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"milestone":null,"comments":10,"created_at":"2015-09-30T20:45:04Z","updated_at":"2015-10-20T06:03:40Z","closed_at":null,"body":"I noticed that while connections between pods were usually nearly instant, sometimes they took a few seconds and they sometimes hung altogether, so I started poking around. I quickly found that `kube-proxy` is taking up between 80% - 100% of a CPU on each system that it's running on. The logs weren't very interesting, but then I checked the iptables rules:\r\n```\r\n# sudo iptables -L -t nat | wc -l\r\n164757\r\n```\r\n\r\nThe rules themselves:\r\n```\r\n# sudo iptables -L -t nat\r\nChain PREROUTING (policy ACCEPT 0 packets, 0 bytes)\r\n pkts bytes target     prot opt in     out     source               destination         \r\n    0     0 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ \r\n    1    60 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ \r\n    4   258 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ \r\n    6   547 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ \r\n   12  1632 KUBE-PORTALS-CONTAINER  all  --  any    any     anywhere             anywhere            /* handle ClusterIPs; NOTE: this must be before the NodePort rules */ \r\n...\r\n```\r\n\r\nThe `OUTPUT` chain is just as bad as the `PREROUTING` chain. The `KUBE-*` chains are all sane, having one rule per service port.\r\n\r\nFor now, I've set up a cron job to periodically flush `PREROUTING` and `OUTPUT` (and recreate the proper 4 rules to hopefully avoid any downtime), but this is obviously a poor workaround.\r\n\r\nI'm running release 1.0.6 on CentOS 6."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/149442591","html_url":"https://github.com/kubernetes/kubernetes/issues/14870#issuecomment-149442591","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/14870","id":149442591,"user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"created_at":"2015-10-20T06:03:40Z","updated_at":"2015-10-20T06:03:40Z","body":"Damn, the debug that is in there is not sufficient.  If I provided a binary\nfrom head, would you try it?\n\nOn Mon, Oct 19, 2015 at 10:54 PM, dolphyvn <notifications@github.com> wrote:\n\n> and it keep continuous adding this rule\n> I1020 12:51:58.061365 13041 iptables.go:193] running iptables -A [OUTPUT\n> -t nat -m addrtype --dst-type LOCAL -m comment --comment handle service\n> NodePorts; NOTE: this must be the last rule in the chain -j\n> KUBE-NODEPORT-HOST]\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/kubernetes/kubernetes/issues/14870#issuecomment-149441298>\n> .\n>\n"}}