{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7370","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7370/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7370/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7370/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/7370","id":71352131,"number":7370,"title":"iptables invocations fail if another process is modifying iptables rules","user":{"login":"sdodson","id":48161,"avatar_url":"https://avatars.githubusercontent.com/u/48161?v=3","gravatar_id":"","url":"https://api.github.com/users/sdodson","html_url":"https://github.com/sdodson","followers_url":"https://api.github.com/users/sdodson/followers","following_url":"https://api.github.com/users/sdodson/following{/other_user}","gists_url":"https://api.github.com/users/sdodson/gists{/gist_id}","starred_url":"https://api.github.com/users/sdodson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdodson/subscriptions","organizations_url":"https://api.github.com/users/sdodson/orgs","repos_url":"https://api.github.com/users/sdodson/repos","events_url":"https://api.github.com/users/sdodson/events{/privacy}","received_events_url":"https://api.github.com/users/sdodson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P2","name":"priority/P2","color":"fbca04"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/cluster","name":"team/cluster","color":"d2b48c"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-04-27T17:34:41Z","updated_at":"2015-04-29T17:30:25Z","closed_at":null,"body":"If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay.\r\n\r\nBelow, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying.\r\n\r\n```\r\nApr 27 11:40:20 ose3-master.example.com openshift-sdn-node[2058]: + iptables -t nat -D POSTROUTING -s 10.1.0.0/16 '!' -d 10.1.0.0/16 -j MASQUERADE\r\n...\r\nApr 27 11:40:20 ose3-master.example.com openshift-sdn-node[2058]: + iptables -I INPUT 2 -i lbr0 -m comment --comment 'traffic from docker' -j ACCEPT\r\nApr 27 11:40:21 ose3-master.example.com openshift-node[2237]: E0427 11:40:21.661746    2237 proxier.go:337] Failed to initialize iptables: error checking rule: exit status 4: Another app is currently holding the xtables lock. Perhaps you want to use the -w option?\r\nApr 27 11:40:21 ose3-master.example.com openshift-node[2237]: E0427 11:40:21.661912    2237 node.go:245] WARNING: Could not modify iptables.  iptables must be mutable by this process to use services.  Do you have root permissions?\r\nApr 27 11:40:22 ose3-master.example.com openshift-node[2237]: I0427 11:40:22.147405    2237 failing_service_config_proxy.go:19] Failed to properly wire up services.  This can happen if you forget to launch with permissions to iptables.  Access to the following services will be impaired: \"kubernetes-ro, router, database, frontend, ruby-example, docker-registry, kubernetes\"\r\n```\r\n\r\nRelated to https://github.com/openshift/origin/issues/1922"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/97512684","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/7370#issuecomment-97512684","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/7370","id":97512684,"user":{"login":"thockin","id":5595220,"avatar_url":"https://avatars.githubusercontent.com/u/5595220?v=3","gravatar_id":"","url":"https://api.github.com/users/thockin","html_url":"https://github.com/thockin","followers_url":"https://api.github.com/users/thockin/followers","following_url":"https://api.github.com/users/thockin/following{/other_user}","gists_url":"https://api.github.com/users/thockin/gists{/gist_id}","starred_url":"https://api.github.com/users/thockin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thockin/subscriptions","organizations_url":"https://api.github.com/users/thockin/orgs","repos_url":"https://api.github.com/users/thockin/repos","events_url":"https://api.github.com/users/thockin/events{/privacy}","received_events_url":"https://api.github.com/users/thockin/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T17:30:24Z","updated_at":"2015-04-29T17:30:24Z","body":"It does retry all up tables ops.  Unless it somehow gets pathologically\nphase locked with your agent, or there is some deeper bug.  We have e2e\ntests that prove you can delete tables and it will recreate them.\n\nAre you saying that it never retries (did you set -v=4?) or that every\nretry fails?\n\nOn RHEL the lock is the xtables unix socket lock in iptables to allow\nmultiple programs to update iptables concurrently. Other platforms may use\na flock since that is the default upstream.\n\nTo solve this kubernetes needs one more more of the following:\n\n   - pass the -w flag to iptables as scott and the logs mention (possibly\n   only available on some platforms). If kubernetes desides to wait for the\n   lock it will need a timeout.\n   - retry the firewall initialization quickly and maybe backoff if it\n   still can't get the lock\n\nIt's probably worth noting that if iptables on a platform doesn't have\niptables lock support concurrency is likely broken already in other ways\nfor that platform.\n\n—\nReply to this email directly or view it on GitHub\n<https://github.com/GoogleCloudPlatform/kubernetes/issues/7370#issuecomment-97506585>\n.\n"}}