{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9382","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9382/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9382/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9382/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/9382","id":86020045,"number":9382,"title":"Indefinite retries of failing objects","user":{"login":"brendandburns","id":5751682,"avatar_url":"https://avatars.githubusercontent.com/u/5751682?v=3","gravatar_id":"","url":"https://api.github.com/users/brendandburns","html_url":"https://github.com/brendandburns","followers_url":"https://api.github.com/users/brendandburns/followers","following_url":"https://api.github.com/users/brendandburns/following{/other_user}","gists_url":"https://api.github.com/users/brendandburns/gists{/gist_id}","starred_url":"https://api.github.com/users/brendandburns/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brendandburns/subscriptions","organizations_url":"https://api.github.com/users/brendandburns/orgs","repos_url":"https://api.github.com/users/brendandburns/repos","events_url":"https://api.github.com/users/brendandburns/events{/privacy}","received_events_url":"https://api.github.com/users/brendandburns/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0-candidate","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/12/labels","id":1086011,"number":12,"title":"v1.0-candidate","description":"These are candidates for inclusion to v1.0 but will be pushed to v1.0-bubble if they're triaged as not meeting the v1.0 bar as a required launch blocker.","creator":{"login":"alex-mohr","id":8729097,"avatar_url":"https://avatars.githubusercontent.com/u/8729097?v=3","gravatar_id":"","url":"https://api.github.com/users/alex-mohr","html_url":"https://github.com/alex-mohr","followers_url":"https://api.github.com/users/alex-mohr/followers","following_url":"https://api.github.com/users/alex-mohr/following{/other_user}","gists_url":"https://api.github.com/users/alex-mohr/gists{/gist_id}","starred_url":"https://api.github.com/users/alex-mohr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alex-mohr/subscriptions","organizations_url":"https://api.github.com/users/alex-mohr/orgs","repos_url":"https://api.github.com/users/alex-mohr/repos","events_url":"https://api.github.com/users/alex-mohr/events{/privacy}","received_events_url":"https://api.github.com/users/alex-mohr/received_events","type":"User","site_admin":false},"open_issues":15,"closed_issues":41,"state":"open","created_at":"2015-04-27T19:03:36Z","updated_at":"2015-06-08T04:31:54Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-06-08T04:31:32Z","updated_at":"2015-06-08T15:54:42Z","closed_at":null,"body":"From:\r\nUday Kumar Raju ruddarraju <rudaykumarraju@gmail.com>\r\n\r\nI have run into a wierd situation using the FIFO queue in the ServiceController. We (at ebay) have modified the openstack cloud provider to suit our installation of openstack, where all the Pod IPs are reachable outside the cluster. So we have changed out cloudprovider to read the pods belonging to a service and create a tcp load balancer with pod IPs as the members. We saw a wierd case recently with the way ListWatcher is enabled.\r\n\r\nHere is the pseudo code of our CreateTCPLoadbalacer:\r\n\r\nfunc CreateTCPLoadbalancer(lbName) {\r\nGetAllPodsUnderService() // A call to API server to get all the pods with the same selector as the service\r\nCallOpenstackToCreateLBPool(PodIPs, method)\r\n}\r\n\r\nYou can see this is a little different from all the public providers where, none of the other providers have a dependency back to the apiserver during this operation.\r\n\r\nI have run into a situation where servicecontroller indefinitely tries to call the CreateTCPLoadbalancer even when it fails everytime. My failure case happened to be this:\r\n\r\nCalled Create Service for a set of Pods. Created a delta on the fifo queue\r\nThe CreateTCPLoadbalancer call failed due to a connection reset on the Openstack endpoint due to a network issue, so the event has been put back on the FIFO queue as the CreateTCPLoadbalancer failure is always considered to be a retriable failure. \r\nI deleted the service to try again. This has been pushed in as a delta again (while the api server has already deleted the service from etcd and hence querying the apiserver after this would always return service not found)\r\nNow my FIFO queue looks like this  (Add Service) <- (Delete Service)\r\nSince my Add Service (CreateTCPLoadbalancer) always relies on the service object (in my cloud provider) in the apiserver, this call would fail every single time hence forth. And is pushed back to the queue everytime (retriable is True) and the delete event does not get processed at all. \r\n\r\nI was able to reproduce this by inducing an error in the createTCPLB call everytime. I know there can be several ways to handle this but wanted to check with you guys first if my analysis on this problem makes any sense."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/110051806","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/9382#issuecomment-110051806","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/9382","id":110051806,"user":{"login":"uruddarraju","id":2271582,"avatar_url":"https://avatars.githubusercontent.com/u/2271582?v=3","gravatar_id":"","url":"https://api.github.com/users/uruddarraju","html_url":"https://github.com/uruddarraju","followers_url":"https://api.github.com/users/uruddarraju/followers","following_url":"https://api.github.com/users/uruddarraju/following{/other_user}","gists_url":"https://api.github.com/users/uruddarraju/gists{/gist_id}","starred_url":"https://api.github.com/users/uruddarraju/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uruddarraju/subscriptions","organizations_url":"https://api.github.com/users/uruddarraju/orgs","repos_url":"https://api.github.com/users/uruddarraju/repos","events_url":"https://api.github.com/users/uruddarraju/events{/privacy}","received_events_url":"https://api.github.com/users/uruddarraju/received_events","type":"User","site_admin":false},"created_at":"2015-06-08T15:54:42Z","updated_at":"2015-06-08T15:54:42Z","body":"https://groups.google.com/forum/#!topic/google-containers/-0tL-_lOV-Q"}}