{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17675","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17675/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17675/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17675/events","html_url":"https://github.com/kubernetes/kubernetes/issues/17675","id":118455238,"number":17675,"title":"Restarting of kube-controller-manager shoudn't lead to re-emitting old endpoints events.","user":{"login":"ssergiienko","id":9850672,"avatar_url":"https://avatars.githubusercontent.com/u/9850672?v=3","gravatar_id":"","url":"https://api.github.com/users/ssergiienko","html_url":"https://github.com/ssergiienko","followers_url":"https://api.github.com/users/ssergiienko/followers","following_url":"https://api.github.com/users/ssergiienko/following{/other_user}","gists_url":"https://api.github.com/users/ssergiienko/gists{/gist_id}","starred_url":"https://api.github.com/users/ssergiienko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssergiienko/subscriptions","organizations_url":"https://api.github.com/users/ssergiienko/orgs","repos_url":"https://api.github.com/users/ssergiienko/repos","events_url":"https://api.github.com/users/ssergiienko/events{/privacy}","received_events_url":"https://api.github.com/users/ssergiienko/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/control-plane","name":"team/control-plane","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-11-23T19:34:23Z","updated_at":"2015-12-02T23:39:35Z","closed_at":null,"body":"We are heavily relying on /endpoints?watch=true functionality, and we are stuck why restarting kube-controller-manager leads to re-emitting old events? We always use ?resourceversions=xxxx but this doesn't solve the problem.\r\nWe also listen for /pods and /nodes events, but they are not affected by this bug.\r\n\r\nFor example, if we don't touch any thing in the cluster, and just restart kube-controller-manager.service then we have this almost after each restart:\r\n\r\nLogs of our application in form \"listener_function | current_resource_version | last_resource_version\"\r\n```\r\nprocess_endpoints_event 1294 1226\r\nprocess_endpoints_event 1295 1294\r\n==DELETED OBJ 1 ==\r\nprocess_endpoints_event 1296 1295\r\n==DELETED OBJ 2 ==\r\nprocess_endpoints_event 1297 1296\r\nprocess_endpoints_event 1298 1297\r\n==DELETED OBJ 3==\r\nprocess_endpoints_event 1299 1298\r\nprocess_endpoints_event 1300 1299\r\n==ADDED OBJ 1 ==\r\nprocess_endpoints_event 1301 1300\r\nprocess_endpoints_event 1302 1301\r\n==ADDED OBJ 2 ==\r\nprocess_endpoints_event 1303 1302\r\n==ADDED OBJ 3 ==\r\n```\r\nThis happends almost after each restart of kube-controller-manager (usually 1-3).\r\nEvents:\r\n```\r\n{u'object': {u'kind': u'Endpoints', u'subsets': []...\r\n{u'object': {u'kind': u'Endpoints', u'subsets': []...\r\n{u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u'10.254.87.2'\r\n{u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u'10.254.42.3'\r\n...\r\n```\r\n\r\nKubernetes v1.0.3;\r\n\r\nIs this a bug or we are doing something wrong?"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/161468145","html_url":"https://github.com/kubernetes/kubernetes/issues/17675#issuecomment-161468145","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17675","id":161468145,"user":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"created_at":"2015-12-02T23:39:35Z","updated_at":"2015-12-02T23:39:35Z","body":"I think what is going on is that endpoint controller is generating endpoints before it's finished loading the pods, and therefore it deletes all endpoints and then recreates them. Oops!\r\n\r\nSo you're not getting old watch entries, you're getting new ones. The problem is that they are being generated at all.\r\n\r\n@davidopp if you can't find someone to fix this I might be able to throw something together."}}