{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3216","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3216/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3216/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3216/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3216","id":53422017,"number":3216,"title":"kube-proxy doesn't recover from outdated events","user":{"login":"smreed","id":100176,"avatar_url":"https://avatars.githubusercontent.com/u/100176?v=3","gravatar_id":"","url":"https://api.github.com/users/smreed","html_url":"https://github.com/smreed","followers_url":"https://api.github.com/users/smreed/followers","following_url":"https://api.github.com/users/smreed/following{/other_user}","gists_url":"https://api.github.com/users/smreed/gists{/gist_id}","starred_url":"https://api.github.com/users/smreed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smreed/subscriptions","organizations_url":"https://api.github.com/users/smreed/orgs","repos_url":"https://api.github.com/users/smreed/repos","events_url":"https://api.github.com/users/smreed/events{/privacy}","received_events_url":"https://api.github.com/users/smreed/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/kind%2Fbug","name":"kind/bug","color":"ff0000"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority%2FP1","name":"priority/P1","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"lavalamp","id":647318,"avatar_url":"https://avatars.githubusercontent.com/u/647318?v=3","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"milestone":null,"comments":36,"created_at":"2015-01-05T17:41:29Z","updated_at":"2015-01-12T16:15:50Z","closed_at":null,"body":"After running a 0.7.2 cluster in GCE over the weekend, service connections seemed flaky. \r\n\r\nThe following lines are recurring in the kube-proxy log on all minions in my cluster. \r\n\r\nW0105 17:28:25.587865    5051 api.go:188] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/endpoints ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [578/235]) [1577] Reason: Details:<nil> Code:0}\r\n\r\nW0105 00:17:07.886554    5042 api.go:124] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/services ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [348/234]) [1347] Reason: Details:<nil> Code:0}\r\n\r\nThe indices in the error message do not change over time.\r\n\r\nkube-proxy seems to be stuck in an endless loop on this condition and seems to fail to get service and endpoint updates."},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/69595097","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/3216#issuecomment-69595097","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/3216","id":69595097,"user":{"login":"smreed","id":100176,"avatar_url":"https://avatars.githubusercontent.com/u/100176?v=3","gravatar_id":"","url":"https://api.github.com/users/smreed","html_url":"https://github.com/smreed","followers_url":"https://api.github.com/users/smreed/followers","following_url":"https://api.github.com/users/smreed/following{/other_user}","gists_url":"https://api.github.com/users/smreed/gists{/gist_id}","starred_url":"https://api.github.com/users/smreed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smreed/subscriptions","organizations_url":"https://api.github.com/users/smreed/orgs","repos_url":"https://api.github.com/users/smreed/repos","events_url":"https://api.github.com/users/smreed/events{/privacy}","received_events_url":"https://api.github.com/users/smreed/received_events","type":"User","site_admin":false},"created_at":"2015-01-12T16:15:50Z","updated_at":"2015-01-12T16:15:50Z","body":"Running v0.8.0 and still seeing kube-proxy get into this state. The following two messages are being repeated in the logs. The etcd indexes are not changing.\r\n\r\n    W0112 16:14:27.835652    3169 api.go:198] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/endpoints ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [10781/5338]) [11780] Reason: Details:<nil> Code:0}\r\n    W0112 16:14:54.260622    3169 api.go:129] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/services ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [10781/3907]) [11780] Reason: Details:<nil> Code:0}\r\n    W0112 16:14:59.510727    3169 api.go:198] Got error status on WatchEndpoints channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/endpoints ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [10781/5338]) [11780] Reason: Details:<nil> Code:0}\r\n    W0112 16:15:26.213788    3169 api.go:129] Got error status on WatchServices channel: &{TypeMeta:{Kind: APIVersion:} ListMeta:{SelfLink:/api/v1beta1/services ResourceVersion:} Status:Failure Message:401: The event in requested index is outdated and cleared (the requested history has been cleared [10781/3907]) [11780] Reason: Details:<nil> Code:0}"}}