{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6513","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6513/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6513/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6513/events","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6513","id":66865415,"number":6513,"title":"Watch shouldn't break every 5 minutes","user":{"login":"wojtek-t","id":10743879,"avatar_url":"https://avatars.githubusercontent.com/u/10743879?v=3","gravatar_id":"","url":"https://api.github.com/users/wojtek-t","html_url":"https://github.com/wojtek-t","followers_url":"https://api.github.com/users/wojtek-t/followers","following_url":"https://api.github.com/users/wojtek-t/following{/other_user}","gists_url":"https://api.github.com/users/wojtek-t/gists{/gist_id}","starred_url":"https://api.github.com/users/wojtek-t/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wojtek-t/subscriptions","organizations_url":"https://api.github.com/users/wojtek-t/orgs","repos_url":"https://api.github.com/users/wojtek-t/repos","events_url":"https://api.github.com/users/wojtek-t/events{/privacy}","received_events_url":"https://api.github.com/users/wojtek-t/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/area/performance","name":"area/performance","color":"fef2c0"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/misc/perf-1.0","name":"misc/perf-1.0","color":"006b75"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/labels/team/master","name":"team/master","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/milestones/v1.0","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/milestones/6/labels","id":969317,"number":6,"title":"v1.0","description":"Issues that need to be resolved for v1.0 of Kubernetes.","creator":{"login":"goltermann","id":9358478,"avatar_url":"https://avatars.githubusercontent.com/u/9358478?v=3","gravatar_id":"","url":"https://api.github.com/users/goltermann","html_url":"https://github.com/goltermann","followers_url":"https://api.github.com/users/goltermann/followers","following_url":"https://api.github.com/users/goltermann/following{/other_user}","gists_url":"https://api.github.com/users/goltermann/gists{/gist_id}","starred_url":"https://api.github.com/users/goltermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goltermann/subscriptions","organizations_url":"https://api.github.com/users/goltermann/orgs","repos_url":"https://api.github.com/users/goltermann/repos","events_url":"https://api.github.com/users/goltermann/events{/privacy}","received_events_url":"https://api.github.com/users/goltermann/received_events","type":"User","site_admin":false},"open_issues":113,"closed_issues":161,"state":"open","created_at":"2015-02-06T20:27:16Z","updated_at":"2015-04-07T16:51:53Z","due_on":null,"closed_at":null},"comments":14,"created_at":"2015-04-07T11:55:56Z","updated_at":"2015-04-07T19:01:27Z","closed_at":null,"body":"Currently Watch is broken every 5 minutes. It is then \"recreated\", which sometimes requires listing all elements under watch (if there were more than 1000 \"uninteresting\" etcd writes since then).\r\n\r\nI digged deeper into it and it seems this is not related to etcd itself. What happens is:\r\n1. a component (e.g. scheduler) starts watch sending http request to apiserver\r\n2. apiserver initiates a watch on etcd\r\n3. after 5 minutes, the request to apiserver finished (this is something I don't fully understand - is this a timeout?)\r\n4. this triggers \"canceling\" the request to etcd\r\n\r\ncc @fgrzadkowski "},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/comments/90698779","html_url":"https://github.com/GoogleCloudPlatform/kubernetes/issues/6513#issuecomment-90698779","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/kubernetes/issues/6513","id":90698779,"user":{"login":"davidopp","id":8264113,"avatar_url":"https://avatars.githubusercontent.com/u/8264113?v=3","gravatar_id":"","url":"https://api.github.com/users/davidopp","html_url":"https://github.com/davidopp","followers_url":"https://api.github.com/users/davidopp/followers","following_url":"https://api.github.com/users/davidopp/following{/other_user}","gists_url":"https://api.github.com/users/davidopp/gists{/gist_id}","starred_url":"https://api.github.com/users/davidopp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidopp/subscriptions","organizations_url":"https://api.github.com/users/davidopp/orgs","repos_url":"https://api.github.com/users/davidopp/repos","events_url":"https://api.github.com/users/davidopp/events{/privacy}","received_events_url":"https://api.github.com/users/davidopp/received_events","type":"User","site_admin":false},"created_at":"2015-04-07T19:01:27Z","updated_at":"2015-04-07T19:01:27Z","body":"@smarterclayton can you explain this attack mechanism a bit more? Regardless of watch timeout, can't someone sucessfully attack by running a lot of clients that keep renewing? Statistically some non-attack clients will get through if the attackers keep having to renew, but if the attacker runs enough clients then I think they can keep that number close to zero. \r\n"}}