{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16600","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16600/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16600/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16600/events","html_url":"https://github.com/kubernetes/kubernetes/issues/16600","id":114322773,"number":16600,"title":"Event recorder is too strict in counting cached events, can OOM a server","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P1","name":"priority/P1","color":"eb6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/CSI","name":"team/CSI","color":"d2b48c"}],"state":"open","locked":false,"assignee":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"milestone":null,"comments":13,"created_at":"2015-10-30T17:57:16Z","updated_at":"2015-11-03T02:05:44Z","closed_at":null,"body":"Example scenario:\r\n\r\n1. Create a replication controller A\r\n2. Set A.Replicas to N+diff\r\n3. A attempts to create diff pods, and is failed for each\r\n4. For each failure in diff, A records an event with the failure\r\n\r\nProblem:\r\nIf the error message across each failure is unique, the number of events produced can unexpectedly induce an OOM on the master by overloading etcd.  Common reasons for error messages varying slightly would include the following:\r\n\r\na) the error message including the name of the pod\r\nb) the error message including a timestamp\r\nc) etc.\r\n\r\nIn our particular instance, we encountered (a) and it resulted in 175488 events being spawned on a rather small deployment that induced OOM scenarios for etcd and killed the node.\r\n\r\n**Proposal**\r\nEvent.Message should not be used in the event_cache algorithm.\r\n\r\nIt's too easy for any controller to go rogue, and it can confuse the heck out of users why seemingly common errors repeat in event output.  In my case, when looking at 77000 of those events in one namespace, it stumped me to figure out the difference ;-)\r\n\r\nInstead, I think event keys should be the following:\r\n\r\n```\r\nfunc getEventKey(event *api.Event) string {\r\n\treturn event.Source.Component +\r\n\t\tevent.Source.Host +\r\n\t\tevent.InvolvedObject.Kind +\r\n\t\tevent.InvolvedObject.Namespace +\r\n\t\tevent.InvolvedObject.Name +\r\n\t\tstring(event.InvolvedObject.UID) +\r\n\t\tevent.InvolvedObject.APIVersion +\r\n\t\tevent.Reason\r\n}\r\n```\r\n\r\nAnd event.Message should just update to the latest observed message.\r\n\r\nThoughts?\r\n\r\n@lavalamp @erictune @wojtek-t @bgrant0607 @smarterclayton "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/153217057","html_url":"https://github.com/kubernetes/kubernetes/issues/16600#issuecomment-153217057","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/16600","id":153217057,"user":{"login":"saad-ali","id":10052848,"avatar_url":"https://avatars.githubusercontent.com/u/10052848?v=3","gravatar_id":"","url":"https://api.github.com/users/saad-ali","html_url":"https://github.com/saad-ali","followers_url":"https://api.github.com/users/saad-ali/followers","following_url":"https://api.github.com/users/saad-ali/following{/other_user}","gists_url":"https://api.github.com/users/saad-ali/gists{/gist_id}","starred_url":"https://api.github.com/users/saad-ali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saad-ali/subscriptions","organizations_url":"https://api.github.com/users/saad-ali/orgs","repos_url":"https://api.github.com/users/saad-ali/repos","events_url":"https://api.github.com/users/saad-ali/events{/privacy}","received_events_url":"https://api.github.com/users/saad-ali/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T02:05:44Z","updated_at":"2015-11-03T02:05:44Z","body":"> I am coding this up now. The only enhancement to @erictune proposal is to add a time interval for event correlation to know how long to wait between occurrences before we treat the incoming event as new again. Defaulting that to 1 hr seems fine on the client side event correlation.\r\n\r\nExactly what I was thinking. This proposal SGTM"}}