{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17186","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17186/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17186/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17186/events","html_url":"https://github.com/kubernetes/kubernetes/issues/17186","id":116665576,"number":17186,"title":"Out of resource killing","user":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/isolation","name":"area/isolation","color":"aba420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/scheduling","name":"area/scheduling","color":"ab6420"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/priority/P2","name":"priority/P2","color":"fbca04"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/node","name":"team/node","color":"d2b48c"},{"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/redhat","name":"team/redhat","color":"d2b48c"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15","html_url":"https://github.com/kubernetes/kubernetes/milestones/v1.2","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/15/labels","id":1278499,"number":15,"title":"v1.2","description":"https://github.com/kubernetes/kubernetes/wiki/Release-1.2","creator":{"login":"bgrant0607","id":7725777,"avatar_url":"https://avatars.githubusercontent.com/u/7725777?v=3","gravatar_id":"","url":"https://api.github.com/users/bgrant0607","html_url":"https://github.com/bgrant0607","followers_url":"https://api.github.com/users/bgrant0607/followers","following_url":"https://api.github.com/users/bgrant0607/following{/other_user}","gists_url":"https://api.github.com/users/bgrant0607/gists{/gist_id}","starred_url":"https://api.github.com/users/bgrant0607/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgrant0607/subscriptions","organizations_url":"https://api.github.com/users/bgrant0607/orgs","repos_url":"https://api.github.com/users/bgrant0607/repos","events_url":"https://api.github.com/users/bgrant0607/events{/privacy}","received_events_url":"https://api.github.com/users/bgrant0607/received_events","type":"User","site_admin":false},"open_issues":54,"closed_issues":17,"state":"open","created_at":"2015-08-29T03:54:16Z","updated_at":"2015-12-05T18:24:49Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2015-11-12T23:28:26Z","updated_at":"2015-12-06T15:54:47Z","closed_at":null,"body":"Forked from #15673, #16889, #13006, #14943, and other issues/PRs.\r\n\r\nWe need a model for out of resource killing on overcommitted nodes.\r\n\r\ncc @dchen1107 @vishh @davidopp @derekwaynecarr "},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/162324881","html_url":"https://github.com/kubernetes/kubernetes/issues/17186#issuecomment-162324881","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17186","id":162324881,"user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"created_at":"2015-12-06T15:54:47Z","updated_at":"2015-12-06T15:54:47Z","body":"@bgrant0607 - agreed.\r\n\r\nAt the node side, \r\n* operator can specify monitoring interval (N) for polling resource usage in user-space\r\n* operator can specify a soft (S), hard (H) threshold and duration (D) for each compute resource C\r\n* kubelet control loop at interval N monitors system-wide compute resource usage via cAdvisor\r\n * rank usage of C grouped by QoS tier (T) at pod scope\r\n * if usage of C exceeds S, but is less than H for period D, record Event, update NodeCondition\r\n * if usage of C exceeds H for period D, record Event, update NodeCondition, initiate eviction to reclaim C\r\n\r\nIt may be easier to model duration (D) as basically the number of consecutive monitoring intervals rather than an exact period of time.  For example, if N = 10s, then D=2 means if C exceeded its usage for 20s, then do something.\r\n\r\nI think soft/hard thresholds would be appreciated by our operators.\r\n\r\nWe need to evict entire pods by QoS tier (T), and there would be a few different options here for how we find a candidate pod (P) to evict.  I think each compute resource C would have its own distinct priority function F for how it identifies pods to target.\r\n\r\n* F=findCandidatePodMemory\r\n * T=BestEffort, find pod with largest consumer of C\r\n * T=Burstable, find pod with max(usage(c) - request(c)\r\n * T=Guaranteed, select most recently scheduled\r\n * in-case of tie, always choose most recently scheduled\r\n* fail P\r\n* check if C is blow T, else repeat\r\n\r\nOn the scheduler side we could then use the information to get smarter to do back-off.\r\n\r\nMy near-term interest is around the node, and low memory situations, which I think has a simpler priority function.\r\n\r\nWill send a proposal we can evolve.\r\n\r\nThanks!"}}