{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/47424743","id":47424743,"diff_hunk":"@@ -0,0 +1,181 @@\n+# Kubelet and systemd interaction\n+\n+**Author**: Derek Carr (@derekwaynecarr)\n+\n+**Status**: Proposed\n+\n+This document presents a specification for how the `kubelet` interfaces with `systemd`.\n+\n+## Motivation\n+\n+Many Linux distributions have either adopted, or plan to adopt `systemd` as their init system.\n+\n+This document describes a series of enhancements that should be made to the `kubelet` to better integrate with these distributions.\n+\n+## systemd units\n+\n+To help understand this proposal, we first provide a brief summary of `systemd` behavior.\n+\n+`systemd` manages a hierarchy of `slice`, `scope`, and `service` units.\n+\n+* `service` - application on the server that is launched by `systemd`; how it should start/stop; when it should be started; under what \n+circumstances it should be restarted; and any resource controls that should be applied to it.\n+* `scope` - a process or group of processes which are not launched by `systemd` (i.e. fork), like a service, resource controls may be applied\n+* `slice` - organizes a hierarchy in which `scope` and `service` units are placed.  a `slice` may contain `slice`, `scope`, or `service` units;\n+processes are attached to `service` and `scope` units only, not to `slices`. The hierarchy is intended to be unified, meaning a process should\n+only belong to a single leaf node.\n+\n+## cgroup hierarchy: orthogonal versus unified hierarchies and cgroupfs single writer\n+\n+Classical `cgroup` hierarchies were orthogonal per resource group controller, and a process may exist in different parts of the hierarchy.\n+\n+For example, a process `p1` could exist in each of the following at the same time:\n+\n+* `/sys/fs/cgroup/cpu/important/`\n+* `/sys/fs/cgroup/memory/unimportant/`\n+* `/sys/fs/cgroup/cpuacct/unimportant/`\n+\n+In addition, controllers for one resource group could depend on another in ways that were not always obvious.\n+\n+For example, the `cpu` controller depends on the `cpuacct` controller yet they were treated separately.\n+\n+Many found it confusing for a single process to belong to different nodes in the `cgroup` hierarchy across controllers.\n+\n+The Kernel direction for `cgroup` support is to move toward a unified `cgroup` hierarchy, where the per-controller hierarchies are eliminated in\n+favor of hierarchies like the following:\n+\n+* `/sys/fs/cgroup/important/`\n+* `/sys/fs/cgroup/unimportant/`\n+\n+In a unified hierarchy, a process should only belong to a single node in the `cgroup` tree.\n+\n+The Kernel direction for `cgroup` management is to promote a single-writer model rather than allowing multiple processes to independently\n+write to parts of the file-system.\n+\n+## cgroup management with systemd and container runtimes\n+\n+By default, `systemd` defines three slices:\n+\n+* `system.slice` - contains system and application daemons\n+* `user.slice` - user login related processes\n+* `machine.slice` - virtual machines or containers that have init systems (not application space containers)","path":"docs/proposals/kubelet-systemd.md","position":61,"original_position":61,"commit_id":"47c56b57bb92e16a0b9e17ed5a76d7bfa523a6bb","original_commit_id":"47c56b57bb92e16a0b9e17ed5a76d7bfa523a6bb","user":{"login":"jonboulle","id":2601015,"avatar_url":"https://avatars.githubusercontent.com/u/2601015?v=3","gravatar_id":"","url":"https://api.github.com/users/jonboulle","html_url":"https://github.com/jonboulle","followers_url":"https://api.github.com/users/jonboulle/followers","following_url":"https://api.github.com/users/jonboulle/following{/other_user}","gists_url":"https://api.github.com/users/jonboulle/gists{/gist_id}","starred_url":"https://api.github.com/users/jonboulle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonboulle/subscriptions","organizations_url":"https://api.github.com/users/jonboulle/orgs","repos_url":"https://api.github.com/users/jonboulle/repos","events_url":"https://api.github.com/users/jonboulle/events{/privacy}","received_events_url":"https://api.github.com/users/jonboulle/received_events","type":"User","site_admin":false},"body":"@derekwaynecarr There are a few things going on here, and i don't really want to get derailed into a discussion about long term division of responsibilities between kubelet/systemd, etc.. just yet ;-)\r\n\r\nBut to try nail down the concepts for a minute: Kubernetes (really, the kubelet) is responsible for launching *pods*, and ideally it delegates everything involved therein to the container runtime. In the case of Docker, since a pod is an ad hoc assembly of different individual containers, the kubelet needs to manage the individual containers, and the pod can't be meaningfully said to exist outside of the context of the kubelet; but in the case of a container runtime like rkt, where the pod is launched as a first-class citizen, it _does_ have an external lifecycle and identity. In this circumstance, I think it's perfectly appropriate for pods and pod-like entities [1] to register themselves with systemd-machined and integrate with it wherever possible - including living under `machine.slice`. (Perhaps unsurprisingly, we do that by default for pods in rkt when systemd-machined is available.) This provides a unified, stable interface that anyone can use to query/manage various aspects of the pod (which is particularly handy with rkt since we don't expose any management daemon).\r\n\r\nAs to whether this is appropriate for **all** pods that Kubernetes launches: I would argue, yes, in the long term. But like I said, that derails into a much larger discussion about division of responsibilities between kubelet/systemd, etc. that I don't think we need to get into right here. I would, however, push back against the idea that `machine.slice` is for the domain of things like libvirt; to begin with, `system-nspawn` registers ALL containers it invokes with `systemd-machined` by default; and among other examples, dwalsh has long pushed for having Docker (and now runc) register with it too. AIUI in every case that registration creates a scope in `machine.slice`.\r\n\r\nTo get back to the original issue, though: I still feel strongly we should defer to systemd concepts and nomenclature when discussing these slices. I'm happy for you to add a caveat that \"Kubelet does not directly launch pods in this slice\" or similar if you want to be very explicit about the current behaviour.\r\n\r\n[1]: I'm using \"pod-like entities\" to refer to the middle of the spectrum between \"single process containers\" and \"virtual machines\"; in [Lennart's parlance](http://www.freedesktop.org/wiki/Software/systemd/machined/), this is \"OS containers\", although I'm not hugely fond of that term","created_at":"2015-12-12T01:52:06Z","updated_at":"2015-12-12T01:52:06Z","html_url":"https://github.com/kubernetes/kubernetes/pull/17688#discussion_r47424743","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/47424743"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17688#discussion_r47424743"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688","id":51592494,"html_url":"https://github.com/kubernetes/kubernetes/pull/17688","diff_url":"https://github.com/kubernetes/kubernetes/pull/17688.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/17688.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17688","number":17688,"state":"open","locked":false,"title":"systemd node spec proposal","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"body":"The following outlines changes that I want to make to the ```kubelet``` in order to better integrate with ```systemd``` systems, and to better isolate containers in their own ```cgroup``` based on the qos tier.\r\n\r\nI think this is a precursor to getting more intelligent low compute resource eviction.\r\n\r\n/cc @smarterclayton @ncdc @pmorie @dchen1107 @vishh @bgrant0607 ","created_at":"2015-11-23T22:50:46Z","updated_at":"2015-12-12T01:52:06Z","closed_at":null,"merged_at":null,"merge_commit_sha":"69d89994f28899ad24c77b9f6caa9317f9525452","assignee":{"login":"dchen1107","id":7740897,"avatar_url":"https://avatars.githubusercontent.com/u/7740897?v=3","gravatar_id":"","url":"https://api.github.com/users/dchen1107","html_url":"https://github.com/dchen1107","followers_url":"https://api.github.com/users/dchen1107/followers","following_url":"https://api.github.com/users/dchen1107/following{/other_user}","gists_url":"https://api.github.com/users/dchen1107/gists{/gist_id}","starred_url":"https://api.github.com/users/dchen1107/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dchen1107/subscriptions","organizations_url":"https://api.github.com/users/dchen1107/orgs","repos_url":"https://api.github.com/users/dchen1107/repos","events_url":"https://api.github.com/users/dchen1107/events{/privacy}","received_events_url":"https://api.github.com/users/dchen1107/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/17688/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/47c56b57bb92e16a0b9e17ed5a76d7bfa523a6bb","head":{"label":"derekwaynecarr:systemd_nodespec","ref":"systemd_nodespec","sha":"47c56b57bb92e16a0b9e17ed5a76d7bfa523a6bb","user":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"repo":{"id":21397341,"name":"kubernetes","full_name":"derekwaynecarr/kubernetes","owner":{"login":"derekwaynecarr","id":6233452,"avatar_url":"https://avatars.githubusercontent.com/u/6233452?v=3","gravatar_id":"","url":"https://api.github.com/users/derekwaynecarr","html_url":"https://github.com/derekwaynecarr","followers_url":"https://api.github.com/users/derekwaynecarr/followers","following_url":"https://api.github.com/users/derekwaynecarr/following{/other_user}","gists_url":"https://api.github.com/users/derekwaynecarr/gists{/gist_id}","starred_url":"https://api.github.com/users/derekwaynecarr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekwaynecarr/subscriptions","organizations_url":"https://api.github.com/users/derekwaynecarr/orgs","repos_url":"https://api.github.com/users/derekwaynecarr/repos","events_url":"https://api.github.com/users/derekwaynecarr/events{/privacy}","received_events_url":"https://api.github.com/users/derekwaynecarr/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/derekwaynecarr/kubernetes","description":"Container Cluster Manager","fork":true,"url":"https://api.github.com/repos/derekwaynecarr/kubernetes","forks_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/forks","keys_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/teams","hooks_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/events","assignees_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/tags","blobs_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/languages","stargazers_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/subscription","commits_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/merges","archive_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/downloads","issues_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/derekwaynecarr/kubernetes/releases{/id}","created_at":"2014-07-01T17:56:58Z","updated_at":"2015-06-02T02:59:28Z","pushed_at":"2015-12-11T18:40:49Z","git_url":"git://github.com/derekwaynecarr/kubernetes.git","ssh_url":"git@github.com:derekwaynecarr/kubernetes.git","clone_url":"https://github.com/derekwaynecarr/kubernetes.git","svn_url":"https://github.com/derekwaynecarr/kubernetes","homepage":null,"size":135689,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"b6f68df3c80585313d1f8be6450e9c99139e9ed0","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","created_at":"2014-06-06T22:56:04Z","updated_at":"2015-12-11T23:56:57Z","pushed_at":"2015-12-12T01:51:52Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":291652,"stargazers_count":11727,"watchers_count":11727,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3193,"mirror_url":null,"open_issues_count":2580,"forks":3193,"open_issues":2580,"watchers":11727,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/17688"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17688"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/17688/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/17688/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/47c56b57bb92e16a0b9e17ed5a76d7bfa523a6bb"}}}}