{"action":"opened","issue":{"url":"https://api.github.com/repos/coreos/rkt/issues/986","labels_url":"https://api.github.com/repos/coreos/rkt/issues/986/labels{/name}","comments_url":"https://api.github.com/repos/coreos/rkt/issues/986/comments","events_url":"https://api.github.com/repos/coreos/rkt/issues/986/events","html_url":"https://github.com/coreos/rkt/issues/986","id":84603519,"number":986,"title":"Investigating user namespaces","user":{"login":"tixxdz","id":1549291,"avatar_url":"https://avatars.githubusercontent.com/u/1549291?v=3","gravatar_id":"","url":"https://api.github.com/users/tixxdz","html_url":"https://github.com/tixxdz","followers_url":"https://api.github.com/users/tixxdz/followers","following_url":"https://api.github.com/users/tixxdz/following{/other_user}","gists_url":"https://api.github.com/users/tixxdz/gists{/gist_id}","starred_url":"https://api.github.com/users/tixxdz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tixxdz/subscriptions","organizations_url":"https://api.github.com/users/tixxdz/orgs","repos_url":"https://api.github.com/users/tixxdz/repos","events_url":"https://api.github.com/users/tixxdz/events{/privacy}","received_events_url":"https://api.github.com/users/tixxdz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-03T14:49:04Z","updated_at":"2015-06-03T14:49:04Z","closed_at":null,"body":"In this issue we try to investigate user namespaces support in rkt.\r\n\r\nFor the moment unprivileged user namespaces are out of scope.\r\n\r\n## Problems\r\n\r\n### 1) CAP_SYS_ADMIN and cgroup filesystems:\r\nrkt pods have CAP_SYS_ADMIN this may reduce rkt's security: containers could remount cgroups in read-write mode and since cgroups are not namespaced, containers can change cgroup settings for services on the host.\r\nGetting rid of CAP_SYS_ADMIN is difficult with the current architecture. Files in the cgroup filesystems are writable only by root (in the system slice) or by a specific user (in user slices). If the root user and other users from the host are not mapped in the user namespace of the container, it becomes a non-issue.\r\nplease see this issue \"stage1: rkt pods should not be given CAP_SYS_ADMIN\"  https://github.com/coreos/rkt/issues/576\r\n\r\nTODO: check that the cgroup filesystem access rights behave as described as above when running in a user namespace.\r\n\r\n### 2) Separation and isolation\r\n\r\n#### 2.1) Capabilities:\r\nuser namespace aware kernel interfaces are handled with the ns_capable() check against the current userns. We may take advantage of this isolation to give some capabilities to container X and give other capabilities to container Y, at the same these capabilities may not be effective on the host in other words the init_userns. This gives advantage to allow some file system capabilities on a specific container for its internal operations without affecting other containers or even the host.\r\n\r\nPlease note that some kernel interface still use the plain old capable() check, if the check succeed then the caps are effective globally.\r\n\r\n#### 2.2) Per-user limits:\r\nEach user on the system has its own “struct user_struct” to count resources (processes, signals, etc.) When several pods are used they share the same user, any operation on a pod may affect other pods. To improve separation and add an extra layer of resource isolation we may use user namespace and assign a range of global kuid_t to pod X and assign another range to pod Y, this may improve the situation and prevent some pods from DoS'ing each other.\r\n\r\nPlease note that kuid_t is not uid_t. kuid_t is the global kernel UID used to identify process's credentials.  \r\n\r\n![userns_heirarchy](https://cloud.githubusercontent.com/assets/1549291/7962574/fe4fb4e8-0a05-11e5-86b0-b906f1dcb79c.png)\r\n\r\n\r\n## What we are trying to do:\r\n\r\n### 1) User namespace mapping:\r\nrkt will use only 1 level of user namespace. The schema above is just to illustrate the general concept of user namespaces.\r\n\r\nEach running pod will have a range of uid assigned. For example, pod1 will have uids 200000 to 200999 (mapped to 0-999) and pod2 will have 201000 to 201999 (mapped to 0-999).\r\nrkt should not reuse uids assigned to other system users (e.g. don’t reuse www-data, geoclue or sshd users!). rkt will be assigned a big range of uids it is allowed to use in /etc/subuid (see manpage subuid(5) and useradd(8)). Example: rkt:200000:65536.\r\n\r\n### 2) User namespace locking:\r\nSince rkt does not have a central daemon to assign uids within the global range allowed by /etc/subuid, we will need some rkt-specific locking to avoid multiple running pods to reuse the same uids. Example: pod1 will lock on /run/rkt/uid-locks/uid-200000; pod2 will lock on /run/rkt/uid-locks/uid-201000. Or maybe something smarter to express the range.\r\n\r\n\r\n## Challenges / things to check:\r\n\r\n### 1) uid-shifts for rootfs:\r\n#### 1.1) At extract time:\r\nthe same ACI could be used in several running pods. The rootfs trees are cached in the CAS (Content Addressable Storage) in /var/lib/rkt/cas/ and have specific uid/gid owners and changing them all (recursive chown) is too costly. We could shift uids on the fly at extract time, then it would not be so costly.\r\n\r\n#### 1.2) At mount time:\r\nWhen a rootfs tree is used by overlayfs, we would need some vfs_uid= shifting option.\r\nThis was already mention in this lwn article \"UID/GID identity and filesystems\" http://lwn.net/Articles/637431/\r\nhttps://lists.linux-foundation.org/pipermail/containers/2014-June/034630.html\r\n\r\nvolumes bind-mounting: similar issue\r\n\r\n### 2) Network namespace:\r\nOn Linux, a network namespace belongs to the user namespace under which it was created, see field user_ns in struct net: http://lxr.free-electrons.com/source/include/net/net_namespace.h#L44. \r\n\r\nRkt currently creates the network namespace in network plugins before systemd-nspawn is launched. If the user namespace is to be created by systemd-nspawn, it means that the network namespace will belong to the host user namespace rather than the pod user namespace. AFAIU, it has unwanted consequences on the access to /sys/class/net/. See also https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=87a8ebd637dafc255070f503909a053cf0d98d3f\r\n\r\n### 3) Bind mounting:\r\nBind mounting a socket file from the host to the container? E.g. sdnotify socket (https://bugs.freedesktop.org/show_bug.cgi?id=89844).\r\n\r\n### 4) Passing file descriptors:\r\nPassing fds in the other direction from the container to the host? E.g. the journal dirfd that @krnowak  is working on https://github.com/coreos/rkt/issues/947.\r\n\r\n\r\n\r\n## Model\r\n\r\nOur model may touch several layers from rkt, nspawn to the kernel\r\n\r\n### 1) kernel: uidshift on the rootfs\r\nAs noted in the Challenges section, we may add a uidshift mount option for containers but also to implement dynamic uids for services. This may allow running unprivileged daemons with dynamically assigned uids without leaking into the persistent file system.\r\n    \r\nImplementation:\r\n#### 1.1) A generic vfs mount option ?\r\nmount(source, target, “bind”, MS_BIND|MS_REMOUNT,  vfs_uidshifts)\r\n\r\n#### 1.2) An overlayfs mount option or a completely new overlayfs-like fs\r\n\r\nSome file systems like NFS are already doing some mapping.\r\n\r\nTODO: continue the investigation.\r\n\r\nThanks to @alban for his help on the subject."}}