{"action":"created","issue":{"url":"https://api.github.com/repos/opencontainers/runc/issues/433","labels_url":"https://api.github.com/repos/opencontainers/runc/issues/433/labels{/name}","comments_url":"https://api.github.com/repos/opencontainers/runc/issues/433/comments","events_url":"https://api.github.com/repos/opencontainers/runc/issues/433/events","html_url":"https://github.com/opencontainers/runc/issues/433","id":121778906,"number":433,"title":"Proposal: Intel RDT/CAT cgroup support in runc/libcontainer","user":{"login":"xiaochenshen","id":16153452,"avatar_url":"https://avatars.githubusercontent.com/u/16153452?v=3","gravatar_id":"","url":"https://api.github.com/users/xiaochenshen","html_url":"https://github.com/xiaochenshen","followers_url":"https://api.github.com/users/xiaochenshen/followers","following_url":"https://api.github.com/users/xiaochenshen/following{/other_user}","gists_url":"https://api.github.com/users/xiaochenshen/gists{/gist_id}","starred_url":"https://api.github.com/users/xiaochenshen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiaochenshen/subscriptions","organizations_url":"https://api.github.com/users/xiaochenshen/orgs","repos_url":"https://api.github.com/users/xiaochenshen/repos","events_url":"https://api.github.com/users/xiaochenshen/events{/privacy}","received_events_url":"https://api.github.com/users/xiaochenshen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-11T19:43:14Z","updated_at":"2015-12-22T17:42:25Z","closed_at":null,"body":"```\r\nThe descriptions of Intel RDT/CAT features, user cases and Linux kernel cgroup interface are heavily based on the Intel RDT documentation of the Linux kernel patches:\r\nhttps://lkml.org/lkml/2015/10/2/72\r\n\r\nThanks to the authors of the kernel patches:\r\n* Vikas Shivappa <vikas.shivappa@linux.intel.com>\r\n* Fenghua Yu <fenghua.yu@intel.com>\r\n```\r\n\r\n## What is Intel RDT and CAT:\r\n\r\nIntel Cache Allocation Technology (CAT) is a sub-feature of Resource Director Technology (RDT). Currently L3 Cache is the only resource that is supported in RDT.\r\n\r\nCache Allocation Technology offers the capability of L3 cache Quality of Service (QoS). It provides a way for the Software (OS/VMM/Container) to restrict cache allocation to a defined 'subset' of cache which may be overlapping with other 'subsets'. This feature is used when allocating a line in cache i.e. when pulling new data into the cache. The programming of the h/w is done via PQR MSRs.\r\n\r\nThe different cache subsets are identified by CLOS (class of service) identifier and each CLOS has a CBM (cache bit mask). The CBM is a contiguous set of bits which defines the amount of cache resource that is available for each 'subset'.\r\n\r\nMore information can be found in the section 17.16 of [Intel Software Developer Manual] [1].\r\n[1]: http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf\r\n\r\n\r\n## Supported Intel Xeon CPU SKUs:\r\n\r\n* Intel(R) Xeon(R) processor E5 v4 and newer generations\r\n* Intel(R) Xeon(R) processor D\r\n* Intel(R) Xeon(R) processor E5 v3 (limited support)\r\n\r\nTo check if cache allocation was enabled:\r\n\r\n$ cat /proc/cpuinfo\r\n\r\nCheck if output have 'rdt' and 'cat_l3' flags.\r\n\r\n\r\n## Why is Cache Allocation needed:\r\n\r\nCache Allocation Technology is useful in managing large computer server systems with large size L3 cache, in the cloud and container context. Examples may be large servers running instances of webservers or database servers. In such complex systems, these subsets can be used for more careful placing of the available cache resources by a centralized root accessible interface.\r\n\r\nThe architecture also allows dynamically changing these subsets during runtime to further optimize the performance of the higher priority application with minimal degradation to the low priority app. Additionally, resources can be rebalanced for system throughput benefit.\r\n\r\n\r\n## User cases for container:\r\n\r\nFigure 1:\r\n![cat_case](https://cloud.githubusercontent.com/assets/16153452/11753748/fe4c73ce-a081-11e5-90e4-f6070e0d3142.png)\r\n\r\nNote: Figure 1 is fetched from section 17.16 of [Intel Software Developer Manual] [1].\r\n[1]: http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf\r\nCurrently the Last Level Cache (LLC) in Intel Xeon platforms is L3 cache. So LLC == L3 cache here.\r\n\r\n\r\n### Noisy neighbor issue:\r\nA typical use case is to solve the noisy neighbor issue in container environment. For example, when a streaming application which running in a container is constantly copying data and accessing linear space larger than L3 cache, and hence evicting a large amount of cache which could have otherwise been used by a higher priority computing application which running in another container.\r\n\r\nUsing the cache allocation feature, the 'noisy neighbors' container which running the streaming application can be confined to use a smaller cache, and the higher priority application be awarded a larger amount of L3 cache space.\r\n\r\n### L3 cache QoS through cgroup:\r\nAnother key user scenario is in large-scale container clusters context. A central scheduler or orchestrator would control resource allocations to a set of containers. In today's resource management, cgroups are widely used and a significant amount of plumbing in user space is already done to perform tasks like allocating and configuring resources dynamically and statically.\r\n\r\nDocker and runc are using cgroups interface via libcontainer to manage resources. They could benefit from cache allocation feature for cgroup interface is an easily adaptable interface for L3 cache allocation. We could define different cache subsets strategies through setting different CLOS/CBM in containers' runtime configuration. As a result, we could achieve fine-grained L3 cache QoS (quality of service) among containers.\r\n\r\n\r\n## Linux Kernel intel_rdt cgroup interface:\r\n\r\nIn Linux kernel 4.6 (or later), new cgroup subsystem 'intel_rdt' *will be* added soon with kernel config CONFIG_INTEL_RDT.\r\nThe latest [Intel Cache Allocation Technology (CAT) kernel patch] [2]:\r\n[2]: https://lkml.org/lkml/2015/10/2/72\r\nhttps://lkml.org/lkml/2015/10/2/72\r\n\r\nThe different L3 cache subsets are identified by CLOS identifier (class of service) and each CLOS has a CBM (cache bit mask). The CBM is a contiguous set of bits which defines the amount of cache resource that is available for each 'subset'.\r\n\r\nThe max CBM, which mapping to entire L3 cache, is indicated by *intel_rdt.l3cbm* in the root node. The value is varied among different supported Intel platforms (for example, intel_rdt.l3_cbm == 0xfffff means the max CBM is 20 bits). The *intel_rdt.l3_cmb* in any child cgroup is inherited from parent cgroup by default, and it can be changed by user later.\r\n\r\nSay if the L3 cache is 55 MB and max CBM is 20 bits. This assigns 11 MB (1/5) of L3 cache to group1 and group2 which is exclusive between them.\r\n\r\n```\r\n$ mount -t cgroup -ointel_rdt intel_rdt /sys/fs/cgroup/intel_rdt  \r\n$ cd /sys/fs/cgroup/intel_rdt  \r\n$ mkdir group1  \r\n$ mkdir group2  \r\n$ cd group1  \r\n$ /bin/echo 0xf > intel_rdt.l3_cbm  \r\n$ cd group2  \r\n$ /bin/echo 0xf0 > intel_rdt.l3_cbm\r\n```\r\n\r\nAssign tasks to the group1 and group2:\r\n```\r\n$ /bin/echo PID1 > tasks  \r\n$ /bin/echo PID2 > tasks  \r\n```\r\n\r\nLinux kernel cgroup infrastructure also supports mounting cpuset and intel_rdt cgroup subsystems together. We can configure L3 cache allocation to align CPU affinity per-cores or per-socket.\r\n\r\nIntel_rdt cgroup has zero or minimal overhead in hot path on following unsupported cases:\r\n* Linux kernel patch doesn't exist on any non-intel platforms.\r\n* On Intel platforms, this could not exist by default unless INTEL_RDT is enabled.\r\n* Remains a no-op when INTEL_RDT is enabled but Intel hardware does not support the feature.\r\n\r\n\r\n## intel_rdt cgroup support in github.com/opencontainers/specs\r\n\r\nThis is the prerequisite of this proposal. I will open a new issue or pull request in *github.com/opencontainers/specs* soon.\r\n* Add intel_rdt cgroup resources in Linux-specific runtime configuration to support ```runtime.json```.\r\n\r\n\r\n## intel_rdt cgroup support in runc/libcontainer\r\n\r\nThis proposal is mainly focused on intel_rdt cgroup infrastructure in libcontainer:\r\n\r\n* Add *package IntelRdtGroup* to implement new subsystem interface in libcontainer/cgroups/fs:\r\n  * *Name()*\r\n  * *GetStats()*\r\n  * *Remove()*\r\n  * *Apply()*\r\n  * *Set()*\r\n* Add *IntelRdtGroup* in cgroup subsystemSet in libcontainer/cgroups/fs.\r\n* Add intel_rdt cgroup unit tests in libcontainer/cgroups/fs.\r\n* Add intel_rdt cgroup stats metrics in libcontainer/cgroup.\r\n* Add systemd cgroup specific functions in libcontainer/cgroups/systemd.\r\n  * Add *IntelRdtGroup* in cgroup subsystemSet in libcontainer/cgroups/systemd.\r\n  * Add *joinIntelRdt()* function in *Apply()*\r\n* Add intel_rdt cgroup entries in libcontainer/configs.\r\n* Add intel_rdt libcontainer integration tests in libcontainer/integration.\r\n* Add runc documentations for intel_rdt cgroup. \r\n\r\n\r\n## intel_rdt cgroup support in Docker (TODO)\r\nWhen intel_rdt cgroup is ready in libcontainer, naturally, Docker could make use of libcontainer as native execution driver to support L3 cache allocation for container resource management.\r\n\r\nSome potential work to do in Docker in future:\r\n* Add *docker run* options to support intel_rdt.\r\n* Add *docker client/daemon* APIs to support intel_rdt.\r\n* Add intel_rdt functions in *docker execution driver*.\r\n* Add intel_rdt cgroup metrics in *docker stats*.\r\n* Add Docker documentations for intel_rdt cgroup."},"comment":{"url":"https://api.github.com/repos/opencontainers/runc/issues/comments/166685663","html_url":"https://github.com/opencontainers/runc/issues/433#issuecomment-166685663","issue_url":"https://api.github.com/repos/opencontainers/runc/issues/433","id":166685663,"user":{"login":"vishh","id":7063592,"avatar_url":"https://avatars.githubusercontent.com/u/7063592?v=3","gravatar_id":"","url":"https://api.github.com/users/vishh","html_url":"https://github.com/vishh","followers_url":"https://api.github.com/users/vishh/followers","following_url":"https://api.github.com/users/vishh/following{/other_user}","gists_url":"https://api.github.com/users/vishh/gists{/gist_id}","starred_url":"https://api.github.com/users/vishh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishh/subscriptions","organizations_url":"https://api.github.com/users/vishh/orgs","repos_url":"https://api.github.com/users/vishh/repos","events_url":"https://api.github.com/users/vishh/events{/privacy}","received_events_url":"https://api.github.com/users/vishh/received_events","type":"User","site_admin":false},"created_at":"2015-12-22T17:42:25Z","updated_at":"2015-12-22T17:42:25Z","body":"+1"}}