Hello, everybody, and welcome to my talk. My name is Peter Hunt. I'm a software engineer for Red Hat. I primarily work on Cryo for OpenShift, but I also work on Konmon and Podman and other container-related technologies. So today we're going to talk about Cryo, specifically the journey in Cryo in dropping the POS container. Before we get into the nitty gritty about what a POS container is, we gotta first talk about what, a little bit about Kubernetes background. So to get on the same page, let's talk about what a pod is. You can think about a pod like a group of, it's like a logical host. It's a group of containers, pretty much. They share some namespaces and some storage and they're kind of like servers. So you may have a web server container inside of it and maybe a logging sidecar and maybe some other containers that warm up the cache maybe. Now that we have understanding what a pod is, let's talk a little bit about the POS or the Infra container. So what is the POS?  In the bottom of this picture, we have POS 3.2. And the POS container sits and pauses. It really doesn't do very much more than that except for at the end of its life. And we'll talk more about what the POS does. The original, the idea of the POS is to hold the pod level namespaces and share them with other containers inside of the pod. And it's contained within the pod C group. It's often referred to as the pod Infra container. So in this talk, we're mostly gonna call it POS but you may see some instances of it being called an Infra container. For some history of Kubernetes pods, in the beginning there was just Docker and this was well before CRI or Rocket or other runtimes were able to go under Kubernetes. Kubernetes needed a way to give each pod an IP address. How would one do that in Docker? The way that they figured it out was they started a container and then they jumped into the network namespace of that container and assign it an IP. Then they started other containers joining that network namespace of the first container and then the containers had the assigned IP and that's how the POS container came to be. As such, it was originally called the net container because it only held the network namespace but eventually other namespaces were added to it, the first one being IPC, at which point it was named the Infra container in the code base. Eventually the PID namespace sharing was added and in the pod's job came a little bit more involved. In addition to just holding the namespaces, it is also by being PID one in the pod level PID namespace, it's responsible for reaping processes in the pod and we're gonna talk about that a little bit more later. So now let me know what the POS is, why would we drop it? We've been using this forever. The first reason is it uses up space for the binary, the container and the image. So per node, the image is, it's a pretty small 600 kilobytes, not bad, not really anything anyone would sweat. However, the amount of memory that it has per pod is not great. I mean, one meg per pod is really not bad. Kubernetes nodes typically cap at around 250 pods, which means that's 250 megs, that's barely anything in the grand scheme of things, but that is a little bit of space for your workloads. Another reason is it does take time to create, mount and start the container. Below we have an example of running it, creating a pod with a POS container and without. And these times don't look all that different and I'm gonna go into a little bit more of a performance comparison. I ultimately know that there are really two things that pod creation requests needs to do and needs to somehow share the namespaces, create the namespaces for sharing within the pod and set up the networking stack for the pod. So this is, by dropping the POS container, we're doing about half of those things. Finally, there's some process management overhead. There's much more code and setup and cleanup and we need to keep track of the POS container. Notice here, we have a devoted conmon, which is a container monitor for our infr container to make sure that it's still alive or hasn't been killed. And ultimately, no process means no process management. We don't have to worry about any of that and the saved code is saved bugs as well. So how would we go about dropping the POS container if we needed to? What does the POS container really do? The first thing that you need to do is find a way to keep namespaces without an associated process. And luckily Linux just supports us out of the box. We can bind mount namespaces. So in this first line here, we have unshare dash dash net and unshare means create or point to another namespace and jump into it. So unshare minus minus net means make me a new namespace and then put me right inside of it. And then that following command mount dash dash bind proc self ns net devar run net one, that's saying mount my personal namespace, which is the namespace we just created in the unshare command mount proc self ns net to the specified location. And then in this terminal below, we can unshare inside of to that bar run net one and then be inside of the namespace of the pod or created by that unshare command above. The second step, which is actually something that took me by surprise in implementing this feature is you need to apply the syscuddles in the pod. So syscuddles are namespace and we need to apply them at some point.  Some examples are like ip net ip forward and message mac they're just ways to configure some things inside of the namespaces. Finally, you need to take the namespaces that you've just created and pass it to the OCI runtime when, so it can use them in creating the container. So on the left here, we have the old way of doing it where we used proc, the proc entry for the pause container proc 864303 and took that, that namespace and then pass it to the runtime, which then unshared the container inside of it and then had that container inside of the namespace. On the other side, we have a created path that we passed down instead. So we have our unshare, our mounted namespace path and we just pass it right to the container and the runtime takes care of actually configuring it for the new container inside of the pod. Now that we've, so now that we know how, let's get to introducing how we actually did it and that is with a binary called pin and s pronounced pin and s not pins. And it currently lives inside of the cryo tree. It was inspired by the ip net and s commands as well as a container networking package that is used heavily in CNI today. It's a C binary that's exact from cryo. And the reason we went with this rather than doing it in cryo natively is because go is kind of funky with how it configures namespaces, well, how like the runtime interacts with namespaces, the runtime has the ability to jump through different namespaces at any time, you know. So you need or an interrupt, you know, your process at any time. So you need to lock the OS thread, do all of your namespace related stuff and then unlock it to safely do that. And instead of doing that, we just decided to do it in C which there's not a runtime mucking around with things and there's more native support for unsharing and mounting. Cryo then takes the mounted namespace from pin and s and hands it to a container and then boom, we did it. Pause process draw for the most part. The one case that we have to worry about still is the dreaded PID namespace. So remember the special responsibility that PID-1 has inside of a private PID namespace. It's responsible for reaping the children from the kernel's process table, basically calling weight PID so that those PIDs can then be reused. If you don't do this, then there's an entry in the process table but there's no associated process and this is called a zombified process that no zombie antidote will fix. For this case, a pod level PID namespace, we actually keep the POD container. Now there's a couple of hacky workarounds we could do instead like what if we had the first process inside of the first container inside of the POD B-PID-1 and made sure that that PID-1 always had the capability to reap the children of the PID namespace. But unfortunately, we would still need to ensure ordering. So if there was another container that jumped in beforehand, then that would be PID-1 and then there would be zombies. And ultimately this is what our pause container already does. We already reap the children inside of the POD level PID namespace and we've ensured ordering with the pause container. Luckily, this is not that much of a loss because most pods have a container level PID namespace instead of POD level. So we can still drop the input container and not worry about sharing a POD level PID namespace and only use a POD container for cases where we have a POD level PID namespace. So given that, let's talk a little bit about Kraus journey through different options and configuring sandboxes, which is an analogous for pods and having getting to the drop-in for us. So we started off configuring pods with namespaces and proc. We saw the, we have straw the string before, proc, PID, NS, net or whatever. And that's how we originally did it with a pause container. Then a little bit down the line, Cota containers needed a way to pre-configure the network namespace before starting the pause container so that the VM had a running network to start up with. And so an option was introduced manage network NS lifecycle, which used the container networking package and natively did it all on go to pin the namespace. And this made it for cryo 1.0. For security, we're moving towards managing all of the NS lifecycle. So about a year ago, we had a CVE where an interaction between the kernel OomKiller and Kanman, our container monitor, and PIDwrap caused an unprivileged container to be able to jump onto the network namespace of a host of the host, which could allow for the host to have the host name change or something like that. Which is very bad. That has been patched in a couple of different ways and we've nearly fully mitigated that. And the final nail on the coffin for that vulnerability is managing NS lifecycle, which is now the default option. Managing NS lifecycle is what introduced the pin NS binary and allowed us to pin all of the pod level namespaces, but without having yet dropped the pause container. And then finally for performance, we moved on to drop the pause container and that's corresponds to the drop in for option, which is slated to be introduced in cryo 1.0. So given that, let's talk a little bit about the future. So yeah, in experimental support for the pause container, dropping the pause container is targeted for cryo 1.19. Next, then after that, we plan on having podman pods also use pin NS to configure the namespaces, as it currently also uses a input container. And then eventually we want pin namespaces and drop pause to be the default so that we get rid of the pause container entirely unless we absolutely need it in the pin namespace case. So let's do a quick demo. So here we have our local cluster running and this is just has cryo and we're gonna start off with not managing the namespace lifecycle and not dropping the pause container. So here we have our two containers, hello one and hello two, our two pods and they're pretty much the same except for their name. And basically what they're gonna be doing is an alpine container that runs top, very simple. So we're gonna start off by running cube cuddle by our hello pod. Oh, now we have our hello pod. So let's look at it in cry cuddle, cuddle pods. So here we have the container ID for this hello pod. Let's look at the pause process itself. So we can do a run C list, yes, A. And notice here we have F3C, run C list spaces. So here we have the namespaces that we asked run C to use for this pod. So notice it has a private PID and mount namespace but we passed down the path for the net IPC and UTS namespace of the infra container. So look at this PID, it matches the 285264 of our pause process because we're taking the entry in the proc table for all the namespaces of the pause container and passing it down to run C. Now let us try it with dropping infra equals true. So now we're gonna create, we have hello to created. All right, cuddle pods, yes, wanna say. And look at this, this is our run C list. And notice here we did the same thing as before except instead of passing the path that had the proc entry we're passing the namespaces that pin in S unshared, configure the syscuddles and then bind mounted to the specified location. Crowd took that location and then pass it down to run C and now we have a pod running without a PID namespace without a pause container. So that is it for the demo. Let's talk a little bit about performance comparison, a moment of truth. So in this, here's a little script that we used to test how well the performance increase, how much better the performance is when dropping the pause container. All we're doing is in parallel we're running a hundred unique pods waiting until that's done and then removing all of those pods and taking the time for that operation. And we did it 10 times each for dropped and not dropped pause container and use a tool called multi-time to aggregate the data and make the comparison. So, and here's the data that we got. In the drop cause pause case we have about a little bit less or a little bit more than half of the real time as when we keep the pause. Now there's a couple of caveats here. Number one, notice how the user insists time is about the same and usually those are actually the meaningful metrics for the amount of time a process takes because they don't, because the real time has the noise of kernel interrupts and other things that are happening on the system. But unfortunately the way that crycuddle works it makes a request off to cryo, cryo does all the work and then in cryo returns a response. So none of the actual work is being of the pod creation and removal is being attributed to the real time of the cryo-cryo process. Okay. Well, so that means the, there's a little bit of noise and these numbers aren't precise but what we can take away is that we're doing many fewer things when we're not actually creating we're not creating the root of FES and then running the container and then having Kahneman listening to the container and then waiting for Kahneman to have you listening to the container and then we're doing none of that. All we're doing is bind mounting a couple of unsharing bind mounting and then going on with our network setup. So this is much better and we like that and this is not even accounting for the amount of time that we took the amount of code savings that we have with all the complexity of doing all of the pause container related things and it's also not accounting for the small but notable amount of memory that we're saving by not having a pause container for every pod. So that is the presentation. If you want to find out more here's a couple of different links you can also contact me and I'm at hair commander on most things. If thank you very much for coming. Do you have any questions and are watching live and feel free to ask them. If you're not live then you can ping me on any of these things and I'd be happy to answer. I really appreciate you coming and are there any questions? Oh, so yes, please questions in the chat. There was four, go ahead. I think we had one question that you said you were going to ask. We read it for you. Are you just going to find it? I can find it. So the question was couldn't we just have cryo create a pid name space and put a process into that that it would later kill. And so I hadn't thought about that prior to this. There's a couple of things that may actually make it so that that doesn't save us very much. So we still use the runtime spec generation for keeping this state of the pod so that we're basically reusing all of the work we did before making a config JSON blob and for the pause container even if we drop the pause container and that's so that we can restore the pod. I think we'd eventually like to move away from that but that's what we're doing currently. So it wouldn't save the spec generation step. We would still need to have some conmon like process because we need to be able to handle if somehow this pause process got killed somehow and if the pause, so we'd probably still want someone listening to it or be able to catch a sick child and Go can't do that very well because of how Go handles signals which is the runtime gobbles them up. And finally, in the Cota container case we still need a pause container because and I didn't mention this in the talk and I should have but for Cota containers the another VM based runtimes the pause container actually there's an annotation that is given to the pause container and then as it's passed down to the runtime the runtime actually creates a new VM for the pause container and then later containers are injected into that VM. So we couldn't have, we couldn't make, we couldn't be totally pauseless because we still need to handle that case. So yeah, but yeah, I think the, oh yeah, and then we need the, the we would need that process to be able to do PID reaping which is not too hard to do but yeah, it would basically be the equivalent amount of work with a little less of the overhead of OCI containers in general. So we're gonna add a whole lot. So it wouldn't save us a ton yet. And there's a follow up question and does pause container handled shared key groups or is it done via cryo? So the pause container used to hold the namespace for the, basically the namespaces for the pod was based on the pause containers PID proc entry in regards to this PID. But now we don't use that anymore. We use pin in S which just mounts the, oh shared C, okay. Sorry, shared C group namespace. The question was, does pause container handled shared C group namespace? And yeah, that's not shared in Kubernetes yet, though we're working on getting that working for the C groups V2. And pin in S to support it. Yes. Are there any other questions? If not, then. If we don't have any other questions, I guess folks can head over to the breakup, to the expo hall. If we click on the link to the left of your screen down there, you can go and meet other attendees of the conference or if you just want to take a break. The next talk is gonna start at 330. So we'll see you over there. Thank you so much. And great presentation, Peter. Thank you. Thanks Beverly. Bye bye everyone. Thanks for joining.


Next, we'll hear from Natalia Kodiakova from Raysono AG. She'll share how they're using CI templates to, as she says, automatically boost setup speed and consistency. Over to Natalia. Hello, everyone, and welcome to GitLab Commit 2021. My name is Natalia Kodiakova, and today I want to present you a talk using CI templates to automatically boost setup speed and consistency. A couple of words about myself. I'm a senior software engineer, back-end and DevOps at Raysono, based in Munich, Germany. I come, so to say, from the engineering trenches. My first language is Java. In my company, I adopted GitLab, and I'm a coach in CI tools and best practices when it comes to DevOps. I organize hackathons on a regular basis, and I'm a fan of algorithms and clean code. Let's look at our agenda today. The ever-increasing complexity of project setup was quite menacing, until we got a new hope, CI templates. The force awakened, and the CI template library arose company-wide. I'll show you how we created the library of CI templates and fine-tuned it, and what our automation Jedi achieved in the end. So let's jump right in. It always takes more or less same steps to set up a project. Create a repository, set up a CI pipeline with common jobs like build, test, code quality scan. Then set up a deployment, which is usually one of the standard alternatives. Copy files over SSH, push a Docker image to registry, or do some help deployment. On top, add config files for third-party tools, like Renovate. Renovate, by the way, is a tool for automated dependency update. We use it a lot, and always get a merge request when some library we use becomes outdated. We must go through all these setup steps for every project, and every time there are some tiny differences. Some one-line changes in CI jobs that are copy-pasted from project to project. Then there is a separate tool for quality scan, SonaCube, and it also needs integration in the pipeline. Patients you must have to finish the project setup. As we got a requirement to add security scans to all our projects, we came to a conclusion. As security grows exponentially, we need to change something. As we looked at CI templates, a new hope arose. We thought, what if we don't adapt our existing flow, but create something from scratch? What would our dream solution be like? It would be easy to use. Transparent and clear from the first sight. Flexible and extendable. Consistence throughout the company. Improving security of our products. Allowing every developer to contribute. Gathering all scanning tools in one place. And speeding up our setup process. Our Jedi masters used the force, evaluated the options, and decided to go with GitLab. First question, as always, is, where do we begin? Create a repository for the templates. No need to reinvent the wheel. Check out the templates from GitLab. How everything is organized and structured. We went in the same direction. We have individual folders for separate drops and for product-specific drops. For example, if we are working with some specific CMS, all jobs related to it will be separated. And we have complete pipelines that can be taken into project as is. The project visibility is internal. Every logged in user must be able to include the pipelines. And this is exactly what internal visibility does. Our repository has two branches, latest and stable. Stable being the default. The development occurs in latest branch, so the jobs there might be buggy or not yet tested. However, if someone wants to take a risk, there's always a possibility to include the job from a specific branch. When everything is tested and ready to go live, we merge the latest branch into stable and the job becomes generally available. You see two examples how we can include a job from a default stable branch or its version from latest. If we were about to test the latest changes in our build job, we would use the second one. Let's dive into some examples how to write job templates. The job you see does a build over Node.js project. It is executed in build stage, uses Docker image of Node as a base image, defines some variables and cache. The build itself is performed with yarn, which is a part of Node base image. There are some points I want to pay attention to. The customizable parts are taken out into variables. If someone includes this job into their pipeline, they will be able to set another Node version, just with a variable without the need to redefine the job. For the same reason of customization, don't use before script and after script. We leave our developers the possibility to add some behavior to this job in their projects. Use default stages where possible, build, test or deploy or document which stages need to be defined for this job to work. Here is another job that calls Semantic Release to release a new version of our application. Release means changing the version in the package JSON, committing it to the branch and creating a Git tag. So this tool needs right access to our repository and the access token is configured with a variable GitLab token. Now, we cannot provide any default value for it. So this variable must be defined by the user in their project. Our jobs check, first of all, if the user defined it. The error message should tell exactly what the job expects. Consider it a node to your future self when you're using your own template and it kindly guides you through the setup. See here in the script, if the variable is missing, the error message says, create a project access token with scope write repository and add it to a CI variable GitLab token. How can it get more specific? Now that we agreed on how to write individual jobs, let's think about what jobs to define. The easiest approach is to take a project that has a huge pipeline with a lot of jobs. Identify which jobs can be used in other projects, what is the common denominator of the projects of this kind, and then extract the jobs to the templates one by one. If we were writing the template library from scratch, we could have just imagined which jobs might be useful. But since we already have some pipelines, we take them as a guideline. What do we usually need in our projects? Another important point to agree on is a naming convention. When your library grows, you should be able to just look at the name and have the idea of what the job does. You see our job names as example. So it's pretty obvious that GitLab release creates a GitLab release and that MavenBuild does the build for Maven project. I'm not insisting this convention is a silver bullet. If you find a better approach, use it and let us know in the chat what are your thoughts on the naming. When you have your building blocks, it's time to assemble a complete pipeline. This one is a babes pipeline for our Maven projects. Include your jobs, include security scans and define the stages that your jobs need. Remember we made the jobs configurable with variables? Security scans are configurable in the same way. Add the necessary variables into your CI file and you're ready to go. The base pipeline for a different technology will look almost the same. There was a Maven pipeline on the previous slide and this one is for Node.js. The only difference are the stages. So the jobs are different and they require different stages. Cache definition and two repository versus Node modules and security scans configuration. These pipelines will be used as base for all the project. So we give them as generic as possible. They have the minimum necessary jobs to build and scan the project for the selected technology. What if I told you that a CI pipeline can have an Easter egg? That's right, Pikachu is excited that the tests passed. We output this ASCII art after the test job was successful. The picture is an instance-wide file variable and we output it with a utility that adds some colors to it. In the after-script section, a variable CI job status is available. So we check if the tests really passed and let the Pikachu out only if there were no errors. I'll show you the code quickly. We won't go through every line of it. You can check out the slides after my talk. The only thing I want to mention, we use reference tags that were introduced in GitLab 13.9. The actual code to output the picture is stored in a separate file and there are two reasons for that. First, we did that to hide the Easter egg, to make it not that obvious for those who include the test job into their pipeline. A big command in the after-script would catch the eye and a reference to some mysterious test helper would not. Second, there are different test jobs for Node, Maven, Gradle. Reference tags exclude duplication and made the code look much cleaner. The template library has already become big and powerful. Can we improve it even more? Code quality. Code quality scan in GitLab uses code climate under the hood. If you check out GitLab's template, you'll get the idea how it is working. And you will notice that GitLab defines default config for code quality scan. We wanted to build our own theme park, so we created a project with these defaults. We zip the files and upload the archive to a package registry. In our CI library, we define the template that adds some behavior to the standard code quality. The before-script section of this code quality drop is described in this template. We download the zip with defaults from the package registry and put the files where GitLab expects them to be. So this is how we override the default code quality rules. Again, some code for reference so that you could check it out later. Look at the before-script. We download the artifacts with widget and to access package registry of the project with defaults. We defined a project access token and stored it as an instance variable. The idea of this project is an instance variable as well. So the users do not even need to know we've configured something custom to get in the defaults. Another code quality perk is the visual HTML report. To get it, you need to extend the code quality drop, which in our case already uses the customized defaults. Set the report format to HTML with the environment variable and deploy it to GitLab pages. So GitLab pages deployment is another drop that has to be coupled with this visual code climate report generation. And we execute these drops only in default branch. No code snippets here because these are exactly the steps described in GitLab documentation. We just wrap these drops into our own template. And last but not least, linting. Again, we saw how GitLab are doing it. And found it an amazing idea. We just put the lint config file into the root of the repository and add the linting job to the pipeline. To the pipeline of the CI template library. Linting job fails if the formatting is not consistent throughout the template library. Everything looks clean and organized. So what have we achieved? We created project templates. And the CI file in a template contains only three lines of code. It just includes the base pipeline from the template library.  Apart from that, we added code owners template, the configs for code climate and renovate and other project specific files like POM, XML, Gradle build or package JSON. It's not a typo in the talk title and not a typo in the slide. The setup is really done automagically. The CI file is really small and references a well-tested pipeline from the library. All scans are already configured and ready to use. Configuration files are in place. The setup is configurable and extendable and easy to contribute. And the cherry on the cake, we renounced Sonacube and other third-party scanning tools. Everything is done now with GitLab. Security scans have become a part of any small change. We achieved consistency in the processes. Without consistency, we couldn't measure deviation. Now we can track the exact changes in security score in every commit and eliminate the vulnerabilities early. Security dashboard gives a great overview of the dynamics. We can see how we're progressing with security fixes throughout the time. Since we didn't have security scans earlier, we got a full report and had to clean up some of our dependencies and Docker images to remove critical vulnerabilities. Our overall confidence in the code has increased. GitLab shows all new vulnerabilities and code smells directly in the merge request. So the reviewers can instantly see if there's more to be done before the merge. We use almost all available scans, but in this example, I'll show you the default ones that we included in our Node.js-based projects. We used SEST, static application security testing, dependency scanning, and secret detection in all Node-based projects. Here is a screenshot from our recent merge request. You see developers dismissed some detected vulnerabilities for one or another reason. But the rest needs to be dealt with and secret scanning detected. No issues. Yay. But the majority of our merge requests looks now like this. All clean, green, and ready to be merged. And all this analysis comes now out of the box from our template library. It automatically appears in every merge request in every project and helps enormously during the code review. I also want to mention Docker container scanning, even though it is not a part of the base pipelines. We use it only in the projects that are shipped as Docker images and include this scan together with a job to build the image itself. When we first scanned our Docker-based projects, we were quite surprised about the amount of its output. After we carefully analyzed what base images we use, it turned out it's quite easy to fix the half of the detected vulnerabilities by just changing the base image. I must say it's a great feeling to confirm the vulnerabilities remediated and to just dismiss it. Not only we added so many security scans, but we also decreased time to set up a new project from days to minutes. The overall quality and the confidence in our setup improved dramatically. And developers are really enjoying working with CI template library. Please do not hesitate to contact me if you have any questions or if you just want to talk about GitLab and CI templates. I'll be happy to chat. So thank you and may the force be with you.