an engineer in maybe the worst way possible by trying to get K3S like four or five years ago before it was as stable as it is now to run on a Raspberry Pi for a miniature self-driving car I was building for work, which sounds ridiculous in retrospect, but I promise we had a reason for it. And the way I explain Kubernetes now is really heavily influenced by that experience, by my experiences teaching bootcamp grads, intro's to Kubernetes, and my experiences handling some comms-related kerfuffles for the Kubernetes project, wherein I learned just how little developers tend to know about Kubernetes even when their app is being deployed on it. So if you want to get a hold of me later, my Twitter handle is Dixie3 Flatline. You can also try emailing me at work. I would not recommend trying that. I'm not great at replying to emails. So let's go. First off, why should developers even care about Kubernetes? The answer is because you are probably going to have to touch it eventually. Even if you're not developing with Kubernetes locally now, it's increasingly likely that the application that you're building will be deployed with it. So it's similarly increasingly likely that you're gonna be expected to be able to test it locally so that there aren't any surprises. At the very least, you're gonna be expected to understand containers as the shift from monolithic architectures to more of a microservices situation moves some of the responsibility for standing up and making infrastructure decisions from exclusive the ops team to more of a shared responsibility, which is kind of the whole point of DevOps, right? We're trying to meet people in the middle there rather than just chucking things over the fence. But nobody is asking you to become an expert. There are entire people whose jobs it is to be an expert in Kubernetes. The average developer does not need to do that unless you wanna pivot towards specializing in Kubernetes, which a couple of people in the crowd here have done. You really just need the basics. We'll start with defining what Kubernetes actually is at a surface level. It is an open source container orchestration tool. And a container orchestrator is a tool that automates the management and deployment and scaling of containerized applications. While it is possible to do all of this by hand, it is slow, it is unreliable, and it is a massive pain. You don't wanna have to do it. We've all heard that Kubernetes is difficult to learn. This is like a longstanding thing that we're kind of trying to get away from. And part of that is talks like this and tracks like this to make it feel more approachable, but I promise you that if you're in a situation where you need Kubernetes, it is much harder to go without it than it is to just learn it. You can think of container orchestration in terms of a container ship. It's a super common way to explain the way this works. Kubernetes and other tools like it, there are other things that do this like Docker swarm and mesos. It's the thing that dictates which shipping containers go where on the barge in relation to other shipping containers and what goes in them. So it's the person at the dock with a clipboard, right? These collections of organized containers are called clusters. So container orchestration means that there are now a whole bunch of things that you don't have to worry about anymore. Scaling your application happens with much less effort. So say you get hit with a ton of traffic, your app goes viral somewhere and suddenly it's very, very popular and you need to handle it. In most cases, that is a one-line change in whatever tool you use to define your cluster and what your application looks like on it. You just bring up more replicas, right? Instead of going to your cloud host and provisioning more servers, deploying your application to those new servers, testing that application on those new servers and configuring all of your proxies again. That's a much bigger pain, at least for me. You don't have to worry about deploying those individual servers anymore or testing them, Kubernetes just spins up another replica and a lot of security and networking concerns do get abstracted away from you. That is not to say that Kubernetes is inherently secure. It definitely isn't. It solves some security concerns that you have with other ways of deploying and scaling applications but it introduces some new ones, so do not fall prey to the long-standing meme that Kubernetes is secure because it's not. Same goes for networking. In some ways, it's easier. In some ways, it's harder. You are making trade-offs here. So complex Kubernetes may be, but that's because of how much previously manual labor it is doing for you. The operational overhead of getting going can be a pain and it does require specialized knowledge to put it up there and to maintain it, but once you get to a point where you really, really, really need a tool that's powerful, like your team or your org should have the resources to handle that, I am of the opinion that not everything needs Kubernetes. In fact, a lot of things super don't need Kubernetes but it never hurts to be prepared with kind of knowing what it is. Besides all of that, the Kubernetes community is massive. It's really, really big. It's really, really friendly. It's made up of me and the track hosts and a bunch of other people in the crowd here and hopefully the rest of you who are new and just went to the 101 workshop yesterday. We all want nothing but to build this better and extend it and make it easier for others to come help build it better. So I hope you join us. If you have a question ever after this talk, 10 years from now in the process of using Kubernetes, we have a huge online community and we will happily answer your question. If we don't know the answer to it, we know who does and we'll point you at them, which is the real power of the Kubernetes community. So come hang. Before we go further and dive into Kubernetes architecture, I want to address the big blue whale in the room. I'm glad that I included this slide because somebody asked me a question about it earlier today. Docker and Kubernetes get conflated quite a lot and it can cause some problems. So what does Docker have to do with Kubernetes? If you were a developer, you might be used to using Docker locally for dev. This is a particular point of confusion for engineers touching Kubernetes for the first time and one that we have underestimated before. So this is worth mentioning for the folks in the room who are touching Kubernetes for the first time and have like literally no context. Colloquially, we like to say Docker image, right? But it is really more of like a Qtap Clarex situation. When we say Docker image, like the image Docker produces is actually what's called an OCI compliant container image. OCI stands for Open Container Initiative and other tools can produce OCI images too. Wildly popular and also being a really important part of the Kubernetes projects history. It was the original and only container runtime for Kubernetes for a while. So unfortunately, it's a little annoying, but at this point, you kind of just have to figure out from context clues, whether somebody literally means Docker desktop, the tool you were using for local dev, or if they are referring to any old container image. But functionally, there is no difference between a thing produced by Docker desktop and a thing produced by like Podman. Now you know, hopefully that solves some confusion for the person I talked to earlier. Now we've cleared that up though, let's talk about the anatomy of a cluster and see where those containers end up living. So Kubernetes is extremely complex. Wow, that's fun. Oh, I didn't even get to the good slide. There is a good slide too. What is happening? Is it my computer? I can't tell.  Actually, how about I plug in my power on the other side and see if this computer is doing that thing, that it's haunted. Is it haunted? It's fine on my display. The next slide. That's just, it's a 2015. 2015. Yeah, yeah. Switch to the second part right here. Okay. No, put the actual video out. Oh, okay, yeah. I think so.  I'm glad everybody actually thought it was funny too. Yeah, I just gotta, there we go. We're back. Not glitchless buddy. I can switch laptops. I just can't use my terminal on the other one cause it's an M1. You know when you try switching laptops? Always have a backup. Yes. This is an anxious habit I picked up from Jessica Dean. Because she always travels with a second laptop. Just in case. Okay, okay. It's my 2015 that's cursed. It can't run. I was concerned that was gonna be a problem cause this laptop is old. It can't run any Kubernetes cluster and video at the same time I guess. Anyway, sorry about that. So, Kubernetes is super complex, right? And this complexity is one of the things that can make it so difficult to teach and learn. But fortunately, you only need the broad strokes at first and we're getting better at explaining this complexity in a way that makes sense to humans. I promise you that almost nobody is a subject matter expert in the entirety of the whole situation. Most people specialize in only a couple aspects of Kubernetes like networking or storage or security or authorization. Absolutely nobody is an expert in the whole kit and caboodle. I doubt even the people who originally built the project are at this point because it's so different comparably. And this diagram or one like it is how people usually explain the pieces of Kubernetes. I've taken this one straight from the Kubernetes docs and it does a perfectly fine job. It's very well annotated, it's very clean. But I think it's boring and it's unmemorable and I had a really bad idea that I have to get out and y'all are trapped in a room with me. So, you have to learn from a different diagram. This is Kubernetes, okay? This is Kubernetes. If you use Kubernetes already, you will understand immediately both like architecturally what's going on here and from the vibes this dog is exuding. This dog has tired cluster admin energy, like for sure. And for everybody else, allow me to explain Kubernetes architecture for you with the help of this Chihuahua standing on eight cheeseburgers. This whole situation here is your Kubernetes cluster. Each cluster needs to contain at least two things, a control plane and a worker node. The worker node contains the pods that make up your containerized application workload and that is the bare minimum. That's what you need for a cluster. In a production environment, you would see one control plane maybe across multiple machines managing multiple nodes. Maybe each node is a dedicated machine to maintain high availability. So the control plane is the thing that manages what's going on with your nodes and your pods. It's responsible for managing everything to do with your application workload and with the cluster itself. The control plane itself includes several smaller components as well. It's got the CUBE API server, which is responsible for exposing the Kubernetes API for use at CD, which I am now pronouncing correctly in this talk, which is a key value store for all of your cluster data. The CUBE scheduler, which takes care of assigning newly created pods to nodes and to controller managers, which are responsible for managing a whole bunch of control logic and one of them is cloud specific. The exasperated Chihuahua is your control plane. So a node is a worker machine responsible for managing one or more pods. The node contains all of the things the pods need to operate, like your container runtime, as well as the pods themselves. Nodes are made up of three parts. The cubelet, which is responsible for babysitting your containers by making sure that they're healthy. CUBE proxy, which is for networking. And a container runtime on which your containers ultimately will run. There are different container runtimes with different advantages. Most cloud providers default to containerd, which is actually pronounced containerd. And each stack of burgers here is a node. A pod contains all of the things that make up your application workload in the form of containers. In the case of this image, our application workload is what I assume is a McDonald's dollar menu cheeseburger. So we have containers for a slice of cheese, ketchup, the burger patty, onions, pickles, the bun, those little dehydrated great onions are incredible, and I wish more places had them. Because we've got friends coming over. We've told this cluster that it needs to maintain four nodes, two pods each, for a total of eight replicas of the same cheeseburger. Thank you for letting me quell the screaming of that particular brain worm.  Deploy some stuff, scale it up and down, good news. There is no reason to go straight to cliff diving into the ocean here. There are tools that abstract away some of the more nitpicky parts of running Kubernetes, like dealing with some specific like networking and security concerns that just do not matter if you are doing it locally for fun testing playtime, but you do absolutely need to worry about in prod. These tools, however, do not generally prevent you from digging into the API if you want to, so you can still experiment. So, we are gonna stand up a little cluster and deploy something really common to it, Nginx. I will say that since I've had to switch laptops, the demo itself is running on this one. I have slides for it just in case of this, but at the end, if my laptop wants to give me a second chance, I will switch back over to it so we can actually poke at the running cluster. So, for my cluster, I use Minicube. Minicube is what I prefer to use when I'm just messing around on a local machine, but K0s and K3s also exist and are really cool. They are both meant for more like IoT embedded applications on like really small, low memory machines. You can, of course, also just use a full featured managed Kubernetes service on a cloud provider like GKE or one of the 85 million ways AWS now has to deploy containerized workloads. It's truly getting out of control and I honestly don't know what most of them do anymore at this point. Personally, I check everything on GKE because I know what it does. Anyway, for situations like this, I do prefer to stick with things that aren't gonna accidentally run up a huge cloud bill if I forget to tear down my infrastructure at the end, even if it's not exactly production ready. So you will see me have to like do some weird stuff and take some shortcuts because I'm using Minicube and not like GKE. So this is step one. I really wish I had my computer for this that would let me show you, but anyway, I would have had to do this slide anyway because installing Minicube takes like 10 minutes on that computer. It will not be that slow for you if your computer is newer than 2015, but anyway, this is Minicube installed and starting. You can see I've set some flags here. One is to specify a driver just to show you that you can. This used to be necessary in order to use some particular aspects of Kubernetes on certain operating systems. So if you go to try this yourself and you're running into a weird problem where for instance, an ingress controller isn't working, if you're on a Mac, it's because you need to be using the virtual box driver.  It will default to Docker usually, but there's also HyperKit and a couple of others. You can also see that I'm sending the memory that Minicube is allowed to use, which can be useful by default. It wants to use eight gigabytes of memory. The total available memory on this laptop is eight gigabytes. So obviously, I can't let it do that. Anyway, running Minicube start is literally all you need to get a small cluster off the ground. It's gonna immediately stand up a control plane and a single node for you. You will also need a cube CTL, also called cube control, also called cube cuddle or cube for any of those, and it has become a little bit of a clicky thing who says what? But it's, I'm looking at you, Quinn. You say cube cuddle? I know officially it's cube cuddle because it's got a little cuddle fish, but I just can't, my brain doesn't do that when I read it. Anyway, you need this tool in order to interact with your cluster directly, and you need it regardless of whether you are working locally or you're working on a cluster on a cloud. You just hand it a file called a cube config, which tells it where your cluster is and gives it the keys, how to access it. Helpfully, Minicube places a cube config for you in a default location that cube control knows what to look for. So this allows us to look at our cluster, see what's going on in there, and make it run stuff. Defining what's going on inside of your cluster can be done in a variety of ways. This is also ripped from the Kubernetes documentation. If you wanna just use vanilla Kubernetes, you can write your deployments and service definitions as YAML and apply them with Cube CTL. There are also a bunch of different infrastructure as code tools that you can use to stand up your cluster and deploy things to it. I will show you one of those later, but this is what the deployment YAML would look like for an Nginx deployment if you were to just use Cube CTL. At the top there, you can see a kind of deployment. That means that this file is going to create a deployment object and its only job is to dictate what container is being deployed under what conditions and how many replicas of it need to stay online. In this case, it is two replicas of a container running Nginx at version 1.14.2, running on port 80. A quick note on Kubernetes objects, by the way, there are a bunch of them. A deployment is just one of them. I will also use a service object later, but there are also namespaces, volumes, replicasets, stateful sets, a whole bunch. They also, different purposes, and most of them are beyond the scope of this talk. Frankly, I think you could probably give an entire talk just on Kubernetes objects, just because there are so many of them. And YAML is totally fine, for simple examples like this, by the way. They do, however, get really, really, really hairy if you have something vaguely complicated that you're trying to define. They get to a point where it can be kind of hard to read. It can get kind of hard to manage what's going on.  I also prefer to write in a programming language, if I can, so that I can stand up my infrastructure alongside or even in my application. So because of this, I'm gonna show you what Pulumi looks like instead of the kubectl approach. And I will tell you where to get this code so you can all just stand it up yourself too. Sometime tonight, I will put my slides up online on notice and it'll include links to where all this stuff came from in the Kubernetes docs. So this is what that same Nginx deployment looks like in Python. It contains the exact same information, but it's expressed programmatically and it's a lot more dense. For me, a person who has only ever been a developer or a developer advocate and has never worked in ops, this is much easier for me to read. And I know that that's not true for everyone, but I like to not have to learn a new syntax if I can avoid it. But yeah, this is the exact same information, though instead I've got six replicas, yeah, six, and it's running a more recent version of Nginx. But to make this thing actually functional, a deployment is actually not enough. We need a service too. This is a service deployment expressed in Python. While deployment objects dictate what's running and how many of them, services in Kubernetes manage network access. This one, if I was not using Minihube, would stand up a load balancer for us if we were running on something like GKE instead, but Minihube does not support those. This is one of those places where I have to take a weird shortcut because it's Minihube. Instead, it is going to give us a cluster IP. That decision making is handled by the presence of a config value that I set before actually running anything. So I got all that Python and I wanna turn it into an Nginx deployment running in a Kubernetes cluster. You just need to, in my case, run, pull me up and it'll give me a little overview of what's gonna happen if I say yes, deployment and service defined, standing up in my project for this demo. Click yes and it creates the resources for you in about 13 seconds. This is what I mean by speed running and also what I mean by any percent because this is not a production ready cluster. This is like how fast can I get a cluster online and something in it. It doesn't matter how functional it is because frankly for teaching somebody Kubernetes, giving them an overly complicated, like full stack application is in my experience not very useful. It's difficult to grasp. So 13 seconds is what it took to stand up these resources in a cluster which at first when I started using this tool, I found it unbelievable how fast it was and I super did not trust it. But it is there and if this computer behaves, I will be able to poke at it. So we have all of this stuff here. It's got an output at the bottom, front end IP. That is the cluster IP because again, this is a mini cube which does not support load balancers but we can still look at our engine X deployment outside of the cluster and prove that it's online. So another mini cube oddity, I do have to do some port forwarding to be able to actually access that. But fortunately it is a one liner with cube CTL and we only need a little bit of information from our running application in our cluster to do it.  If you run cube CTL get pods, you can see that I asked for two pods to be up and they are up.  You can see it's got two out of two ready, two available to up to date. And it had been online for 42 minutes when I took the screenshot. But what we actually need is the name of the service we stood at because remember that is our network access. So you guessed it, cube CTL get service and it lists the services we have running. If you grab the name of that container there, you just forward the port for the container to my local and then you can call it and see that engine X is online. Well, I'm gonna try to get this computer up there without slides so that I can actually open up my terminal and let's see if I just kill Chrome, it'll go. And then we can look at stuff running. Let's find out. Thank you.  Let's just mirror it. So do I still have things running? I do. So the whole cluster is still up here and a thing that I wanted to show you what I mean by Kubernetes scales fast is we've got two pods there, right? So say that we need this to be suddenly much bigger. Literally just add more replicas and update the config. It's gonna say I've got one thing to update. Letter rip. And now there is six of them immediately. So this is so fast because it's such a small workload, right? It's literally just one deployment, one service. It is engine X, but it's still, Kubernetes is a little bit more approachable this way. When you're working locally with a really small workload and it's a good starting point, right? Engine X is a thing that many of you have probably had to use in the course of doing your job. You at least know, this is one person raising their hand. Is that big enough? Yeah. Let me see if I can get it. Does that go? Ooh, it's computers starting to struggle a little bit there. Yeah, so we've got our six pods online, but this is a good starting point, a good jumping off point from like building something yourself a containerized application in Kubernetes because it can just like pull containers from wherever. You can build a piecemeal application if you want. You can also write your own application inside of a Docker container, chuck it onto a public registry and go from there, right? So I know that demos like this aren't technically impressive, right? There's not a lot going on here. It's not very complex. I've only got two pieces. I don't have like database interaction or like a whole bunch of third party stuff. I did not stand up a service mesh or whatever, but try when you're like learning something new, especially something as big as Kubernetes, not to like jump straight into the deep end. It's more important to understand like the foundational aspects of how a tool like this works so that when you do get more complex and start adding on pieces to a demo like this small, it is a little bit easier to intuitively figure out where you're going wrong if things aren't working the way you expect. If you understand like the way the pieces of Kubernetes air out with each other, you're not gonna get lost as easy. It might not be as exciting, but it is, you know people advise that instead of like banging at your keyboard for 30 minutes, you instead spend like 15 minutes reading the docs. It's that. It's that. Do not just like go in blind with a wildly complicated cluster with you know, 15 microservices making up a JavaScript website. It's like not, it's not necessary. It's not necessary. It works instead. But this is what Kubernetes looks like. This is how it works at a very high level. If you would like to learn more about Kubernetes, if you wanna go for like a, you know, 200 level situation, the interactive tutorials on the Kubernetes website are actually pretty good in my experience. I really, really like them. They also use Minikube. So it'll be a familiar tool And also, the CNCF Twitch account and the Kubernetes Slack are both super helpful. The Kubernetes Slack in particular is very, very, very active. There is a Getting Started channel and I cannot repeat this often enough.  There is no, like, don't apologize for not knowing the answer to something. We want to help you learn as much as you wanna learn and nobody's ever gonna make you feel bad here for not knowing something because I didn't know stuff for a while and there's still, like, so much stuff I don't know. So come, come help. And then once you're off the ground, you know, pay it forward, answer somebody else's question that there may be a little bit embarrassed to ask. And now I will unplug my computer because it is still screaming and I don't wanna give somebody a motion-sensitive migraine because of the flashing on the right side of my screen. But if anybody has any questions or comments, let her rip. Can you use Pulumi to manage home-based deployments? Yeah, yeah, yeah, you totally can. There is a Helm provider. It also, like, doesn't really, there isn't really a whole lot Pulumi can't manage as far as, like, Kubernetes resources and, like, cloud-native stuff, but yeah, I use it with Helm all the time. Yeah? Can you put the picture of the dog up again? I didn't get to take a picture. Yeah, yeah, absolutely.  Okay, yeah, no problem. More questions? That's, that's fine. Is... No, no, no, it's okay. You know what? The dog came from Ian Coldwater's kid, was writing a talk and tweeted asking, like, what, what is Kubernetes? For people in the room who don't know who Ian Coldwater is, they are a Kubernetes security expert and also very popular on Twitter. So, let me get the dog back. But Nara, Ian's kid, tweeted asking what is Kubernetes and I responded with that dog. I was, like, super hungover. But I just responded with this dog and I was, like, I can't explain it, but this is Kubernetes and now I've explained it. So, yeah, there's my preferred Kubernetes architecture diagram. I have one question. Oh, sure. Is there a path of migrating from mini-cube to a real Kubernetes cluster? I have found in the field that developers end up putting the stuff in production stuff in mini-cube. If you have been building your entire app in mini-cube, mini-cube does have a lot of limitations that you're not going to see with something like GKE or EKS or whatever. And I would not say that you can, like, directly port a deployment that was intended for mini-cube, taking the, like, shortcuts and roundabouts that mini-cube requires to a full-fledged situation. There are issues that you have to consider and some of them are those, like, networking problems that you saw me go through. Like, those problems, you can't port that directly over to GKE. You are going to have to, like, use a real load balancer. Do not return the cluster IP. You can return the cluster IP, but, you know, don't. Right, so then... Right, also there's a mini-cube stubs out all the storage. Yes, like, mini-cube is, it's got a lot of, it's not intended for production. It's like, it's extremely not. Use that for testing. Use that for a playground. Do not, you try to use mini-cube for a pod. It's got all kinds of issues that, like, cannot be addressed in mini-cube, security issues, networking issues, storage issues. So don't build from the ground up with mini-cube in mind. It can. I'm not sure, like, what, from a technical perspective, prevents the load balancer objects from being available in mini-cube. Go in, do you know? No? Hi. Yeah. Yeah, do you want to repeat what people are saying if they're not on the mic? Oh, yeah, sure, sure, sure. The question was, like, can, in mini-cube, is there not peer-to-peer connections available between pods? They're, like, it can communicate between pods, but I am not sure why the load balancer object doesn't exist. Does anybody actually know the answer to that? Yeah, it does. Good point. It kind handles it, though, so. Yeah, I have a related question. So you're saying don't just do your entire development in mini-cube, but sometimes it's useful to have sort of a development replica of your production cluster so you can do local development. So I'm curious what the story you think is appropriate there is, do you, how do you end up building that such that you can have better developer experience for people working locally? Yeah, so there are just special considerations for, like, for mini-cube that, unfortunately, you can't get around if you're trying to clone your production environment. I think kind gets a lot closer to having, like, a true ability to clone a production environment. That is what Gwen here in the front uses. I like kind. I'm using mini-cube for this strictly because it is what the Kubernetes interactive tutorial on their website uses, so I wanted to make sure that there was some consistency there. But yeah, unfortunately, if you are trying to use mini-cube as your local dev environment and you want it to be a true clone of prod, you are probably going to have a bad time and you're better off just using kind or something like it's a little bit more flexible. There was somebody in the back that had a question. Thank you, Josh. Hey, thank you. So does this mean that, like, if the pod owns that the dog will just eat the hamburger? The dog does not get to eat the hamburgers. The hamburgers are reserved for users, which are our friends that are coming over to eat those hamburgers. Unfortunately, the dog is just there to make sure that the hamburgers exist and perhaps protect them from other dogs. Gotcha. Yeah, well, maybe the cat. I find that, like, using kind, because you can use programmatically, use the animal to spin it with multiple clusters as well as rancher desktop. Additionally, you may want to add metal LB to get that cluster happy. Okay, thank you. Hello, more of a Pulumi question rather than a Kubernetes question? Sure. I'm wondering if there's any good static analysis tools since we get to use real programming languages. I know, like, a really basic example. I'm more practiced in CloudFormation, so one really annoying thing, right, is if you don't set your auto-scaling group properties correctly, like, oops, I set my desired server to less than what I said my minimum was. Submit the template and then wait 10 minutes, and then you're like, oh, you didn't do it right. So I'm curious, you know, since we're using real programming tools, or real programming languages, if we can take, if there are any good tools to take advantage of that, to do this kind of checking and make sure, hey, nope, don't submit that yet. There's some mistakes. There may be a third party one that I'm not aware of. There is, the closest thing, like, built into Pulumi is if you run Pulumi preview, it's going to give you an idea of what changes are happening, and some of those potential changes can cause a failure on Pulumi preview. So some of them you will know in advance if this is just going to barf, which is good. But as far as actually analyzing whether or not you've made a logical misstep in your code, Pulumi doesn't do that itself. There may be a third party tool. Gwen, do you know of any? Gwen is also, she is an engineer at Pulumi. Are there any stack analysis tools for Pulumi that would identify when a, you've made a logical error in your deployment, like setting a minimum different from a maximum in a way that, like, conflicts? For Pulumi? Yeah. Like, you would just run Pulumi preview to see a lot of that stuff, right? Yeah, you see, you run Pulumi preview, and the info output should have most of the standard logs, and you can also access diagnostics, run it in verbose mode, all of that, yeah. Yeah, do you know of a third party tool that goes more in depth? I don't.  That's a hot tip, actually. Don't tell my boss I said so, though. This is recorded. I'll also say that because it's a complicated stack, there's a whole industry in analysis tools, so look in the CNCF landscape. Yeah, definitely. I realize that's a bit of an adventure, but there... You're asking for a lot here, Josh.  Oh, and you absolutely need an observability tool. Like, do not just, like, go pawing through logs by yourself, ever, unless you, like, really need to double-check something. Like, there are observability tools that handle that for you and make it a little bit more human-readable, less messy, because the logging output can get very messy. Uh, are there any more questions? Yeah, hi. Hi. Thank you very much for a very nice talk, and Dope is simply amazing. Thank you. Well, it's kind of maybe a new big question, but there were comments that Minicube likes a load balancer, and I was playing with it, like, literally yesterday, and there is a plugin or add-on to Minicube called Ingress, so you can say something like Minicube-enabled Ingress. Isn't it the same as a load balancer, or what was your comment on that? Nginx Ingress is not quite the same as a load balancer. It's an Ingress controller, but that is actually, like, a thing I mentioned. It was why I was setting a driver. If you are standing up Minicube and you are setting up an Ingress controller, it can cause problems with old MacBooks and you need to use a different driver. Yeah, an Ingress controller is a different thing, but in that case, it can serve a similar purpose to a load balancer, so it kind of gets you there, right? It gets you the same result, but it is not the same thing as a load balancer anymore. We have time for one more question. Pulumi, do you use your local Minicube cluster? Do I have that code on this computer? Probably not. It is on this one. So my code makes that decision for me, actually, because I knew that I was gonna be doing this with Minicube, but sometimes I do this with a real provider, and so, so when you are setting up Pulumi in the first place, it is going to look for a cube config, and it's going to look for a cube config in the default location first, exactly where Cube CTL looks for it, which is in the hidden cube directory in your root. So it looks there, and your cube config just needs to contain the access keys for your local Minicube cluster, and then Pulumi will just pick it up, but if you want the same decision-making I had where this can run on a local cluster or this can run on a remote cluster, I use a Pulumi config value, so it's just is Minicube, config require is Minicube, and before I run Pulumi up, I just set that config to true or false, and then it makes a decision from there, there's a little if-else in the bottom of my code. This particular sample code is online in the Pulumi GitHub account in the examples repo. So just github.com slash Pulumi slash examples, and there's like a whole ton of Kubernetes examples in there, and this is the Nginx one. So you can just rip that code, and it will go. And then I think that's it, that's time. Yeah, thank you very much, Kat. Okay, stick around, what's next? What is a Kubernetes controller? Why would you want to replace yours? What would you want to replace it with? Jim will be answering those questions in 10 minutes. Test, test, test, okay. Perfect. If you have your cell phone in your pocket, on the pocket. Interference, got it. Then to turn it off, I just, okay, welcome back everybody. And next up, we have Jim Tario, who is going to equate you with the wonderful world of Kubernetes controllers. So welcome, Jim. Hello, everybody. That's a lot of the wonderful, because Kubernetes can be, it can be great, but it can also be a pin in the ass. So, you know, as we all, if you play with it, it's very useful. And I have been playing with it for the past five years. Actually, so yeah, we'll start off. To break the ice, I wanted to introduce myself as, this is actually pretty much my first time doing a talk at a conference, so I figured I'd introduce myself. Thank you. So, actually, I'm a local. I grew up in San Antonio, California, which is about an hour away if you go a little bit down south. It's in Orange County. My name is Benic. My second generation, parents are from El Salvador. And yeah, right now, currently, family of four, wife and two kids, one year old and four years old. And I graduated from CSUN, so I've stayed in California pretty much my whole life. And CSUN, not familiar with the area, it's right here up in North LA. And I graduated in 2016. And now, since I graduated, I went to work at Blizzard. I actually did an internship when I was in CSUN, and then got hired full-time from there. So I started off as a junior associate, junior, sorry, and then I went to Mannington to senior. I worked in the battle net team. I was there for about three years. Yeah, three years. And it was mainly dealing with the public, like facing APIs and websites, so like, you know, PlayoverWatch, World of Warcraft.com and stuff like that. Currently, I used to go over to team three, which is the Diablo team, which is the Diablo franchise. So if there's like Diablo issues, either one of my coworkers is being called up if there's some reliability issues. And just hobbies of mine, I enjoyed playing poker, fencing football, watching anime, reading manga, One Piece, if anyone was familiar with it, my favorite, I enjoyed snowboarding, and of course, video games. And in case you want to get my social media, I'm somewhat active, I don't use it as much as I should, but you can follow me, it's Jinx. It's funny, because this account, I got it in 2009, I think, and I don't know why I added a three on there. Now I can't get it, someone took my account, I can't even put Jinx on there. But yeah. So, what are controllers? So controllers, they work as a control loop. They try to keep a desired state on your system. So you have a currency, you have a desired state. The community's documentation is a very good example of a thermostat. A thermostat, you set it to, let's say, 70 degrees. It will try to keep that desired state over time. So if the temperature increases, it's going to go back and try to decrease it. So that's how a controller works in your system. So what it does, it watches the state of your cluster, so it looks at the community's resources and it will create them, so let's say, a good example would be a pod, like pods. If your pod goes down, it goes to a state, it goes to a state of zero, but your desired state might be one or two. So the controller will do, it will spin up another pod, and that's how pretty much it operates. There's also a term called operators, which are domain-specific controllers, and these can be very specific to an application. An example would be, you promote the Prometheus operator, and it will look at Prometheus customer resources that you've created or spun up, and it will pretty much, let's say, a Prometheus rule that you create if you create a new resource, the controller's going to look at it and create it for you. I'll go into more details as I go on, but just want to give a heads up, I kind of blur the line between a controller and operator because they are different, but I just like using the term controller. So in case I do a specified application that it's actually an operator, I might have said controller, just a heads up that I'm going to do that. So let's see. Okay, so we'll start off with the first one, the ingress controller. I kind of caught it at the end of the last conversation, the last presentation, we'll talk about ingress controller. So I'll go over three, just because these are supported by the Kubernetes project, but there's a lot, a lot of controllers. Some of you might be familiar with Kong, there's traffic or traffic, I'm not sure how to pronounce that one, but those would be one of the more popular ones as well. So the ingress controller, what it does, it looks at, so it depends on which controller you're using, so like the ingress controller, it will look for, it looks at the, the controller looks at, it talks to the control plane, the API, and if there's like a ingress or service that gets created, it will look at that, and if it matches certain requirements, it will spin up an ingress for it. So, IngenX uses annotations. Use annotations, let's say hostname, and then you will have the address under that you want. The controller will look at it and spin up an ingress with that hostname that you have in the annotation. The reason I have the ingress controller as well is because it's pretty much, you don't need a cloud provider, you can have your own, so you spin up like you have your own cluster and it's in a different cloud or private cloud, open stack or whatever it is. You don't have to rely on a provider. So the AWS load balancer controller works in a similar way as the IngenX controller, but it actually creates load balancers on AWS. So it could create ALBs or network load balancers, and it works in a very similar way where it's annotations that get added. And then the ingress.gc is just the Google Cloud version of it. So these will be three in case you want it to allow external access to one of your services. And those will be the ingress controller. And then I think I'll share this slide I think after or the skill. So in case you want to like the links, I put them on there. Just in case you want to go to the GitHub repo. A lot of these are open source and they also have my preference is home charts. In case you do want to install them, there's home charts as well, but these are just applications themselves. And then as I mentioned, there's a lot of ingress controllers and if you click this link, it's the community documentation. They have, last time I saw it, they'll take 15 plus controllers on there. So in case you wanted to look through and see none of these fit your use case, you can go on this site and go through and see if there's anything that you might be able to check out. Then for monitoring, one of the controllers that I prefer, that I like, it's mentioned the Prometheus operator. The Prometheus operator, it creates custom resources onto your cluster. So you have, for example, if you're familiar with the Prometheus operator, they have, they create a custom resource definition that is called just like Prometheus rules. There is service monitors, pod monitors, and these are pretty much resources that the operator is gonna look at. So in case you do create a resource with that kind, it will spin it up for you. So if you create a Prometheus rule, you can create it, depending on the time that it scrapes, it will spin up either alert and alert manager or whatever, however you set it up. It has service monitors, which is like an abstraction layer in case you want to monitor a service. It looks at labels and it makes it very dynamic instead of having to manually create or make changes or maybe even like an app deployed to add stuff, so it makes it very dynamic. It's also, of course, this one is an operator and it looks at the API for any of these CRDs. And then the Prometheus stack, this one is actually, it's not a controller, it's more of a home chart. And the reason I put it on here is because it's super useful because it installs the operator, it installs Grafana for you, it installs alert manager, and pretty much end to end stack of Prometheus. So once you install this, I won't give a warning that the values file and helm is very large and it's massive. So if you go through it, it can be kind of painful, but you get kind of adjusted to it and you can kind of see what you can do with it. But a better minimum, you can enable everything and you would get even the metric server to scrape metrics by default, you get default Prometheus rules that are super useful. Yeah, so then secrets, secrets, if you play with communities, like they say secrets but it's actually just a base foreign coded string that you can cat it or you can view the resource and you could decode and get the string. So that's not so secure, but how do you put this in, GitOps is very popular, how do you put this in source control? And it's not good to put passwords in source control just because, I mean, obvious reasons. But my favorites is external secrets just because it's very agnostic and what it does, it allows you to create secrets without having to like hard code them. It looks at a provider, it pulls them and injects them onto your community's cluster. So same thing, it's an operator and the external secret management systems which can be like AWS Secrets Manager, it could be the Google one, yeah, Google Secrets Manager and then of course agnostic would be Vault. It's not specific to a cloud, you can spin up your own version of Vault or spin up your own Vault instance and manage your secrets there. So the external secrets makes it very easy to play with these different providers and you can go for that cloud agnostic way or you wanna go cloud specific you can or if you wanna, you have multi architecture, infrastructure, you can have one external secrets that can talk to different providers which is super neat. And it's also neat because it's very dynamic. So as the operator, it always watches. So there's a change on the secret, let's say a secrets manager, you update the secret and depending on how your app's configured, if you update the secret, it is very quickly the controller updates the secret on Kubernetes resource. So you don't have to actually like go and apply a change onto the Kubernetes cluster because you just update on secrets manager and it updates it on the secret on the cluster. And if your app can make changes in runtime, you don't have to do any restarts, you can do life changes and let's say a password gets exposed or something happens, you could just go on secrets manager and update that secret. And you also have, depending on your secrets manager setup, you can have versions. So in case someone deletes a secret or you need to get the old one for some odd reason, you can just look at the versions and pull it up. And yeah, so this one's kind of different because it actually is not talking to the API for those changes, it's actually talking to the provider for the controller to keep that state. And then DNS, our favorite, external DNS would be my go-to just because it's also kind of similar to external secrets where it could be quite agnostic or different providers. And it works in a similar way as the NGINX controller where it looks at annotations. So you have an ingress and you add an annotation for external DNS, it will create that record. And yeah, so pretty much it's just annotation based. You add a annotation to your ingress and you'll get DNS and then auto scaling. For this one, I put a vertical pod autoscaler. What it is, it dynamically sets resource and limits on your pod based on your usage. So in case sometimes you spin up a application or service and you're not sure what do I set from a resource consumption, right? Like you can put limits but you truly don't know unless you're kind of just guesstimating. The VPA, what it does, it kind of does that work for you. It looks at trends and it will scale your deployment or pod with those resources and limits based on usage. One thing though, the VPA can be pretty dangerous in my opinion, it uses admission controllers and admission controllers are different type of, it's a controller and what it does, it intercepts communities API request before modifying an object. So VPA, if you create a pod and what it does, admission controllers have one thing called mutating web hooks and validating web hooks. So depending on the application, it can use either one or both. And so if you create a pod and it will intercept it, if it's a mutating web hook, it would modify that request and then apply it to the communities API. One of the reasons why I say it's dangerous is because it's intercepting request. I've had issues where the VPA had issues and the cluster couldn't scale any pods because every request for the pod creation was getting stopped by the controller or by the mutating web hook and the validating one. And so we were like, there was enough resources the nodes were there, but for some reason we would scale the say pod to like 50 and it just, nothing would happen. It would just be context that line. I'm not sure if anyone's seen that before with the play with communities and it's super vague because it could be anything, it could be timeouts, it could be config errors. So we just get constant context deadline errors and then it was the VPA and the mutating web hook that was stopping the create action on the API. So this one has to be careful as well because also the trends, if your traffic is very spiky, like it's very dynamic, there might not be a pattern and it might not give you the best recommendation based on the behavior. And it does come with three components. So the recommender, the updater and admission plugin, the one I recommend is the recommender and this one, it runs on your cluster but it doesn't apply the changes. So what it does, it tracks your trends, it sees your usage but it doesn't apply them. So this is the more safer route because you see what's actually recommending but it's not gonna change your resources. So you can see from there that you can update your deployments with those resource limits that you want rather than having the VP dynamic skills for you which can be pretty risky especially in production. The other one would be certificates. This one, if you have either like an internal, you do self-signing certs or you don't want to pay for certs because there's let's encrypt, you're unfamiliar with it, you can pretty much get free certificates. There is some throttling and so you have to be kind of careful with using it sometimes but you can get certs for free rather than paying a provider for them when it's all totally free. It also makes it easy to renew certs so you don't have to constantly, I'm pretty sure everyone or people that are working production have had a cert expire and it's just one of those things where kind of shake your head and you're like, crap, like not another one, you chat a modern one, it's always one that's missed. Cert manager, what it does, when it's getting close to the expiration time, it would renew it for you and hopefully if you have a learning, you can kind of get a heads up even before that because even if you have to watch cert manager, make sure it's not down or having issues because that's another point of failure. It uses vault, benefit, if you have your own private PKI and of course let's encrypt. So all this is great and it's so many applications but how do you install it? I'll put this more like high level which is because I don't think it's part of the, you want to learn by controllers, not deployment tools. But there's many methods in case you do want to install it. You can use two other ones for packaging your applications as either home or customized. Personally, I like Helm myself. Customize, at least for me, has been difficult to read but I've seen some people do some awesome things with it. And then you have automation tools. You have Argo CD, I'm not sure if you're familiar with it. It uses, which is tricky, it uses Helm templates, not Helm itself.  But Argo CD is awesome because it, at least from my experience, the maintenance has been almost nonexistent. It practically just been running and it gives you a kind of a visual of your applications. So you can see all the resources. You can see the controller, you can see the pods, you can see the staple sets of services. And then we have Spinnaker which is another deployment tool. This one was built by Netflix. It's a pretty much, it's built a bunch of microservices and there's use cases for it and it could make a Helm chart for you. Depending on use case, I found it too cumbersome to install and to maintain just because of different microservices. And then Jenkins, which either you could do simple Jenkins job and deploy an application or even Jenkins, personally I don't have experience with that one but it is an option. And then Terraform, Terraform you can use a Helm provider in case you do wanna stick with Helm. Terraform and Ansible is kind of like a, you're kind of crossing a thin line because in my opinion, Terraform should be infrastructure. It shouldn't be managing your applications but there are use cases for it in case you wanna spin up a, say you have a Terraform module or cluster and you wanna boost up a cluster and you wanna auto install something right when you create a cluster. Like let's say you wanna install external DNS and external secrets automatically. You can use Terraform in the Terraform provider so when you spin up a cluster via Terraform it already has the components there for you. So there's use cases but I would say it's a thin line that I kind of be careful that you're not managing all your applications with Terraform. And then of course with great power comes great tech depth. All this is neat and it's nice but it's a lot of applications, right? You're running this in production and it's open source so there's different release cycles, there's vulnerabilities, there's all different kind of things that you kind of have to stay on top of because you can get very behind. I'm pretty sure like in the past month like just applications had like either five chart releases and yeah, so you have to be conscious of all the different apps that you're installing and also always, it's difficult to kind of keep track of them but I would say it's like a cadence. My preference is every either, every quarter, like you can even follow the community's release cycle and follow, like don't leave it behind because you'll get very behind and then what would happen, in my experience what has also happened is that communities have been deprecating and removing APIs. So if you have an old version of application it might have an old API that it's using. So then you kind of have to update because you have to upgrade your cluster but then you get in this line where now you're like, oh crap, like timbers is behind and now it's like there's a bunch of breaking changes that happen in between those and now you're either dealing with fires or you're not spending more time than if you were just to gradually update every quarter and update your charts every quarter. That would be my recommendation but it's very difficult to keep track of everything. Right now I think I just mentioned 10 applications that would be running and then all those have different cycles. And yeah, so actually I went faster than I thought but I'll leave it open for questions and also wanna put some closing words and this is completely unrelated to everything in this presentation but I just want to kind of say, don't forget the golden rule, treat others so you wanna be treated and in the current environment just being nice to each other, be respectful and help each other. This is the open source community and we're here to learn from each other, help and this is how I started myself. Like I came to conferences, open source conferences and even mentors that helped me out and asking questions, failing fast and just help each other and as you start off this, go on in this conference, talk to each other, network and just help each other. That's it. So with that, do we have some questions? Don't give me a hard one. You have any experience working with Helmphile and can you say anything about that? Which one? Helmphile? No, what is it? I'm curious. It's a wrapper. No, I haven't had the pleasure but I'll look into it to you. I mean, we have plenty of time so I have a quick overview if anyone wants to learn something. Would you mind? Well, it's a wrapper around Helm. Helm. Gotcha. Okay, awesome. Sorry? Who's familiar with Helmphile? I was asking if there was any knowledge of Helmphile which acts as a wrapper around Helm and facilitates automation. Yeah, I may have the chance. I've mainly been using Argo CD and that one you pretty much, depending on how you set up your repo, now there's application sets which makes it a lot more dynamic but it just values file and there's a hierarchy and it just merges them all. That's primarily my use for Helm so far and a little bit to reform but just Argo CD has been my main deployment tool. Okay, we had another question. There was actually one in back there who was in the back. Yep, this is my exercise for the day. Thank you. Great talk today. I'm a total newbie for Kubernetes but can you just confirm that the controller is the beginning of like turning an application into like a LAMP stack application, what you would need? Do you need the controller basically to set up a Apache MySQL or like Nginx MySQL PHP and have them talk to each other? Yeah, so the controller, so there's a lot of controllers in Kubernetes. There's already some that are baked in when you spin up a Kubernetes cluster so like for pods and just the native Kubernetes resources, there's already controllers for them. The ones I specified are kind of extensions of the ones already that are already in Kubernetes. Pretty much the ones that I mentioned, they're more for in case you're ready to make your application external. Like once you say you did your hello world, you have a LAMP stack that you spun up on Kubernetes and now you wanna apply it to, let's say you wanna access it remotely. You have external DNS, I would point to the Ingress if it's external. But yeah, all those would be pretty much just to kind of extend whatever, because you can just install like, I think you can use micro Kubernetes or K3s, I don't express that one, but you can use one of those and you can spin up a LAMP stack just like that. You can just have a deployment file or a pod file resource file and you just add your containers that you wanna spin up and you can just easily spin up a LAMP stack like that. But these would be an extension once you kind of wanna go on a Buffet Beyond. So let's say you've done like Kelsey's high tower, the hard way and you're like, I'm a cuss, fuck, like what do I do now, right? Because you spun up a cluster and what do you do now, right? Like, what can I do more? And these would kind of help you kind of answer that question as to, what can I do now, right? But yeah, in case you wanna chat after, I'm open to Kubernetes. There's a lot of things, even stuff that I mentioned here, there's just open source and there's so many things going on and it's very fast. But if you wanna chat after, I'm really happy to. Over here on, I'm glad you brought up the technical debt because the cadence of changes in a Kubernetes cluster are just crazy. So as far as the people portion of that do, is there one team who's managing all the helm charts for the infrastructure or is it broken up into, like the networking team is dealing with the Ingress Controller and the security team is dealing with secrets controllers and things like that. It just seems hard to manage and you have to. Yeah, so it really depends on, and you're referring to like, let's say a company or where I work, right? Like, so it varies by my team, to be honest. Some teams or the company, they might have the bandwidth or the money to pay for an infrastructure team that can manage this. For us at Blizzard specifically, we're a team of, at least for the game teams, we're a, let's say 30 SREs and we're split up in different game teams. And we're kind of siloed in that way where like me and two other people, we maintain our Kubernetes infrastructure. So we maintain all these charts and we have our own workflow for Kubernetes. We have our, of course we share, we kind of try to not do things twice or doing the same way or do double the work and we try to share everything and try to be agnostic in the way we create modules or stuff to share. But yeah, at least from our side, we maintain our own infrastructure and that's why we try to be on top of these things because we've already been bit by this multiple times and it's a massive undertaking of maintaining our cluster just because there's so many components that are just constantly changing. Even the Kubernetes versions, they, I think they reduced it from four to three, I think every year just because they were releasing versions every cycle and it was getting too much, it was too much for people, I think. Okay, we had, yeah, thank you for the talk. You just slightly touched the question how controllers and operators are related and do I get it right that they're kind of similar thing or can you give some more insights on that? Thank you. Can you repeat the question? I missed the first part. What's the relation between operators and controllers in Kubernetes? So an operator is more domain specific so you have like, it could be application based so you could have like a vault operator that just looks at vault resources and the vault operator might have like custom resources called like CRDs and it would look at those for like events in the API or changes and it applies, dependent operator applies resources based on that changes to the resource. The controller would be kind of like the, that are native to communities so like the job controller would be one. I think it's called Pod Controller, I'm not sure if I might get the name wrong on that one but they're all native to Kubernetes resources and they're already like baked into Kubernetes. Operators are just either someone built it for their specific application and their custom resources. Don't a lot of operators contain a custom controller though? So they have to, actually you stump me on that one because they do a track with the different controllers depending on what events you're watching but yeah, I'm not gonna have to ask that question, I'm stumped, I'll probably follow up with you and give you a better answer, yeah, sorry. Yeah, usually controllers are just like, looking at the world and iterating and an operator is a controller coupled with CRDs that have custom resources that it's operating on and so like the controller pattern came first, it existed for a long time and then they just kind of named this pattern afterwards called the operators where you create your own custom resource and then you modify objects based on CRDs. There you go, smarter guy than me right there, answered it, thank you. Okay. So I've got a quick question on Argo CD for rollback or canary deployments. That would be an awesome tool. So my question is for rollback, what would trigger a rollback on a canary deployment? Would it be a crash loop event or would it be a health check endpoint on a deployment? So it depends, Argo CD just does like the native like deployments, so depending on the deployment how you have it set up, depending on the health checks you're checking and what's the other one? The amount of paths that are unavailable but there's a thing called Argo rollouts which is an extension of Argo CD it's maintained by the same I think Intuit the company that built Argo CD. There's a, they have another company called Argo rollouts which has those blue green deployments canary and those have certain conditions you can set. So let's say you have a percentage or something for deployment, rollouts will look at that depending on what thresholds you set and it will rollback depending on the threshold you set. Any more questions? Have maybe it's more like a comment or a question to the audience. Since you haven't noticed this scaffold within this automated deployment and maintaining things. So I wonder if anyone has any experience running scaffold for like production environment? You mean scaffold with a K the cloud native project, yeah? Okay. I think it varies by, you know, just like the company or whatever you're trying to do. I think everyone might have templates and stuff but production ready is a very, it's a loaded term I would say, maybe someone has a better answer but there could be many different ways you can do something and set different templates. Not sure if there's anything open source that kind of has something that would be production ready. Yeah, I don't think I have a better answer for that one. Sorry. Okay, one more here. Scaffold is a great tool for development but it's also a CLI. So if you're using scaffold in production that feels like a single point of failure. If ever that CLI stopped or if ever the box that was running that CLI was reset then your deployments would stop. I think I would rather some cloud native something that was continuously monitoring that did the thing. So would I wrap scaffold in GitHub actions or you know, something like that? I think I'm starting to over engineer it. Other questions? comments? Okay. Awesome. Well thank you very much, Jim. Thank you everybody for attending lunch time.  Two, test. One, two. One, two. Test. Okay. Hello everyone. Please come in, take a seat. We're gonna get ready in just a few minutes here to talk about data security using Rook and Seph. But yeah, please have a seat. Oh yeah, also I'm totally smiling behind my mask and you all can't see that. I'm trying to look welcoming. So instead I'll be awkward on the mic. All right, welcome back to the cloud native track. I hope you had a wonderful lunch. And we are ready to kick off the afternoon with data storage security using Rook and Seph with Anna McTaggard and Federico whose name, my first last name is... It's a hard one. Say it again for us. Lucifred. Lucifredi. Please welcome them and take it away. Thank you. So nice to see you all after two years. Now my marketing manager would not forgive me if I gave up on introducing myself point. So here is the short version. I had the privilege of spending my entire career in open source software. I'm the product management director for the Seph platform at Red Hat. Before that I was the Ubuntu server at Canonical. If you know that was my favorite. And if you go back to decade, I was the dreaded systems management czar at SUSE. Here's a list of a few things that I worked on. Slas is a SUSE Linux enterprise server, but that's what we call it. I was the maintainer of MAN for about 10 years. That's the MAN one there at the bottom. And shameless plug, I have a book on AWS operations out by O'Reilly and that's why you see me all surrounded by clouds there. And I'm joined by Anna. Hey, I'm Anna McTaggart. I'm newer at Red Hat, but I work on cybersecurity here. And I did my undergrad at UMass Amherst graduate school at UC Santa Cruz. I went from sort of an academic background in deniable file systems, moved to oblivious computing, to formal methods, to incident response at Red Hat. And I also work on a formal methods community of practice at Red Hat and also SAP security. And in my free time, I like to hike with my dog and garden a lot and work towards a more inclusive world. Righty, so if you're here, you probably already know what SAP is, but the 62nd version is that SAP is the dominant software defined storage solution in open source. We work on SAP like to think it as the future of storage and we're quite proud of the community that we built around the project and its technical prowess. SAP is highly scalable or horizontally scalable, highly available, highly resilient, capable of serving file, block, object. And that's why there is more than three exabytes of SAP out there. Many other things can be said about SAP. It's the bee's knees, is the short version. But you are here, so hopefully you already know that. If you don't, go to see Mike at the SAP booth this afternoon and he will give you a proper education. But for now, think of it as the Linux kernel of storage, if you're not familiar. Less famous, but equally awesome, Rook is the CNCF project that fronts storage and it's basically a SAP operator. Rook helps reduce the operational burden a storage team faces by delivering cloud native storage for Kubernetes. Creating horizontally scaled and self-healing clusters in SAP is complemented by Rook making them increasingly self-managing and even self-scaling if you want to look at it that way. Rook enables SAP to deploy on Kubernetes with ease, enabling all the benefits of container orchestration on what is now the dominant private cloud platform. Storage running on top of the compute infrastructure like any other infrastructure would or an alternative storage running on bare metal as an external entity for petabyte scale storage delivery to many Kubernetes clouds. Rook is the storage operator, so now the stage is set for security of these two fellows. Now, security practices harden a specific point of the infrastructure, cherry picking practices without the model of the threat and the attacker is not a viable strategy. The joke usually goes, if you want to protect from all possible threats, you need to turn the computer off, bury it in concrete and drop it at the bottom of the ocean. In other words, absolute security is not usable and probably not even possible, but that's secondary. It's just not a useful thing. You have to have a model of what you're protecting against. So, you're defining your security in the context of a threat model. Are you facing script kiddies or the GRU or the dreaded privilege, the insider, that we should all be worrying about? These are very different scenarios and the things that you do are significantly different. Some of these want to steal your data, others want to crypto lock your data and hold you for ransom, others yet may be satisfied by causing disruption, deleting a few files or causing a transient denial of service. So, with that in mind, let's dive right in on what things you can harden specifically. Let's look at network security with Seth and Rook. Let's dive right in. The public security zone is an entirely untrusted area of the cloud. It could be the internet as a whole or just networks external to your cluster that you have no authority over. Data transmissions crossing this zone should make use of encryption. And note that the public zone, as I just defined it, does not include the storage cluster front end, the Seth public underscore network, which defines the storage front end and properly belongs in the storage access zone. The Seth client zone refers to networks accessing Seth clients like the object gateway, the Seth file system or block storage. Seth clients are not always excluded from the public security zone. For instance, it's possible to expose the object gateways as three or Swift API in the public security zone. Next, the storage access zone is instead an internal network providing Seth clients with access to the storage cluster itself. So the cluster zone refers to the most internal network and provides storage nodes with connectivity for replication, heartbeat, backfill, and recovery tasks. This zone includes the Seth cluster's backend network called the cluster underscore network in Seth. And operators often run clear text traffic in the cluster zone relying on the physical separation or VLAN separation of the network from any other traffic. This, for example, would not be a valid choice if your threat model includes adversarial privileged insiders, going back to what we were just saying. And these four zones are separately mapped or combined depending on the use case and threat model in use. There is another network that is common, but usually, but not properly part of Seth itself, which is what you could think of as a management network. It's nice to have a separate network that cannot be flooded by user events or service events for things like SSH, monitoring, maintenance, Pixie. So that is not one of the Seth networks, but most Seth users or many Seth users choose to have it physically separate for insurance reasons for special situations. So with that model, components spanning the boundary of two security zones with different trust or authentication requirements must be carefully configured. These are natural weak points in network architecture and should always be configured to meet the requirements of the higher level of trust between the zones that are being connected. In many cases, the security controls should be a primary concern due to the likelihood of attack at this point. Operators should consider exceeding zone requirements at integration points, which for a storage product is actually often easier to accomplish than in the general case. Your storage system may be able to serve any kind of storage, but in your specific enterprise or agency, you may be using it just for one type of storage, like object only. And that gives you a much smaller envelope that you can use to narrow down what you're allowing people to use. For example, the cluster security zone can be isolated from other zones easily because there is no reason for it to connect to the other zones. Conversely, an object gateway in the client security zone will need to access a lot of things. The monitors on ports 6789, the OSDs on ports 6800 to 7300, depending on how many you have, and will likely expose to the world its S3 API to the public security zone on port 80 and 443. So different SEF demons have different characteristics and you have to leverage them accordingly. Now let's move from network to encryption. Server side, Red Hat customers overwhelmingly choose to encrypt data at rest. That's, I stopped tracking this in 2016. At that point, more than 60% of our customers were using at rest encryption, and it was rapidly going up. It's basically a default. And the mechanism that they used to encrypt the data at rest is the Linux Unified Key Setup Mechanism, better known to all of us as LUX. All data and metadata of a SEF storage cluster can be secured using a variety of DM crypt configurations. And the security best practice here is to locate monitored demons on separate hosts from the storage demons. So the mons on separate hosts from the OSDs. And we do this because the mons keep the DM crypt keys. So if someone walks away with one hard drive, they don't have the keys because they're on the mon. But if somebody walks out with a box because the box has either an OSD or a mon but not both, they cannot decrypt the data. They don't have the keys for that. Ensuring anti-affinity of the keys and the data they encrypt, this results in physical removal of a host, not including its decryption keys. The Object Store Gateway has additional capabilities, including at encryption, at ingestion. The use of per user keys as opposed to per drive keys. There are rotation using tools like Vault and soon SFADM. Support for Amazon AWS, SSE, KMS, and more. Department of Defense certified ciphers under the FIPS 140-2 standard can be used if they are supplied by the operating system. So you can get those on RHEL or on certain versions of Ubuntu. Here, the thing that's interesting to compare is you can encrypt the disks with the encrypt and have at rest data encrypted just because it's on the drive. In that model, you have operator-managed keys. The operator is managing all keys for you. You, as the user, are not doing anything with the keys. The alternative is that you use at ingestion time encryption. So RGW can encrypt when the data is ingested or the crypto and retrieved by providing a user key at runtime. So if you're doing this, the keys are user-managed. The responsibility is all on the user. But there is also a slightly different thing in that the encrypt, once the drive is open, it's available to the system. And, OK, you have to be rude to access that drive. But in a sense, you can access that data. Well, in the RGW ingestion model, the key is only there while writing the data or retrieving the data, and then it's gone. So the exposure time of the data is also significantly smaller. So lots of trade-offs to play with as you build your model. Network communication can be secured by turning off SAF protocol encryption in the Messenger 2.1 protocol introduced with Nautilus, which is a recent release. Here, clear text on the network front. I told you that almost all of our customers encrypt data at rest. On the network front, the practice is that customers encrypt the data as it goes out of the cluster to their application. Depending on what the application is, block or object, they encrypt it accordingly in different ways. But the internal SAF traffic is overwhelmingly in clear text. With Nautilus, the option to encrypt the internal network traffic became available. This is relevant in two ways. One is because there are parts of the SAF system where the SAF protocol is in use to the client, like CFFS. In that case, the only way you could secure that connection is by encrypting the SAF protocol, so you need that support. The other part where it's relevant is that a significant number of customers asked for this, and I wasn't quite clear why, because it seemed like a silly thing to say I'm going to encrypt all the traffic that's going on a network that nobody can touch. So after a couple of customers came with this, I said, well, do you have a Snowden problem? Do you have a privileged insider problem? Is that what you're trying to protect against? And they said, yes, we have a requirement to protect against privileged insiders. Now this was a customer discussion, so you cannot tell a customer that they are wrong, but it's extremely difficult to protect against a privileged insider. You could argue that it's impossible if someone is sufficiently privileged. So in this scenario, it seemed like a weird thing to say. Then after a while, I built a different pattern, which is all of these customers were coming from the same country, and they had a regulatory requirement that that country's telco agency had created to make NSA eavesdropping on their citizens harder by basically encrypting all links no matter what or where. So that finally explained why this was happening. But this is still useful functionality. In most cases that are not regulatory driven, this can still be helpful, because let's say that you're a large enterprise and that you have a policy that everything must be encrypted no matter what. You can go and argue with the security team that they don't encrypt the backplane of a NetApp appliance, and so they shouldn't encrypt the private network of SEF and go back and forth until they give you permission. Or you could buy a few more Intel CPUs and not have that discussion. So it's a very good way to bypass certain authorizations and why not? It's going to make you more secure. And since the overhead of encryption can be generally shadowed by the latency of the fact that this is a network storage system, so you add a little bit more CPU to come up with what's needed to support the encryption pass itself, the performance of the cluster for most use cases can be the same to the application. So it's a very valid option in that sense, and we won't judge what your reasons are. Looking at more specific protocols, S3 service is usually secured between RGW and S3 client with TLS, obviously, on port 443. Also, RGW very often serves clear on port 80 because you could use RGW as a web server and not all web server use cases are necessarily encrypted. TLS termination at HA proxy in front of RGW is a special case, so you'll have to decide what's happening in the link that goes between HA proxy and RGW. Does that need to be encrypted or not on your network? If it's in the clear, it should be in the right security zone. Obviously, I'm not going into standard network practices like firewalling individual nodes to only expose the clear list of ports and stuff like that. But standard network hygiene applies. And now I'm going to hand over to Anna. Hey, so now we're going to talk about specifics for Rook. So Rook can use custom resource definitions to encode a lot of security preferences and settings. For example, we can configure trust certificates for Rados gateways web server. Rook also supports at rest data encryption like we discussed earlier. We recently allowed in-flight stuff protocol encryption in 1.9. This allows us to implement a lot more secure features and just generally get better security. We can do that also to segregate our traffic using the software-defined cloud network fabric. And in addition, Kubernetes user permission system applies to all the persistent volumes. So our permissions, quotas, everything like that that comes from Kubernetes applies. Nothing Rook needs to do here. Rook also supports a key management system in the container storage interface driver which allows individual volumes to be encrypted with their own key. This limits the scope per key, which is a really important security practice. All of this ensures that we can follow best practices easily. We can follow key rotation, revocation, and limiting the scope from each key. This also limits the scope of our unencrypted traffic, which is really important. Now let's talk about the control plan for Rook. As popularized by Ansible, SSH is used by Seth Admin, Seth Ansible, another deployment in day one tools to provide a secure command path for install and upgrade operations as part of host management. This is important so that we are limiting the people that can have access, but still enabling us to get access to host management. The dashboard shouldn't be exposed to the world, but it needs to be reachable by the operator's workstation to be of use. You can't just say no one can access the host management system and then expect us to use it. It has to be accessible somehow. So we're doing this via SSH. And this is done via the support specified. It really limits access and locks it down. We also can have our dashboard access zone be tailored by our operator to suit our local threat model. Again, you probably don't want it to be exposed to the world, but it needs to be accessible to some people. You can have SAML authorization, Kubernetes native authorization, lots of options there. The Seth manager is also accessible via SSH. Now let's talk a little bit more about identity and access for Seth. Seth uses shared secret keys. It protects clusters from in the middle attacks by default. Now you still need to have some good practices. One good practice here is to grant key ring, read, and write permissions only for the current user and route. Client admin user being restricted to route only. You don't want all users to be route. That would not be very secure. But you want to make sure that somebody has route so your client admin user can be restricted to route only. And then talking about RGW, it also supports the key and secret model of AWS S3 and the equivalent model for OpenStack Swift. So those are some more authentication models that you can use. With S3, we also have the bucket policy as an option, which can enable you to basically apply bucket permissions like an S3 bucket accordingly. Now, of course, your administrator key and secret, regardless of if it's S3, native, whatever, Swift, you have to treat it with appropriate respect. You really want to use your administrative users sparingly, which we're able to do in Rook, like I just said. With RGW, the user's data is also stored in stuff pools. And again, we have great ways to secure this, Federico said. So you can store them securely with your data at rest. You can couple with OIDC providers such as Key Cloak, back with your organization's identity provider. This can really give you a granular role or access attributes really granularly and make sure that it's locked down as much as possible. Really limiting your scope of access. We also support LDAP, Active Directory, Keystone Identity Vaults. These are all supported in Rook with Ceph. And again, auditing is a really important part of security. This is an example of a Ceph audit log. You want to check your operator actions against a cluster. They're being logged.  We aggregate these to our log. You might want to aggregate these to your log management system as appropriate. But again, you probably want to delete them at some point. So let's talk data retention. So for example, with Rados, your users generally don't have the ability to read, write, delete objects directly in a storage pool. You can apply a bucket object lock, so all objects in a bucket have the same security requirement. You might even want to have multi-factor authentication requirements to remove an object from one of these buckets. It's a little bit different on a Ceph block device, object gateway, or file system. Your user can create, delete, modify, volume images, objects, and files. But again, with Rados, it's a little bit more locked down. When we delete data from a Ceph cluster, though, whether we be a user or an admin, it generally can't be recovered for practical use. There's a couple exceptions. RBD supports trash bin dynamics with spare pool capacity. This can also have issues with versioning of object store buckets resulting in deleted objects being preserved until deleted by policy or the administrator. But again, generally speaking, once we delete something, it's not accessible. So if you delete your log, nobody can find it, which is great from a security point of view, because nobody can dig through your deleted data.  And then additionally, you might want to be aware individual data blocks, they are present on persistent storage until overwritten. So that is a risk with deleted data. But again, it's generally not accessible. Now, if you're actually retiring your media and you don't want somebody digging through your dumpster to go through and find all your data, you might want to encrypt it at rest and then discard the key. Nobody's going to be decrypting it. That would require pretty intense resources that are probably unlikely. So if you do that when you replace an OSD, just encrypt it and delete it. Or if you don't trust your cloud storage provider to properly clear the disk. So again, these are some options that you can use. We also have infrastructure hardening as some options. These are highly vendor dependent. So we're going to talk about what Red Hat does. Your self-distribution might vary. At Red Hat, we ship with SE Linux on by default in enforcing mode. This is important. It enables greater security and it doesn't require anybody to configure it by default. We can also make use of FIPS 140-2 certified ciphers as supplied by REL. For example, REL 8.2 is our most recent certified version. We regularly certify these. In addition, we also harden certain binaries. Red Hat's self-storage binaries has these options that are listed here, such as Fortrify Source, your F-stack collision protection, all of these options. You want to consult with your vendor to see what is being done to harden your binaries just to make sure that they're consistently secure. These represent the intersection points of the network security zones as we've discussed. In addition, at Red Hat, we run covariate pretty regularly. We run those scans. We look through those scans. We see what's going on there. We also regularly ship fixes for any reported CVE or exploit, working with our incident response team to make sure that nothing is left to be insecure. Now, there's always a risk, but these hardening options really do increase the security. Again, their vendors are specific, but this is what we're doing at Red Hat. And you can apply all of these accordingly. So thank you so much. And turn it back to Federica for any questions. We have a couple of bookmarks for you that we haven't written the paths because they're super long, but you'll get the slides in the next few days, so you can just click on them. These are some interesting resources for pretty much everything that we've discussed to dig down in things like compile time, hardening, which we don't really have the time to go into. At this point, if there's any questions in the audience, yeah. I wanted to ask, you said you delete your logs. Do you overwrite where the logs were or do you encrypt the logs and then delete them? Is that what you're doing as a standard? It all depends on your configuration, I believe. Yeah, you would have to choose how to do it. But a normal practice would be that you don't store the logs or the monitoring data on the system that you're monitoring. So you're not deleting the logs out of SEF, you're deleting the logs out of a Linux file system, so you can use usual sanitizing practices there. If you're not in some very high debug mode, the logs should be pretty clean in terms of not containing anything confidential. But like Anna said, if you're just going to keep a history of everything that happened in the cluster, that potentially is a leak, so you want to purge them at some point eventually. Do you do this automatically with Rook? Didn't, the log management is not automatic. That's why aggregating logs is a good best practice, because once you have them in Splunk or whatever other tool, RSS log, whatever tool you use to aggregate the logs, it's much easier for you to manage them in a central point. Thank you, any other questions? Don't be shy, we got lots of time. All right, well thank you everyone, and thank you especially to Federico and Anna. Thank you. Super important how to keep your stuff safe, and the next session will be on a different angle on safety, basically how to prevent people from hacking into your Kubernetes cluster. Should be exciting. Please join us at two for that. Thanks again. Testing one, two? Sounds like you can hear me. Oh, and welcome to the two o'clock session of the Cloud Native Track here at Scale. I am super pleased to introduce Eric Smolling, who is going to have a few exciting demos about how to break into Kubernetes clusters that are perhaps not as well configured as they should be. So I am very much looking forward to it. Please give him a hand and send a prayer to the demo gods on his behalf. Thank you. Eric, take it away. Thank you, thank you everybody for coming. Good, my slides are up, good, we're good to go. So as she said, I'm Eric Smolling, I'm a senior developer advocate at Sneak, but I'm not really going to be talking about Sneak, so don't worry, not a vendor pitch. We're going to talk today about hacking Kubernetes. So you can get a feel for where I come from, you can read that if you want. I'm a developer background. I'm not an operational background. A lot of Kubernetes folks come from SysOps and learn development. I kind of came the other way. I have about, depending on how you look at it, 30 years of software dev experience and dev ops experience or whatever. Doc or captain have the CKS and all that good stuff. Been using Docker though since early days, like 2013. And I have a tweet of Solomon mocking me publicly that old so you can go check it and check my, if I'm telling the truth. If you care, I'm Eric Smolling all over the socials. But what we're going to talk about today, and forgive me if I keep touching this, this is my notes and I'll be doing things with it, so sorry I had to have that in front of me, but you know how it is. But we're going to be talking today about how the combination of app vulnerabilities and misconfigurations can allow an attacker to spread the blast radius of their attack. And of course we're going to be talking about this in the context of Kubernetes, because that's what every talk today almost is. That pattern of exploits to expansion is kind of how most of these attacks come about. An application vulnerability gives an attacker the initial foothold. And then infrastructure level misconfigurations can allow that attacker to spread into other parts of your system. So we're going to walk through an app vulnerability to from going from an app vulnerability to basically owning a cluster. Now this of course is, we only have 45-ish minutes. This isn't going to be a full on huge demonstration. It's going to be fairly contrived on rails so you can kind of follow along. But what I'd like you to see is what are some of the things that if you stand up a Kubernetes cluster, if you don't think about, if you don't customize, you don't configure, how they can bite you and what you can do to head those things off. So, setting the scene. We're going to pretend that we are a hacker and we have found this vulnerable server out on the internet. We don't know much else about it other than that it has this particular vulnerability. For the purpose of this demonstration, this is just a mock flask app that we wrote that has a remote command execution vulnerability. What that means is it's going to allow me to run commands directly on the server where that web app's running. It's a simple flask app, honestly. But these kind of vulnerabilities do exist in the wild. Things sort of, software like Tomcat or gosh, Spring Boot or basically anything in Java if you were running log4j last year. You can craft, a hacker can pass malformed or specifically crafted requests by HTTP or whatever that allows them to run commands on that server. So, while we go through this, I'm going to have this timeline of Doom that's going to show us as we go along from the left where we have found a vulnerable container in this case. And as we go to the right, we'll see the things that lead up to escalated exploit. So, enough of slides. Let me get over to the wonderful hackable app. So, of course, this is very, again, contrived. Understand that this could very well be any RCE you've seen out there if you have seen them. But this is going to be, I've done this so that it's easy to see it from the point of view of the audience out there. So, the way this RCE works is we have found that if we pass a CMD into this web admin context, you can pass a string into it and it's going to run it. It's pretty bad. So, I just ran who am I on, this technically is running over here on this machine. And because I'm not brave enough to test the demo guides that much, I'm running kind on Docker desktop just on that machine. I'm using this machine so that you can see that, A, first of all, because it's an Intel machine and I have some images that aren't cross architecture, this M1 machine has problems with that. But B, so that you can say I don't have Docker running over here, Docker engine. I don't have anything running here what you're going to be looking at. So, it's all remotely connecting over there. So, what I've hit is the front end of that. I just passed a command in. One of the most interesting commands I might do as a hacker is I want to see what I can get out of your environmental variables. And here, is that big enough for you to need me to grow that? Oh, is it cut? Okay. Yeah, let me drag this. Better, better. Cool. So, here's all the environmental variables that are in the environment of that Flask app that you can see. And there's a ton of Kubernetes ones. So, that's, aha, I'm running in a container, I'm on Kubernetes. I can also see other things about it. But one of the interesting ones that you can see from this point of view is this Kubernetes service host. There's another one here, actually, that I usually look for. Kubernetes port's right up top. So, this internal 10.IP address, that is the API server from the point of view of this pod. That is the control plan. If you're not a Kubernetes expert, that is where you send commands to Kubernetes to do things, it's the API endpoint. So, what else can we find out? Let me copy this. And I'd kind of like to know what is the IP address where I am. So, I just did an IP space A and it looks like I'm at 10, 244, 162, 135, inside that cluster. This is the pod's IP address. Interesting to know. What else can I find out? Let me open another one of these and actually I'm gonna copy for my notes over here because it's easier than typing.  I have to find it in my notes. Sorry about that. So, live demos, folks. It's the way you do it. So, I've got that. I'm gonna do, I've figured out what IP I'm at and I'm ignoring my slides because slides are boring. But I'm now going to cat something that is too long for me to type. So, copying. Copying from iPad. That's always fun. Paste it right into here. Come on, Apple. All right, look at that magic. What is that? That is a token. So, what I've done is I've gone to the default place for the service account token for a pod. That is the credential, if you will, that this pod can use to talk to that API server. Now, those of you who are Kubernetes savvy may know that that is in versions 123 or older defaulted to true to auto mount that in every pod that you start. Thankfully, Kubernetes has made that default to false starting in 124. This is a 123 cluster. In fact, if you run on most Kubernetes clusters, you're going to not be on 124. That's why I continue to show it right now. So, hopefully soon enough that, oh, I'm sorry, I'll grow that so you can see it. That's just EKS, just I pulled as an example. If you're starting up an EKS cluster, you're probably still on a 122 or earlier. So, many clusters you're going to see out there are still running an older version because 124, that's bleeding edge crazy. Who would run something that's brand new? Well, if you're not, and you're not explicitly marking auto mount service account token to false in your service account definition, you're going to have this thing available to basically every pod out there. Well, what can I do with that? Well, let me go back to our timeline. What do we know right now? So, we know we can hit this thing from port 80 from the outside. We know from the port information in those environmental variables that it looks like it's a service listening on 5,000. We know the IP address of the pod. And now we know that we have the pod token because it's available to us inside that pod. So now I'm going to take another string, copy it over here, do, do, do, do, do, copy. Come on, copy, going to replace. I'm gonna go back and overwrite this one. So what this is, is you can see in the larger text here at the top, what I've done is I've run curl. Now, curl's in my image, yay, I can use it, fun. So now that I have that, and now that because that's available to me, I can pass in the, into the authorization bearer that token that we found earlier. And I'm assuming the CA cert is right next to it because that's kind of the default place for it as well. And I hit that endpoint IP, and I'm going after the default namespace endpoints. And sure enough, I got a response. So, token's good, API endpoints good, and I can get at the default endpoints. Now why do I care about endpoints? Because it's often open. That's one that is good one to check. But another interesting thing here, if I scroll to the bottom, you see that IP address, that would be the publicly facing IP of the API server. So you see the port 6453 on that IP. Now, in this demo, it's not because I'm running inside of Docker desktop, which wraps things in another, all the layers that you have there with kind and everything. So I'm not gonna actually use that IP here, but if this was a Kube ADM started Kubernetes cluster in your development environment, say, you started up in a sandbox off of your B-Sphere, that very well would be available. Luckily, and I'll throw this in, most managed Kubernetes clusters do not expose it that way. So, managed Kubernetes clusters for the win. However, let's take what we've found here, and I'm kind of tired of using, or actually I should show my slides because we're moving along the timeline of DOOM. So we now know that the internal IP of the API server works and that the Todd token allows access to that endpoints API. So let's go back to what we had here. I'm kind of getting tired of using this RCE website because it's kind of cumbersome. I'd like to actually get at this server from a command line. So I'm gonna copy this whole token. I'm gonna come out here, and just like I said earlier, just to prove a point, I am not connected to a Docker engine at this point, and I do not, I have Kube CTL aliased. If I try to get something, I'm not connected to a Kubernetes cluster at this time. I do have in this directory a little setup script that will create a Kube config from a token, just because I'm too lazy to do one myself, and I'm gonna use the host name I know. This is why I would use that IP address if this was a regular Kubernetes cluster. Are we cutting off? Sorry. Better? Cool. So basically I just created a Kube config file. I'm going to export that into, oops. Ah. Is it, oh, is it off the bottom? Okay. Give me a second. Let's over scan. Come on, grab the corner. Are we there? Okay. Clear that.  There we go. It's gonna export my Kube config, and now if I try to get something, it is going to demo God's Please Work. Hey, that's actually a good sign. I'm getting an error from the server, so I am connecting to the Kubernetes cluster, and that token is actually working. It's just telling me that I'm not allowed. So it tried to pull pods in the default namespace, and it's saying you're forbidden. You can't do that. That's craziness. Why would we let you do that? But it exposed something to me that's interesting. If you look at this error, let me grow that font some. It says that user system service account secure web admin can't do this. The namespace secure is where that service, what that pod is running in. So let's try this. Get pod dash n slash secure. Don't confuse the word secure with meaning secure here. This is just the name of the namespace. Naming your namespace secure does not make it secure. So I was able to do a get pod on the secure namespace, and I got one. Let's see what else I can do. Let's do an auth. Can I, oops, can I dash dash list? We're gonna pass the token in. I think I still have it in my pitnip. Let me go copy that again. Copy, paste. So I'm gonna have to shrink that font just a little to make it fit better. So if you're familiar with can I, it's basically asking the API server what can I do? What can I using that token do? And that user in the default namespace, because I didn't provide one, is able to, let me shrink it a little more because it's so ugly. Endpoints, as we saw earlier, can do any of the verbs in the default namespace. But really there's very little I can do in the other resources with this. But if I use the same command and I pass in secure on it, namespace secure on it, you can see the very first resource is the wildcard with create, get, watch, list, patch, delete, delete, collection, and update. I can do a whole bunch in this namespace with that service account using that token. So that's handy. Let's see if, skipping through some notes here, we can, well let's go look at our, let's see what we know about this now. So we now know that there is a secure namespace and there's always a default namespace, so we have some namespace data we know about. And let's see if I can use this okay, get pod, and secure, so I can see the name of it again. Let's do an exec, IT in the secure namespace and I don't have my autocomplete configured, so just copy and paste and I want to run bash. Yay, I can do that. By default, I am the web admin user. That's good, at least I'm not rude. That's somebody who's listening to one of my talks. Let's see if I can sudo. No, sudo's not available, that's good. That's good, they at least didn't have that in there. Let's see where am I, I'm in this user src, let's see if I can make a file. I can, so I am in a read-write file system. Why does that matter? Well, let's go back to our little slides. Oh, I skipped one here, I was talking about the fact that our role, we have too many permissions, honestly, all of those permissions for this pod, this pod probably does not need all these permissions, especially if this is a business application, it probably doesn't need any of them. You don't need to be talking to the API controller if you're an e-commerce app, most likely. You probably don't even know where you're running in Kubernetes, so limit your permissions as appropriate. The read-write file system piece, though. What that means, the read-write root file system, when a Docker or a container D or a prior, whatever starts up a container, it layers a read-write file system at the top, on top of the read-only image. That's where all your mutation happens. So if you make a file, you create a file, you edit a file, it does a copy on write, brings the thing up there and makes the mutations at that read-write layer. If you pass Docker, for instance, dash dash read-only, or in Kubernetes, if you use a security context and set read-only file system to true, it just doesn't put that read-write layer there. This is not a silver bullet, but it does make it harder on a hacker to hide their tracks, customize your app configurations, delete logs, do things to modify that container. If I happen to be rooting that container and it's a read-write file system, I can very likely run app get inside that container and start installing other tools in the space of the container. Now you may have detection for that kind of a thing with intrusion detection systems or whatever, but it's better to head it off than react to it when it happens, in my opinion. So that being said, let's see if we can get our privileges extended. So this is just showing, we know we have a PS, oh, that's actually, I'm skipping ahead. Don't look at that slide. Let's see if we can do more. So I have a few manifests here. I'd like to try to apply. Let's get out of the exec and I'll show you one of them. We're going to do demo YAMLs, root pod YAMLs. This is a simple alpine image I wanna start up and I know that by default that alpine image runs as root. So I wanna see if I can get an image in there that'll get me root in a container on this cluster. So let's do apply F into the secure namespace. Oops, put the file after the F, Eric. Demo YAML, root pod. And it says it created it. Let's do a get pods on the secure namespace to see what's going on. Create container config error, hmm. Well, let's do a describe. Let's see what's going on here. Describe pod secure root pod. Error, container has run as non-root and image will run as root. So the cooblet, when it tried to start this container up, detected that the default user is UID0 and said, oh no, we don't allow that. Now, why did it say that? Well, we probably have a pod security policy or something in place to restrict that. And before anyone says pod security policies are going away, I know that, we'll talk about that in a minute. But you can see in the annotations, sure enough, we see a PSP resource called restricted has been annotated onto this pod. So we have a PSP and that was the giveaway here because we have a PSP in place. But let's see if, so I'm actually gonna kill, delete that root pod so we don't have it sitting around trying to start. I've got another one, of course. We'll look at demo yamls. Which one am I doing now? We're gonna do the non-root, prove. So this, a little bit bigger. This is, we have an image called sneaky that we're going to try to deploy that actually wants to start in the privileged context. I wanna be a privileged container. And I wanna do some nasty things with mounting of volumes here. So let's try applying that non-root, prove. Immediately got an error. This is a pod security policy. So we had good evidence, we had one before. This one tells you right here. Pod security, unable to admit pod privileged containers are not allowed. And that's by default, that's normal. That is, most places would restrict that. But let's try something else. I'm going to run another one. Demo URL or yamls, non-root, non-proof. This is the same thing without the privilege or the volume mounts. So same image. So we'll go ahead and apply into secure. It says it created it and it's running. So we have that running. Now, I am going to, I'm just gonna go ahead and exact into this. This is in secure, it's called sneaky. So there we are. We are inside the sneaky pod. And I'm going to, you know, say who am I? I am the user sneaky because again, non-root is set where can't run as root. But this is my image. I do have pseudo in there and now I'm root. So why was I allowed to do that? If privilege is disallowed, why can I do that? Well, that is because this isn't really actually that clear in the Kubernetes documentation surprisingly. If you set your security context to privileged false that does not make allow privileged escalation redundant. That is not defaulted to false. It allows it as you can see. So what we see here is we have a pod security policy that A has too many permissions. You shouldn't be allowed to be doing a lot of this stuff. And especially the fact that it doesn't have the restriction on allow privileged escalation. So that's the next step in our timeline of doom. So now we're root on a container in your cluster. Now we're in a container. So what can we do in a container? Let's say, go back over here. I would like to know what's the IP address I'm at right now.<PHONE_NUMBER>1. If we go back to our prior, just show that open. Now let me rerun IPA. So this is<PHONE_NUMBER>5. This is 144. So I know I'm in the same subnet. So I kind of get my bearings that I'm in the right place. But now I'd like to poke around. I want to see what's going on in here. So I'm gonna copy another command over here because I can't type this long one. Copy, clear my screen. Oops, that is not what I wanted to copy. Sorry guys. Copy, Apple. Okay, we're gonna start typing. We're gonna run Nmap. If you're not familiar with Nmap, this is a tool for scanning networks. And we know that our service is listening on 5,000. I'd like to know if there's any other copies of this app out there. Dash, dash, open in the 10 dots. What was that? I have to go back and look at my IP address. <IP_ADDRESS>.162.0 slash 24. So we're poking around. Because I'm root, in a container, I can do this. That's one of the things you need elevated privileges to be able to do. And of course, this is gonna sit here and take forever. Come on. Oh, it did not find one. Oh, that's not good. That's not what I wanted in my demo. So I need, I don't have to provide the full. Hold on a second. Do, do, do, do, do, do. What did I do wrong? Sorry, maybe I have to give it my IP. That would be weird. 162.14. I'm also not a network engineer. So what I am expecting this to return to me is that we find another one. We find another IP address listening on port 5000. And because we've been looking around in this network, we know that only one pod is running in the secure namespace, that means it's somewhere else. And why is this not working? I'm sorry? There are, right, but one of them is the sneaky pod. Right, but what I'm looking for, let me copy this whole IP. Maybe I'm just doing this wrong. I've slept since I've gotten this far in my demo. What I'm looking for is another of the exploited, or exploitable applications running in another namespace, is what I'm hoping to find. And normally, oh sure enough, there it is. Oh gosh, see, I'm blind. You get blind on stage, folks. Sorry about that. So yes, so 135 is the original one, and that's the one we see here. Thank you. 136, I'm gonna bring you up here and hand over the keyboard. 136 is something else. Somebody else is running at 136, and I'd like to find out who that is and what's going on there. So in order to do that, in this exact, I am gonna run Socat. And I'm gonna, Socat, socket cat, it's like a tunneling tool, if you will. It's more than that, but that's what I'm gonna use it for. We're gonna do a Socat, and we're going to get lost in my own notes. I scrolled it off my screen. We wanna do a TCP listen. We're gonna listen on 5001, which is an open port. Use ADDR, fork, and we wanna send traffic to <IP_ADDRESS> on port 5000. So we've got Socat listener going right now, right there. And now what we need to do on our local console, I'm gonna open another tab and get out of Fish, because Fish doesn't like me today. Am I in the right directory?  So we wanna do a K port forward into the sneaky, oh wait, hold on, export, there we go. K port forward into sneaky pod at port 5001 on the secure namespace. So if you're not familiar with that command, Kube CTL has now reached out to the sneaky pod, because we have that token, we can get to it, then we're the same token that we exacted with, and we have opened a tunnel up from my local host where Kube CTL is running on 5001 through traffic hitting local host at 5001, we'll hit sneaky pod at 5001, where we in turn have a Socat tunnel listening at 5001, pointing at whatever that thing is on 5000. So I'm gonna go back to my browser, open a new tab, we'll hit local host 5001. Same app, surprise, surprise. Some developer apparently is running another copy of the app elsewhere in the cluster. So we're gonna do the same kind of thing here, and I'm going to, actually I just copied from over here, run the same, let's copy this whole thing, front end of that, there's the token from wherever that's running. So let's go back now to our command line, and open another tab, actually I don't need an open another tab, I'll just, I don't need my port forward after that. Let's get out of there, and we're just gonna edit this Kube config file. Oops, where am I? BI, demo, Kube config, oh it's just not visible, there we go, and I'm gonna comment that out, paste that in, is that indented right? I think so, and now I'm able to get pods in default, because that guy is running in default, because this is a development cluster maybe, this is some place where he has access, or she has access to deploy into the default namespace, that I think you can see where we're going. So, what have we found? No network controls are in place. So end map, besides the fact that I was able to get in as root, and run end map, I could have installed it if I needed to probably. I obviously don't have any network policies in place, or other firewalling that's keeping me from perusing around outside of my namespace, or places I shouldn't go, and this technically could be somewhere else, but it's nice that it was in the default namespace for me. We found a pod listening on 5,000 somewhere, and then determined that it's in the default namespace by going and getting its token using that now, and we were able to access pods in the default namespace. So now, I'd like to see if I can get a privileged pod into the default namespace. So we'll come back to the command line, and let's go back over here, and I'm just gonna go ahead and get out, we don't need that SoCAD anymore, or anything, so I'm gonna get out of there, and I'm going to do a K apply into the demo. K apply into the demo, yamls, why, oh, I'm not out of the pod, sorry. K apply into the demo, yamls, and we're gonna go back to that non-root privileged manifest I showed you earlier, but we're gonna deploy it to default, and sure enough, that worked. That's because the namespace wasn't restricted by the pod security policy that was in place, and this is more common than you may think, because developers writing an application for their namespace often will craft their pod security policy or other restrictions for that namespace, deploy it to that namespace, and if your cluster ops aren't restricting defaults, default won't be restricted, and sure enough, somebody decided to start the app in default, and I was able to get in there through all the things we've said, and now I've got a pod running in default, so let's go ahead and exec into that, and do-do-do-do, go into non-root-proof, oops, not boss, bash, there we go, and I am able to become root, and if you remember from that manifest, we were mounting the root file system from the host, and we put it into, well, there's more in there, I thought it was, there's a lot of stuff in there. We have a volume that's slash charoot that it's mounted to, handling name, because if I do a, let's do this, let me do a PSX to kind of make this more dynamic. If I do a PS right now, I'm in the process name space for this container, so all I'm seeing of the process is running in this container. That goTTY, there's another piece of the demo I could be showing, or I could get a web TTY, but that was too fancy for me, but if I now do a charoot, which changes the root, change root, of the current shell to whatever I'm going to tell it, I'm gonna change it to the volume named charoot, or the directory, and if I do that same PSX again, there's all the PIDs on the host, because the way PS works is it looks at the proc file system for that kind of stuff, and the proc file system in this container, effectively for this shell, is the one that's on the host. I now have basically host access on this worker node. So that's bad enough, but what can we do more? So let's come back and see what we've got. So we have no restriction to the default name space in the PSP, that's obviously bad. We now, we wanna get at, what do we wanna get at? We only get at CD, right? If you're in a Kubernetes cluster, you wanna get to CD because that's the keeper of all things, and that's over on the kubesystem namespace, and we're not there yet, but we do have a privileged container over here in the default namespace. So let's see what we can do to continue to expand this exploit. So right now I am going to, so what I wanna do is from this pod, I wanna try to connect to kubesystem. I wanna get at something in kubesystem. In order to do that, I know my sneaky pod has kubectl installed in and I have the tool, but I need a key that can get me a token that can get me over there, and the default token's not gonna do it, but what is on every node that might have it, especially if you have root access to the host volume, I'm not gonna CD to it, but I'm going to export kubectl config to etsy, kubernetes, kubelet.conf. That's the default place kubedman, or most people will install the token that kubelet uses to deploy to do its things, to talk to the API server. So now if I do a kubectl, get pods in the kubesystem namespace, I can see them. Ooh, fact kubectl, get nodes, yay, I can do that. So, I already showed you the slide that we now have, well, I've showed you the privilege, I don't wanna show that again. I need to see something here though, let's do kubectl describe pod in the kubesystem namespace I wanna take a look at that etsy kind control plane. Is that right? Yes, that's right. What I'm looking for here is where on the control plane host, PKI, the tokens for etsy dr, and etsy d certs volume is the standard place, but I know where it is, and also should show me it's on node kind control plane. So that's interesting.  One moment, oh yeah, sorry. Okay, so if I try then back in my sneaky pod, if I try to do a kubectl run, I'm gonna try to start up a pod, and just try to start a busy box. Midge equal busy box, restart equal never, dash dash shell. Forbidden, I knew that was gonna happen, that's the first thing I would try, but I can't do that because the kubelet, ironically enough, cannot start pods, it can start containers, but it's actually restricted by default from creating them. Now, I could go edit etsy kubectl runes manifest and add a static pod to that, a shadow pod, mirror pod, sorry, and start one that way, that's no fun, I don't wanna mess with that, so what I'm gonna do is let's do this. I'm going to, I've got another, obviously, I have tons of YAML, I'm going to run another one that is an etsyd client, and you can see that this is going to start the Kubernetes etsyd image, it's going to set a bunch of things, now I've already filled this in with the info we saw there, it's going to try to connect to the etsyd endpoint and get access to things, so let's do 10 minutes, oh, I am hitting my time, okay, kexec, not exec, sorry, kapply, see now you got me all flustered, you're showing me the time, fdemo URLs, YAMLs, etsyd clients, so we're gonna start up this etsyd client in the defaults and we're gonna get in there with etsyd clients, thank you, this is why paired programming works, and I'm just gonna type this because I know where it is, local bin etsyd, cli, member list, oops, maybe I don't know where it is, exec etsyd, what did I type wrong? user local bin etsyd, CTL, okay, so we did get a good connection, so we know that we actually, even though it didn't like this, it gave this warning, we did get a connection, so now I'm going to modify that to, instead of doing a member list, I'm going to get dash dash keys only from key grep, what do we want, we want secrets of course, so we always want out of etsyd, and there's the list of the names of all the secrets, yeah, well the one I care about is, let's see, where is it? Cluster role aggregation controller token, what can I do with that? Let's copy this line, copy, edit this, edit this, I'm going to do an etsyd get on that, right, right, right, there's a token, let's see what I can get with this, and grab from there to there, and we'll do a k off can I dash dash list, I'm running out of time, so I'm just going to hit the, get to the end of this quick, you must be logged into the server, hmm. You know what's probably happened is something has been upgraded behind my back, and it's no longer vulnerable, hmm, come on demo, do your thing, there it is, it just didn't like the annotation at the end, so what we can see here is cluster roles, are back authorization kates.io, escalate, get list patch update watch, and that Greek means you are God on this cluster with that token, so if we go back to our notes, we were able to get over there, cluster rights gains, and that is the end of your server as far as you know, so how could we have prevented this, I talked all throughout some of the things you could do to this, but the first thing is scan your application code, don't let RCEs get into your, do your best to not let RCEs get in there, and honestly that contrived Flask app, if I throw that through a Python static analysis scanner, I know ours catches it, but they should catch that and sure enough ours does, it'll say high vulnerability here, you're running a command from unparsed input, don't do that, scan your container images for the same reason, scan your Kubernetes YAML for best practice, there's tons of scanners out there, use them, don't trust your default, speak explicit about things, specify your pod security policies, or admission control, whatever, to not allow permission, a privilege escalation, use network policies, they are your friend, I know as a developer when a lot of us think network firewall rules, it seems complicated, I have to open tickets for that, network policies is really not that complicated, doing it right takes work, but you wanna make sure you're using that tool that's available to you, and use admission control, so I mentioned PSP as being deprecated, it's gonna be removed in the next release, it's been deprecated since 121, I believe, and pod security admission is coming along, but your admission controllers that are popular out there today, your OPA gatekeeper, your Kyvurno, you can do a lot of the same things with those tools, and they're very popular, there's a ton of support out there for it, so use some kind of admission control to enforce what you would normally do with PSP, as well as a lot of these other things, don't let people deploy to the default namespace, just don't. Finally, I just wanna give thanks to a bunch of people, I've only listed a few here, but everybody in the SIG Security, Mark Manning, Ian Coldware, Duffy Cooley, Chris Nova, all these people out there, a lot of what I've learned here, what we've presented to you, I've learned from listening to these people. So SIG Security, Tag Security, OpenSSF, all these groups are there to help us learn security and implement it well, so I just wanna thank them all and tell you to join our SIGs, hang out with us, and now, with the time I have left, I'm open for any questions you might throw at me, I'm gonna take a drink. Yeah, awesome, please raise your hand if you have any questions, and I will come over with the microphone and hold it for you to avoid contact. All right, let's start with you over here. Thanks for the great talk, that was awesome.  Without pitching for anybody or any single project, there's a lot of cube scanners to scan your clusters to tell you whether or not your cluster itself is meeting CIS standards or other standards, cube benches, one of them. In fact, if you're doing the CK, take the CKS, get your CKS cert, you will learn a lot about these kinds of things, but there's a lot of tools out there that to implement these from many vendors, particularly sneak actually, we don't do runtime stuff, so I'm not, I'm gonna pitch us, I'm gonna say look at the open source projects that are out there. If you are on a managed Kubernetes from a distribution, often they will bake these things in, because that's why you're paying them. So, EKS, GKS, many of these things that I did aren't as easy to do there because they've already set some, you would have to undo some of the things they're doing to attack them, so I would say pay attention to the vendor you're using for your Kubernetes, don't be undoing things, so if for instance, the service account token auto mount that is going live in 124, when 124 gets rolled out, if your apps all break because they're trying to talk to the API cluster, question why talk to your developers, why do you need to talk, why do you even know you're on Kubernetes?  Okay, let's talk, are you doing a business application? WTF, why are you even aware that you're in a container, so that kind of stuff, does that help? Anyone else? Well cool, we'll all be around afterwards, we get a break after this, so I'll be hanging out, if you have any questions, like I said, join us in security, tag security, open SSF, supply chain security, all those things I hang out. Yeah, I promise to Kubernetes community is super friendly, super welcoming, thank you so much Eric. We will be back here in half an hour after the afternoon break, I believe that is 3.30 for our next talk, and I just, I'm going to peak, oh yeah, exploring memory usage and IO performance, so hope to see you back, please enjoy the rest of the conference, I know that we have a tradition of chatting after a talk, so please feel free to come up and hang out. Thanks Eric. Testing mic is off. It was the best meetup that I have ever been to, why? Two reasons, f