. Now the interesting thing here is that these expressions are actually just go templates. So if we come to the helm documentation, we can take a look at the various functions that we have and, you know, start to leverage those. But they are just go templates. So we could also just, I don't know, flip over to the go helm, the go template documentation and start to understand how we might use interesting things. Now we definitely could go really intense here and build up this whole logic tree. But in this case, let's just go simple. But I do like this if and else thing. So here in our ingress, if we don't have an ingress, let's say something like if values dot ingress enabled. Now one other trick here is I'm going to put the dash here. Now it's mad because it wants the dash here as well. But I only want a dash here. The dash says space trim. So because I've chosen a space trim here, if ingress is enabled, I'll not start this on the next line. I won't have a blank line. I'll just push this straight up here. And then here, let's do and and I'm going to space trim there too. Cool. Okay. Yeah, I'm going to put the space trim everywhere because that makes my editor a little happier. Fine. Cool. So a little bit of logic here of how we can start to glue these things together in interesting ways. Now we've got this if here. Now let's take on these annotations. Now here in the annotations, I do want to dig into the annotations. But the annotations are well, a bunch of yaml. So I can't just go grab the value. I may have more than one, especially in this resource section, I have some content embedded in it that I want to capture. So instead here, I'm going to say let's say space trim to yaml dot values dot ingress annotations. Annotations. Did I spell that wrong? As long as I spell it wrong consistently. Okay. So I've pulled in my annotations. And now I need to make sure I indent it for spaces. Now I'm indenting it for spaces because well, that's how this is indented. It's indented for spaces. So it's going to grab that whole block of yaml, set it in place right here and move it in four spaces. Cool. So we've got that one in place. Let's also come into the deployment here and we'll grab these resources. And in this case, I need to not delete it values dot resources. And I'm going to indent it 10 spaces. Nice. So we could continue adding more logic depending on the needs of our system. But let's go lint it again. And Oh, nice. No typos so far. I'm going to pull in one more file. I'm going to pull in this um, read notes dot txt just to save us some typing. This notes dot txt just kind of shows some interesting information about our app as we deploy it. Thanks for installing chart.name. Here's how you use it. Here's some more stuff about it. That's kind of cool. So now we've got our thing. I'm going to lint it one more time and everything looks good. Cool. So now let's grab the content of this template and see if we can run it in interesting ways. Now in this case, we only templated a deployment, a service and an ingress, but we could also template persistent volumes and persistent volume claims, config maps, cron jobs, or the yaml that you're sticking inside of your application. It doesn't need to be a Kubernetes thing. It only needs to be a yaml thing. So if you have a particularly gnarly config file that you're waiting to, that you have to set this into it and set that into it, well, Helm might be the right way to take more control over that yaml file. Focus in on the values that you are just looking for. So let's say helm template, and I'll give it the folder name which is named chart and out pops the yaml. Cool. So we've got the 80 in place. We've got the at the helm in place. We've got some stuff here. It looks like we're doing pretty good. Let's do this helm template chart. And I'm going to redirect it to dist.yaml. So we've dumped it into this file here. And so if we take a look at this file, we can see dist.yaml. And I'm going to grab kds.yaml, and I will compare them. So we've got now kds.yaml here on the left. That's the file that we started with. And dist.yaml here on the right. That's the file that we just built. Now we do have some comments specifying what file it came from. That's cool. We also have the quotes that we chose to put around the data here, which is pretty cool. And other than that, it looks like we have achieved the file that we started with. So no bugs so far. Nice. Oh, it looks like I can upgrade this app too. So now that we're able to template this out, we were able to focus only on those 13 values that we cared about, and we could build up our yaml. So now let's start to set variables. Helm template chart. I might just choose to set image label equals v0.1.0, and ingress enabled equals false. Ooh, I didn't get an ingress this time. And instead of spitting out image label in all the places, I was able to spit out the version that I set for it. So let's imagine a use case where we just finished the build. We want to grab some variables associated with the build context and just dump it into our Kubernetes cluster right away. So I can do a similar thing. Let's say Helm template chart. I'm going to set image label equal v<IP_ADDRESS>.1.0, and ingress host equals quote quote, and registry equals quote quote. I'm going to just pipe that to kubectl apply dash f dash, and we'll be able to blow that straight into our cluster. Now that's cool. We were able to capture the build context and focus in on just those variables that we cared about. kubectl get all, and we can see everything is running except for, ooh, we've got some problems here. Let's take a look at that. kubectl describe this pod, successfully assigned the thing failed to, ooh, what is the name of my image? My name of the image is slash at the helm. That isn't quite right. Let's pop open our template again and take a look. Okay, so we have our registry slash the chart name. That looks good. If we have a registry, if we don't have a registry, that's probably not okay, because well, we don't want to do this slash. Okay, so let's do this. If dot values dot registry, space trim, then do the registry and do that slash, space trim, end, space trim. Cool. So now that we've been able to conditionally put in that slash, let's try this again. Let's template this chart, pipe it to kubectl apply. The service is unchanged. The ingress is configured. And the deployment is configured. kubectl get all and ingress. There is no, oh, I did create an ingress. I just created it with no host. Nice. And now my app is running. Cool. We just took the details from the build and we just stuffed it into our Kubernetes cluster. If that's the need that you have for this templating system, then you can definitely stop here. This is a great use case for helm. I just focused in on those values that I cared about and I blew those into my Kubernetes cluster. And just as easily as we created them, let's go delete them and kubectl get all and ingress. And we've got nothing running. Cool. Well, that was a lot of fun. Let's level up a little bit and talk about how we might package our helm charts into something that we can then install into multiple environments. Now, again, we don't need to go here. If your need is just to blow the results of a build into a cluster, then you may not need to go here at all. But let's do this. Let's say helm package chart. Cool. So I built this thing, this at the helm 0.1.0, this TGZ file. And if we pop open this file, here's that file. Let's seven zip. Let's unzip on G zip it to there. Let's untar this to there. And let's take a look at what we've got in this folder. Oh, it's our helm chart. In fact, if we diff this folder, let's compare this folder with our chart folder right here. It basically just packaged up all the things. Now I'm choosing not to show identical files. We can see that all the black ones are identical. So let's just focus in on the differences. And it just removed white space and sorted things. That's fine. Cool. So we're able to package that. And that's pretty cool. Let's come delete this one and delete this one. And let's package that with a little bit more details. Helm package chart. I'm going to set the app version to <IP_ADDRESS> and the version of my software to 0.1.0 and the destination folder to the disk folder. Cool. So now I have this at the helm inside the disk folder version 0.1.0. Okay, so let's install it. Helm. Oh, so that was cool. I got to set my app version and my version. What if I want to deploy, what if I want to add those variables associated with my build context? So I wanted to override that image label. So let's do that exact same thing. Except for now I'm going to say dash dash set. And I'm going to set image label to be 0.1.0 and registry to quote quote and ingress host equal quote quote.  Oh, unknown flag set. It turns out that the package command doesn't have the set flag into it. But there is a really cool helm plugin called helm pack, which has exactly the same syntax as the package command except package plugin. And so because I've got that helm package or helm pack plugin rather, I can come in here and I can say helm pack, not that helm pack. And now I've got that at the helm 0.1.0 that includes these variables set. Now I'm not going to untar this tar ball, but we could just see that inside values.yaml, it just has those things set the way I needed it to. So now that we've got this file here inside our disk folder, this at the helm, let's install that chart. So let's say helm install my app, I'll just give it a name, and I'll point it at where my helm chart tar ball is. And now I can choose to set some environment specific variables. So let's say ingress enabled is false and replicas is two. Cool. So that was awesome. And as I packaged my chart, I got to set some build variables. Maybe I'm setting the git hash or the branch name, you know, other details associated with the build context. And then when I installed it, I got to set the install context. Maybe I'm setting environment variables or the environment name. And now that I've got my content in place, we get that read me that kind of explains nice things. Oh, helm status, helm status, my app, which by the way gets us that page that we just shot. So helm get all my app, and it spits out the Kubernetes YAML that was actually applied to the cluster. That's pretty cool. Or kubectl got all. And we can see the resources that got deployed. Now we chose to deploy two replicas. We chose not to deploy an ingress. So we have no ingress running. And if we now go to this particular port, we can see the application running.  And now we can see our app running. Cool. So if I now do this helm list, we can see the helm charts that we have installed. Now the helm chart is just that package of stuff. And if we had just done a cube seed of helm template, pipe it to kubectl apply, we wouldn't get this helm list. But we can kind of see, here's the chart tar ball that we use. Here's the version, which is just an auto incrementing thing. So we've installed it once. And we haven't upgraded it. Here's the app version that we had installed here. And that's pretty cool. Now let's imagine that we needed to upgrade this. Maybe there was a bug in our chart. And so we wanted to upgrade our app. So let's say helm upgrade my app. That's my install name. And I'm going to point it at my tar ball, which in this case is the exact same tar ball that I used previously. And it will now upgrade it, altering whatever resources it needs to be able to get to the current state. And so I can now helm list and we are on version two. Now in this case, I didn't change anything. So it didn't replace any Kubernetes resources. But it did nicely show how we might upgrade an application. So that's pretty cool. We've got the helm chart built. We're able to apply that into our cluster. But I noticed something that kind of bothers me here. Cube, CTL, get all. What if I want to install two copies of my application? Now right now I'm using at the helm right here. And so if I were to install it a second time, it would tell me that that name is already in use and tell me that I can't install this second version. Hmm. Well, that's a bummer. Let's come back to our chart and see if we can fix that. Now in all of the places here, we're saying the name of the thing is the chart.name. But here in notes, we have the release name. That's kind of interesting. That was the name that the user chose as they chose to install our helm chart. So let's use that here in our resources. Let's rename our service to chart name dash release name. Now in our selector, we're also going to say here's our release. We don't need to change the port or the target port. And so now we've got our service configured. Similarly, let's come to our deployment. Our deployment is named chart.name-release.name. We don't need to change the replicas. Let's change the selector to release. We'll also change our labels here, release, there. We don't need to change our image name because that is the image name baked into the pod. So if we wanted to, we could, but I'll just leave it alone. We don't need to change our container port or our resources. Let's flip over to the ingress. Let's change the name of our ingress. And we also need to change the name of our service that we're pointing to. Now our service name is not just chart.name, but rather chart.name-release.name. Cool. Now we have a release specific deployment. So in this case, let's do a Helm Lint chart. Let's go validate that we didn't make any typos. That looks good. Excellent. Helm chart. Our app version didn't change. It's still 0.1.0, but our chart did change. So this is version 0.1.1. Our chart incremented versions, our application did not. That's pretty cool. The destination will still be the dist folder. We'll set some values like image label equals v0.1.0 and our registry is quote, quote. Now I'm noticing that I'm setting it to blank. If I'm going to do this all the time, I probably ought to pop open my values.yaml and set the registry here to quote, quote.  That'll let me know that if I haven't set it correctly then, but maybe in my organization that's a business rule that it's always the same. So maybe I don't need it to be one of the values that I care about here. I can just leave it in my yaml. Okay. So I've set my registry to nothing and I'm going to set my ingress host to blank as well. And now we've packaged this into 0.1.1, a new target. Cool. So let's install this. Helm install my app to, we'll grab the dist slash at the helm. We'll make sure that we grab 0.1.1. We'll set ingress enabled equals false and replicas equals one. And that looks good. So now we have two versions of our application running helm list. We have my app that is app version 0.1.0, my app 2 that is also 0.1.0, but we can see that they're using different tar balls. The first one is using 0.1.0 and the second one is 0.1.1, which at this level of wrapping is really hard to see. So let's cube CTL get all and take a look at the resources that we've got. Cool. We have a deployment and a deployment dash release name. That's pretty cool. We have a service and a service dash release name. We have some pods and some pods dash release name. So let's upgrade our original one. Helm upgrade my app to dist at the helm.1, cube CTL get all. And now we can see that because we've upgraded our version one as well, now we also have that release specific values in our cluster as well. That was kind of cool. Now we got to focus in on only those values that we cared about. We didn't need to worry about all the other YAML pieces. We got to choose those 13 values that were important to us. And if our business needs are different, then we can adjust that and maybe only focus in on 5 or 12 or maybe we need to focus in on 25. And just as easily as we installed them, helm uninstall my app, helm uninstall my app too, cube CTL get all and all of that content is gone. Did you grab the right YAML file? Do I need to set the values on the way through to get it to delete correctly? All of that just kind of melts away and helm was able to take care of that for us. Cool. So we got to just run a template straight into Kubernetes. We got to package that in an interesting way and leverage that package to reinstall our app in the cluster based on our environment. What if we want to share these packages with others or even consume packages created by others? Let's come to the artifact hub. Now, artifact hub is the registry of helm charts. Now, similar to Docker hub where we're looking at particular Docker images, here we're looking at particular helm charts. So let's pick one. I'm going to say engine X and let's search for an engine X helm chart. Now, here's a good engine X helm chart and it tells me about it. Oh, here's the install instruction. It's nice. Helm repo add. That's kind of cool. Artifact hub is not a registry of helm charts. Artifact hub is a registry of helm chart registries. Now, the difference is subtle, but it's really cool. You know how Docker hub is really struggling with you're consuming all our bandwidth. Will you really quit it? Artifact hub doesn't have that concern because all it says is the server's over there. So let's do that. Let's helm repo add bitnami and we'll give it the URL to bitnami. Yep, now we've got that repo in place. And our next step is to then grab this and install it. Now, what Docker image do I need? What configuration values do I need to set? I don't know. I don't care. I'm just focused on trying out this product and it gives me some cool instructions about how I might get this. I can get the service port and the service IP and then, you know, echo that nicely. I'm just going to go the hard way and look at that particular port. That's cool. My app is running. So now let me come over here to local host and that port number and we can see that we've got engine X up. What's the settings? What are the details? I don't care. I just grabbed this helm chart and I quickly got this application running. That's cool. Now, if we dig into this, we can see that there are some configuration values that we could choose to set. We could say dash dash set or we could set these inside of a values.yaml file and we could choose to apply those as we start this. And these are all the things that the author of this helm chart deemed important. So I don't know. That's kind of a lot. Not all of those are important to me but some of them are and some of them are important to you and those are probably different. So it's kind of nice that they gave us a lot of configuration options so that we could really configure this any way we needed to. But I also assumed that some of the assumptions baked into this chart, some of those opinions, may not apply to me. So maybe this helm chart isn't the ideal way to deploy this software if my opinions differ from the opinions of the helm chart author. That's easy enough. Let me go grab the source of their helm chart and pick different values.yaml, adjusting the template to match. I could definitely do that if I wanted to. Or sometimes I just want to understand what docker image they used and I'll just start up the docker image myself. So we were able to spin up this helm chart, helm list. There's our thing, my release. That's what I happen to call it as I copied and pasted it. So helm uninstall my release and poof, it's gone. No registry rot, no DLL hell, no all of the things that we might associate with trying out software. I just quickly grabbed the helm chart, I spun it up, it did the thing, I got to play with it and now I understand more about the capabilities of the software. I might now choose to go mess with those configuration values to get it tuned to exactly what I need or maybe I discovered this isn't the piece for me and let me go find the next software. That's cool. Now we could definitely choose to publish this helm chart to artifact hub or we could choose to publish this helm chart to an internal helm registry. A lot of docker registries, especially cloud-based registries now also support publishing helm charts so we could choose to put in helm details there too. Cool. So that was a lot of fun. We got to see how we built up a helm chart template, just putting in the values that we cared about so that we could focus in on just those 13 things that were important to us. At the end of our build, maybe we just want to take the build context and blow that into place and that is an excellent use of helm. We also then upgraded to be able to package this into a tar ball and we saw how we could then set the build context variables as we built that tar ball and also set environment context as we push that tar ball into place in each environment and then we upgraded again where we started talking about sharing helm charts, pushing them into a helm chart registry much like artifact hub. That was fun. The code for all of these is up on GitHub and so that chart that we just built is right here and you can see there's those 13 values that we cared about. So it was cool. We chose to scaffold a deployment and a service and ingress. You may also choose to scaffold PVs and PVCs, cluster roles and cluster role bindings, config maps and secrets or you may choose to scaffold a particularly gnarly YAML file in your application. Focus in on just those pieces that you care about and let helm melt the rest of that YAML away. Helm is great for being able to install things. We saw how we could install them and upgrade them, but it can't monitor, scale or heal these things. Now we could definitely bake those pieces into our YAML. We could get it to talk about how we might auto scale things, how we might self heal things, but helm itself doesn't really give us any of those details. Helm was great for installation. We got a little bit in upgrades and helm definitely helped us there. But if we want to take more control over this process, that's when we reach for Kubernetes operators. Now I totally get that this kind of mirrors AngularJS. As soon as you get to custom directive, the learning curve just goes straight up. Operators are much more difficult than the helm charts that we built. But if you want to take more control over that, if you want to get more into the lifecycle insights or even autopilot where you can get this operator to start to control the pieces, then that's when you'll reach not for helm, but for the operator SDK. But for just taking care of installation and upgrades, helm worked out really well and it's a really simple tool to get started with. Helm was great for removing all that duplication. We saw even in service.yaml how we could focus in on just those two values that we cared about, the name and the port. All the rest of that YAML just kind of melted away. It was cool. We used the three different pieces where we could just template it and blow that into place or where we could package it and start to install this package in interesting places. And I'm really excited that we got to set some build-specific variables on the way through, but also set environment-specific variables as we installed this tarball. So let's get the build version, maybe the git hash, the branch name, and bake that into the tarball. And then let's also set the environment name, any environment secrets that we need to to be able to get that installed configured for that last mile. And then if we wanted to level up again, we could start talking about sharing our packages with others. Artifact Hub is a great example, but you can also get private helm registries for each of the clouds. That was a lot of fun to be able to show you helm from start to end. If you're watching this on demand, hit me up on Twitter at rob underscore rich. If you find a question tomorrow or next week, hit me up on Twitter as well. But for those of us here live at the event, what are your thoughts? What are your questions? What should I do next time to expand this talk to better fit your needs? Does anyone have questions? Sweet. So yeah, the first question is how does it manage dependencies? Would it automatically pull the dependencies it needs for like let's say engine X or something like that? Good question. How does it manage dependencies? The short version is it doesn't. But the medium version is as we look here in this helm, we had here in chart.yaml the application type application or library. We could create a helm library and have that pull into helm charts as we build them. But that's still not quite dependency resolution. I imagine what you're looking for is when you install this helm chart, I also want that helm chart run to and helm kind of doesn't really do that. On the other hand, you could create a super helm chart that included both of those. Probably not the level of separation you're looking for. Great question. Thank you. Any other questions? Cool. That's awesome that the first question came from here and the second question came from there.  Oh yeah. I was wondering if you had any thoughts on helm file and whether that would be an extension to your talk. Good question. Helm file. A docker file and docker compose helm chart and helm file. Helm file is kind of a yaml that describes and all of these helm charts together. It is really elegant. That is a great idea to extend the talk to describe helm charts and that may also help you with a dependency need where in your helm file you can specify multiple charts that need to kick off at the same time. Great idea. Any other questions? Of course. I was kidding. Are there any alternatives to helm? Any alternatives to helm? Yes. Dozer was a good alternative, although that one's kind of fallen out of favor. Some people will do this with Terraform or with Pulumi and that's definitely possible. Some people actually automate helm with Terraform or Pulumi, which is really intriguing. And I guess the biggest competitor to helm is Kubernetes YAML. If your YAML is simple enough, you don't need this level of complexity. You don't need a helm chart just to deploy one Kubernetes resource. Great question. Thank you. Speaking of alternatives, I'll try to make it a question and not a rant. I mean, there are many other alternatives, right? Like, you know, KPT and customized a little. That solves a little bit of a different problem. And Jason Net and Q and Dahl and CDK8 and so on. And like, what would you say to the argument that you should not be using helm that is actively harmful? Oh, good question. So there are indeed lots of alternatives. And you mentioned much more than I did. That's awesome. What about the argument that helm is actively harmful? I agree. You're focusing in on just those values that you care about and kind of abstracting out all of the rest. Now it is definitely a leaky abstraction because well, what if the bug is over there? If you don't get how Kubernetes works and you're learning helm straight away, that's probably not going to help you very much because as soon as your helm chart breaks, you're stuck. Learn Kubernetes and how Kubernetes works. When you get tired of that Kubernetes YAML duplication, then pick up helm or another tool that works in a similar way. But if you just have a very simple YAML file, helm is definitely overkill and can be harmful. I have a question. What are the options for doing continuous integration with helm? So like, if you are depending on somebody else's helm chart and you just want to have a hands-off approach where if a new update comes in, you don't have to touch it. Just reply it again and keep your cluster up to date. Good idea. So what are the options of integrating helm with a DevOps pipeline, a continuous integration pipeline? That is a great question and we poked at that a little bit here. Let's imagine that I've got this helm chart that I'm building and I also want to depend on the latest version of somebody else's helm chart and that's kind of all I have here in my DevOps build. So I might have two lines towards the end of my Docker build step. The first one is helm install or maybe helm upgrade. Maybe I want to do a helm list, pipe it to said, figure out if it's installed already. And at that point, I'm just installing the latest version of their helm chart. And then the next line is helm template chart, you know, whatever folder name it is, set some environment variables and pipe that to kubectl apply. And that's a great way in my DevOps pipeline to integrate helm into that process to ensure once the build succeeded, that latest version is installed. If you want to take it a step further, helm LS and grep that for your current version, the version of the helm chart that you just deployed to validate that that's the thing that's actually running, maybe even kubectl apply, pipe it to grep for, you know, running or even more low tech. And I both love and hate this. Curl the public URL of your application and make sure you get back at 200. If you don't fail the build. Yeah, that's what help check does. You're right. Any other questions? All right. Everyone give it up for Ron. This was a lot of fun. Thanks for letting me play. That's just what I'm looking up. All right. Next up at what is that 430. We've got Daniel Walsh on podband new features. Ooh, that one looks like fun. It does. So see everyone back cloud native track in about a half hour. Thank you. Testing testing. Is that good? All right. Welcome to the cloud native track at scale. Our next speaker is going to talk about pod man, which is a full featured container runtime. And he's Dan Walsh, who I would contend is the world's foremost expert on that subject. He's spoken at scale before, and he's a great speaker. So with that said, take it away, Dan. Thank you. Okay, so basically my team built a whole series of container tools, and we're going to be talking about pod man. How many people here have never heard everybody's heard of it? How many have never used pod man before? Okay, so what way I like to start is I like to get a little anticipation going. So everybody stand up, please. Okay, please out all text that is written in red. Now try to get this together. All right, everybody together. Okay, so this is my, I like to think of myself as Dan Coyote here. You know, this might be a dated slide. How many people don't know who the guy on top of the Mr. McGill. But I figured some people under the age of 40 probably don't know who he is. So there'll be a lot of dated slides in here that, you know, we'll have to explain to these young ones that are in the crowd. And under 40 is now young ones. So pod man came out a few years ago, and this talk is really an update on what pod man is. I'll explain those that don't really know what pod man is, what it is in a minute. But first of all, I want to talk about the health of the pod man project. So here you can see that our GitHub statistics, we just took these back last week, and there's basically about 1500 forks of pod man at this point. And we're approaching 15,000 likes or stars in the project. And here's another dated one for anybody that doesn't know Sally Struthers, they like us, they really like us. So tell each brothers for those that don't know won the Academy Award and she came out running out saying I can't believe they like me. So to give you a little more statistics on the pod man project compared to the Moby project, which is the upstream of the darker. In the last six months since January 1st, there's been 122 people have contributed to pod man where it's been 52 contributing to Moby. There've been over 1000 commits versus a little under 700 closed issues, 1010, 1110, and then merged PR. So basically everything's two to three to one statistics going into them. Pod man and Docker are made up of multiple other sub projects down the line. So cryo is in a sub project of pod man, but those are also cryo and container D to container engines. This was pod man is not a container runtime. It is a container engine. Okay, this, but people often make that mistake. But there's two other container engines as container D and cryo and their primary function is to actually run Kubernetes workloads. Pod man and Docker really aren't used. Normally people have packed together kind and other tools to run Kubernetes on Docker and Kubernetes on pod man, but for the most part container D and cryo are optimized for running Kubernetes. There's a mailing list for pod man. There's about 500 people on it at this point. There's a couple of messages a week. It's really just announcing, you know, like when we have release candidates, things like that. So last, you can see the growth in liked pod man over time. Since it started, I think it started in about 2017. But you can see also, you know, the sort of a steady state of likes and then something happened last. Third, anybody know what happened in September? Third that caused that huge spike in people looking for pod man. Docker rate limit. No, close. Docker desktop being proprietary. Okay, basically Docker closed source. In my opinion, Docker or at least basically said you can't run Docker if you have more than 100 employees in your company without paying Docker money. So basically the biggest question on literally became what is an alternative to Docker? And the answer was pod man. So for those that don't know what pod man is, I'll give you a talk that I gave in the last couple of years. So this is called replacing Docker with pod man. And everybody should have said pod man because it's in red, but you're not focused. So this is how you replace Docker with pod man. You do a DNF install to get a pod man. And then you say alias docker with pod man. And then I go questions. Okay, so when we were building pod man, the goal was to, the goal was to basically, we knew that Docker and if you Googled how to do something, it's going to give you a Docker command line. So how to make sure that am I fading in and out? Sorry about that. Am I back? Excellent. Now I'm loud. Okay, so basically we knew that when you looked up how to do things with containers, you would always get a Docker line from Google. And anybody, so it was common knowledge of how to run containers. And Docker, one of the great things that Docker invented was a, can you turn it down a little bit? I think I'm, I feel like I'm loud. One of the great things they did is they simplified running containers, right? And everybody sort of learned how to run containers if you played with Docker at all. So pod man copied their CLI. So basically for the first couple of versions of Docker, we really implemented the entire Docker CLI. And anytime if we get a something that says, I run this Docker command line, it works. And I run the same command line with pod man. And it doesn't work. We consider that a bug. Okay, with a few exceptions. I'm usually when I think that's a bug in Docker, I don't implement it. But anyways, so basically all you have to change is DOCKER to PODMAN. So pod man, by the way, I didn't mention pod man stands for pod manager. Okay, has nothing to do with sexuality. Okay. I'm glad you guys asked. That was, that was good. So when we replaced the Docker command line, people said that's great. We wanted to have a different tool for running containers or a alternative tool for running containers. But there's a whole bunch of tooling that's been built in the CICD systems. And different, you know, tools have been developed over time to talk to the Docker demon. Okay, to talk to the Docker socket. And, you know, pod man didn't do that in the originally. So over the last year, my screen looks fine. Okay, did I do something again? I'm going to go over an hour over now. Is it the thing that converts it to every USB-C too, if you want to convert that to? I haven't even got to the demo part of this. This is just going to be a disaster. Okay, so basically over the last year, we added an API to pod man. We run pod man as a service. So I've been renting and raving for years that you should not have a demon for running your containers. You don't need a root running demon. So I call this a socket activated service. And what happens is basically you connect to pod man with the help of system D, and we're going to explain this a little later, basically can be socket activated. So when the API connects to a pod man socket that's provided by system D, system D then activates pod man service and the pod man service will then create a container on your behalf. And then when it's done, when your container goes away, or when you stop talking to the service within five seconds, the pod man service shuts down. So you don't have to have a constantly running service on the system. Because we've added an API, we also wanted to take advantage of advanced features of pod man. Pod man is not only manages containers, but in its name it manages pods. And for those that don't know pods are one or more containers working in the same environment. It's a concept invented by Kubernetes. So pod man can interact with Kubernetes type workloads. And actually we believe that that's sort of the future. So pod man PY is a Python bindings to be able to talk to the pod man's system activated service. So you can write pod Python code to talk to the service, but we also support Docker PY. So Docker Pi, if you built any APIs, any CI, CD systems, any type of services using Docker PY, you can actually point it at the pod man service. So you don't need a Docker demon running on your system. Matter of fact, you can run Docker compose. So full Docker compose support against the pod man service. You don't have to have the Docker demon running. One of the big advantages of pod man over Docker is pod man by default runs rootless mode. So you can run all of your containers without having to be root on the system. So non privilege user, we don't put any special set UID runs with standard permissions on all Linux distributions. So unlike Docker, which runs a rootful demon to run containers and has to have multiple demons up and running in order to run containers, pod man can run simple command line, fork exec, a container underneath it, totally rootless mode. So now you can run compose, Docker compose talking to a rootless pod man socket activated service and run your entire compose suite without ever having to be root on the system. We have full support for, we just recently in the, I think, as of the 14th of this month, last month, I guess now, GitLab gave full support. So GitLab worked with Docker and now they've replaced, not replaced, but added full support for pod man as well for running services using this API server. Oh, I guess at this point we go into a live demo. What could you run? Okay, so what we're going to do right now is just show you that I actually have Docker installed on here, but you'll see that the Docker see the error at the bottom that's basically saying the Docker demon is not running on the system. So I am running Docker client, standard Docker client, against basically right now against the Docker demon, it's not running. And to prove it to you, I run a system, sorry, shouldn't hit twice, but basically you see the red line there showing you that the Docker demon is not active on the system. Now I'm running the pod man command. So I'm running sudo pod man remote for version and showing you that the pod man client, which I didn't cover yet in this talk, but pod man client can talk to the pod man service. And basically you see that both versions of pod man on the local system are obviously run both running the same version. But now I'm going to run. I show you that the pod man socket is actually activated on the system and it's listening. So this is not a this not running service. It's a system D listening at the pod man's socket. And what I'm going to do right now, the bottom line of the screen shows I'm setting an environmental variable Docker host, which basically allows the Docker client will respect the Docker host. And instead of pointing at the Docker socket, it's going to point to the pod man socket. So this is the Docker version of the Docker command running against the pod man service on the system. You can see that the Docker refined responded with the pod man engine here. And that's all information about pod man.  But if I had run containers on the system, you would have seen the part number of pod man containers. But again, this is the Docker client talking to the pod man service shows that we've implemented enough pretty much everything that Docker client needs to be able to work with the Docker service. And so now what I'm going to do is I'm going to run. So this is actually going to run a Docker client is actually going to run a container. Right now it's pulling it down as a live on the system. So I pulled down a UBI micro image from registry.access.redhat.com using the Docker client against the pod man service, pod man pulled it down and you see inside the container that's running that the environmental variable is set to pod man. So that shows you that Docker is running a pod man container on the system. So now what I'm going to do is I'm going to now I'm going to switch to running. So that was everything I just ran was root full. Now I'm going to run a rootless mode. So I'm showing you again, it's just now now notice I'm not running the sudo in front of the commands. And I'm showing you that the pod man service is up and running on the system. Now I'm going to run Docker client again against the so this just ran. So now I ran a Docker client against a rootless pod man service. I ran a rootless container using Docker client against pod man. And that should show you a dash a there. But so now what I'm going to do is I'm actually going to run Docker compose. So I'm going to run right. I just showed you that there's no containers running on my system at this point. So I'm going to run Docker compose script. And it fired off a container. So Docker compose running against the pod man service fired off a Docker in a pod man container. And now I'm going to do Docker compose down going to wait 10 seconds because the application I'm running inside of it doesn't receive the signal properly. And so sit down the service and I jump to the next demonstration. But trust me, there are no containers running on the system at this point. Okay, so I showed you pod man remote. So pod man at this point talk can talk to the pod man service as we showed in the demonstration pod man clients talks to the darkest service and watching containers. You can set up pod man to work across SSH connections. So we wanted one of the things I didn't like about Docker is Docker basically has the ability to listen to TCP sockets and people use it to set up TCP connections to it. And pod man can do the same thing, but I really don't like that. What I prefer to everybody to do is use SSH connections between. So you would talk pod man through an SSH tunnel to the pod man service. So you talk Docker P Y Docker compose, you can talk over SSH connections. Pod man has the ability to create these SSH connections using command. And one of the reasons I'm talking about this is like, well pod man can create local containers. So why would I want pod man to talk remotely to a pod man service? Well, it might be interesting for that. But I want to highlight to everybody in this room. Does everybody understand containers are Linux and Linux is containers? All right, there are some windows containers. But for the most part, when people say containers, they're talking about Linux containers. Almost everything at Docker IO and Quay.io and Artifactory, they're Linux containers. So when I'm sitting on a Mac or a Windows box running containers, I'm running Linux containers. And in order to run Linux containers, I have to have a Linux kernel running. So what on a Mac and on Windows, what they're doing is there's a virtual machine running and pod man or Docker is talking because it's a client server operation. The Docker client is talking to the Docker demon running inside of a VM when you're running Docker desktop. So pod man. Similarly, we've had a project called pod man machine, which sets up a command. So pod man can basically pod machine will set up and pull down a virtual machine to your box to your Mac or Windows box on a Mac is pulling down fedora core OS and running pod man inside of it using similar commands that we just shown with pod man remote. It can run in both rootless and rootful mode. So when you start up your service, you can say which socket you want to use, you can actually switch back and forth between rootless and rootless mode on a Mac and on a Windows box. So on a Mac, and if you want to get pod man on your Mac working, you can do basically these three commands brew install pod man, then you do pod man machine init, which will pull down the virtual machine on to containing fedora core OS will launch the virtual machine.  So the pod man that we ship on a Mac that brew is installing on your Mac is actually pod man remote. So it's a remote version had coded to always talk to a remote socket on the system. And the pod man machine command will set up full connections over SSH to the virtual machine running on your system. There is a native installer being worked on right now being developed and it should be out this fall. So that you'll be able to just basically go up to a website and double click on give me that on your Mac double click on pod man and it'll kick off and install on two machines so you don't have to use brew pod man on Windows takes advantage of WSL 2. So WSL 2 is basically this fancy I'd like to think of as a fancy way of running virtual machines on Windows. So Windows developed a really nice integrated system for running virtual machines basically running Linux kernels on their Windows machines and it's very simple to install pod man matter of fact there is a Windows installer if you go to pod man.io look for the latest release you'll see a Windows installer click on it it will install setup and install WSL 2 on your machine on your Windows machine and then start running pod man locally. So what happened in September we're addressing with these tools. There's also a very large effort going on right now in the upstream project to create a GUI for pod man it hasn't been released yet but this daily releases of what's called the pod man desktop this is going to be fully open fully free as free as in liberty and free as in beer so there will never be any charge for this tool to run on as many Macs as you want because we believe in fully open source. It's available it's going to be available or it is available on Windows Mac and Linux I prefer if you all use Linux with Fedora on it but most people don't listen to make but basically pod man loves people who like you know machines with dead fruit on the back or Windows or Linux. So pod man really loves docket compose as you see a lot of people in the world want to run docket compose have lots of lots of docket compose but we really really love Kubernetes okay we believe that Kubernetes won the orchestration battle and I believe that the future way that you want to write orchestration if you're going to write a YAML file to define how you want to run containers multiple containers in a application or multiple containers in your environment on your local machine you should do that with Kubernetes YAML files as opposed to writing it in docket compose YAML files so going forward that way you could run the same I envision at some point people can be running in their hopefully open shift but Kubernetes environments Kubernetes are open shift environments you build a Kubernetes YAML file that defines how your application is going to run underneath Kubernetes you run it through the entire test suite in that environment you write your CI CD system you have massive cloud power and now you basically decide that you want to run it on a single node well you might not want to have to install all Kubernetes to run that application on a single node think edge devices well why not have use pod man to take Kubernetes YAML and run it locally on the system so it's not an alternative to Kubernetes it's running but it will understand the Kubernetes YAML file in order to launch containers so pod man playcube is the way we do that so pod man playcube and pod man generate cuba two commands that pod man supports if you build your containers using the traditional way you build containers you know pod man run command pod man gray command you can actually do pod man generate cuba and it will take the command that you generated and will translate it into a Kubernetes YAML file so we'll out let it so if you only know how to build containers using a pod man or a Docker workflow we can generate it similarly if you have a Kubernetes YAML file you can just do pod man play actually we have new command coming out comment it's going to be pod man cuba play but pod man playcube basically takes the Kubernetes YAML file they're going to be aliases of each other and run runs the environment on it that will make it easier to migrate to and from Kubernetes with those two commands that I just talked about but we also understood that the certain commands that docker compose sort of invented that people really really like like being able to build an image from from a docker or a container file so you basically could put a container file into a directory and you can say docker compose up and it will actually read the the container file and will actually create an image for you without having to pull it down from container registry so pod man playcube now has build support so you can actually do the same thing with Kubernetes YAML files not only that but we have the down flag so in Kubernetes in docker compose as I showed in the demo you can do up and down to bring up and down a whole series of pods and contain well Kubernetes compose only runs containers but in a Kubernetes YAML file you can bring up multiple pods multiple containers in your local environment and have them come up and down with a simple pod man command to bring them up and bring them down so docker several years ago I think this was the first docker con had one of their engineers I won't name the person was proud to wear a badge that said that they rejected all system dprs most of these system dprs were coming from me so but I'm not spiteful like that so pod man is proud to say that we support fully support embrace the power of system d system d is the init system is a default init system for all linux systems for the most part I know there are a few out there that are still but for the vast majority of linux systems out there system d is the way is the init system that you use so containers should support them so pod man treats system d as a first class partner system d is pid one in a container is supported by default so if you want to run a complex service inside of a container with using system d standard unit files pod you just put it pod man sees s been a net or s been system d it will set up the container in system d mode in the mode the system d expects by default and run the container you can generate system d unit files directly from pods and containers remember I talked about pod man generate kube generating kube yaml from your running containers pod man also knows how to generate the best practices unit files so if you design an application running on your system and you want to generate a unit file that you could then take to a thousand different machines you can basically generate the unit file using pod man system degenerate and then you can take that unit file and put it on a thousand different machines when you stop the service it will run pod man on that machine to pull down the container from the container or pull down the image from the container registry and launch the container on it so it's fully integrated pod man supports socket activation one of the things you could never do with docker is because docker is a client server operation you can't pass when you run a docker command line inside of a unit file start of a service file the docker client if you want to do socket activation what system d does is hope it receives a connection opens attaches the socket to and then leaks the socket to it's the the process running inside of the unit file so the docker client got that server socket but docker client is not going to launch the container it basically had no way to hand it over to the docker demon since pod man is a fork exec model pod man can take the socket the open socket and pass it all the way down to the running container so you can use socket activation fully with pod man where you can never do it with docker sd notify is a concept in system d that basically says if you have a service that takes 10 seconds to start and you want to have other services that rely on it say a database that takes a while to start up system d to find this protocol where it can say have this the actual service just say your database server to send a signal to system d and say i'm up and ready to receive connections well that doesn't work with you running it inside of a docker container if you run it inside of a pod man container it's fully supported lastly and probably most important is pod man has built in the concept of auto update so imagine you're on an edge device so you're running a service on a system and you want that service to run for years and years and years and it's running out there just chugging away and all of a sudden you have a cve or have some kind of vulnerability well wouldn't it be nice if you could just fix your container image update it push it to a container registry and you have 10 000 of these boxes all running pod man inside of a system d unifile well what pod man will do is pod man auto update will fire up at some period of the day using system d timers will go out to the registry see that there's a new version of the image that the container it's running is based on pull that image down to the system and recreate the container running on top of that new service so you get updates to 10 000 machines instantaneously not instantaneously but over time with pod man running pod man also supports health checks so a pod man health check would fire off and make sure that the updated container rise service is running properly if it is not if the health check fails pod man will then roll back to the previously running container on the system and you will later update hopefully update your serve you know service again figure out what went wrong and update it so with pod pod man auto update you start to have a service that can be self updating in your environment i talked about rollback support so let's do another live demo okay so let's create a system d service to run a container image so what i'm doing here is there i just ran so this is the the red hat shifts ubi 8 dash dash in it which is running system d inside of a container and here you see system d running on roll 8.6 on my fedora core box just like that it's just standard there's nothing special in that command and all i all i did in that command is run this command pod man run ubi 8 in it pod man saw that the system d was in it um ran the container on top of it now the system d doesn't receive the um this is when i should put my glasses on because i'm an old man oh you can't see it there so uh what you see here is i just did pod man stop dash l this is a kind of a uh hacky thing that we added to pod man so since we're developers uh we were anybody that's ever played with darker if you want to stop a darker container you got to figure out what the id is you have to find figure out what the um um you know the name that you named your container well being lazy engineers we just basically added dash l which basically says stop the last container i started on the system so uh originally i thought that was kind of hacky and stupid but we use it all the time so we love hacks so you can see that the service stopped um running on the system so that was system d running inside of a container now i'm going to do pod man system generate uh so this is all the fields that you know you can actually add if you want to add additional fields to your unit file that we're going to generate from running containers so i'm going to run a container on my system with the top command basically just a system running top and then i'm generating out a um a unit file so this is a unit file that i just generated and we have studied and worked heavily with the system d team on the best practices for running pod man in a system d unit file so this we would heavily recommend that you use this unit file that was generated um based on this uh service uh we called the top service so now i'm gonna so now what i'm going to do is i i created a new unit file so i want to i have to do a demon reload to tell system d that this new unit file is available you see on the system that there's no services running uh there's no containers running on my service and now i'm going to start that service kick off the service you see a container running so basically full integration with system d for launching services and if i want to stop the service i do it and by the way this is all happening in rootless mode so there's no root root running um system control you have to have a special container you see i stopped the container it's that simple to run pod man in this as a service um on your system all right i mentioned a couple times um i've actually um i've been leading the container team at red hat for probably about 10 years now and i recently switched over to pod man pod man on the edge i wish it was called that um it's um rel for edge uh team at red hat um so we're working on edge devices basically small computers to run on the edge small computers to run inside of automobiles and i'm working to get containerized applications into these small working working environments as we move to the edge devices and i believe that pod man is the perfect tool for running containerized applications on edge devices i kind of jump the gun on this so you can define your containerized application inside of kubernetes yaml files running inside of open shift kubernetes deploy it then on rel core os for edge doesn't quite exist yet this is what i'm pushing for whether or not red hat probably not thrilled i'm putting up this but um okay and then so once we have that environment it's a combination of kubernetes yaml files and system d unit files we've actually now have a this blog talks about it we can actually take a kubernetes yaml file and with a simple system d command tell the system d to run the kubernetes yaml file underneath the covers system d launches pod man to interpret the unit file so you don't have to even generate the unit file you can just hand it the the yaml file and with a simple system d command you can actually tell it to run run it and we'll let run pod man underneath the covers um to run your services like that um the beauty of this is one of the beauties of running pod man on edge versus running uh versus running docker is when you run docker you always have to have at least two services running you have the docker demon a container d demon always have to be running when you run pod man pod man during the time that it's established the containers it's running but if the container is going to run in detach mode which most containers on services run pod man will disappear will exit from the system so you won't have pod man using up any system services while your application on an edge device is running and doing the only time pod man will fire up as if say system d launches it or human being launches it to go and change the the way the environment is running so pod man gives you no overhead when you're running on it allow system d to manage the lifestyle of the application and we talked about auto updates fillovers health checks all that stuff is powered under the system d so one of the things we also look at with pod man um these are things that i constantly push at um for pod man as i'm known as a security guy um the desi linux for many many years um i'm always pushing to add security to pod man to make it the most secure um way of running systems so what pod man actually supports multiple registries one of the things docker did is that hard-coded docker io is being basically the only true place to get containers images from but we wanted full support to allow you guys to to choose where you get your container images from and you can actually specify multiple um multiple um registries when you want to run your containers but that potentially opens up a risk that you could basically use what we call a short name so anybody that does a you know docker pull alpine um you expect that it comes from docker io well if you basically did pod man pull ubi eight we would want you to get it from red hat so we actually in command line mode we will actually prompt you for all the registries you have out have registered when you use a short name but we also have the ability to pre configure we have this at short names dot pod github containers short names dot com we have the ability to register names short names um in the environment so that if you use a user short name say ubi eight we basically have a uh a table that says get this this exact long name matches to that and that means um you as a uh a user of the tool you can actually drop your own short name to long name um translation table and guarantee that you always get the container image that you expect from the appropriate registry and allow users and that's this shows your short name aliases short name aliases at github short container short hub short names uh only allows we will only allow names in there that we know are correct so nobody can go grab a random you know Maria db and says that comes from you know always comes from one site but you know sent us you get it from the sent us uh registry alpine you get from the docker registry block you know so we we really um so the people allowing contributions to that are always watching for that another thing in security one of the things that uh many years ago uh when docker was first starting up there was they added the concept of linux capabilities so linux capabilities allow you to run root with less privileges on your system and there were 14 capabilities chosen to be able to rerun as docker was developed three of those capabilities i think are really stupid that they were allowed in docker but it's hard to change because docker's been running this way for 10 years um the three capabilities that we drop in podman versus docker automatically are audit right audit right allows you to write this to basically allow a container process to by default right to the auditing subsystem we think the auditing subsystem should be protected from containers and not allowed to be modified net raw allows you to create random ip packets on random any type of network packets on the system the only reason that's allowed by default is so you can ping linux allows you to ping in many ways without requiring net raw for many many years so podman will support setting up ping packets but will not allow network raw and network raw has shown to be allowed vulnerabilities and people to escape network namespace and allow you to do bad evil things the last one is really strange that it's allowed by default which is make nodes the device nodes is one way that you communicate with the kernel rootless users have never been allowed to use make node almost no one in the world ever creates device nodes on a system in in a podman container we create the device nodes that you are likely to use on your system so why is cap make node allowed by default on docker containers i don't know but i think it's because docker build at one point built a uh an application that tried to create a device node it blew up someone said hey i need cap make node and someone said okay you got it and that was the end of it so for 13 years and we've been running podman for five years six six years now without that access and no one has really noticed podman also has the ability to mask over lots and lots of the kernel file systems so by default a lot of slash proc and slash sys on the system are mounted into your containers but there is some information that could leak into it the way docker and podman gets around this is by mounting either dev null or tempfesses on top of it and you cannot unmount them so you can't see what's underneath them but podman also allows you to basically unmask those if you would need one specific one of those to unmask you can do it on docker you have to run the containers privileged so a lot of people the way they learn to run containers is i run a container i get permission denied okay i'll run it in permissive mode which is basically saying i run a command is rootless oh i got permission tonight i'll run it as root okay and that's that's basically what happens so we want to allow people to figure out and help them figure out how to basically lessen the security in order to get their application to run without turning security off altogether the last one is you can actually figure out that there are certain things that we don't currently mask so you might want to prevent your containers from seeing so we allow you to add additional masks um and then the last one i'm going to talk about right now is username space so one of the cool things that red that podman does is that podman runs in a username space that's how we do rootless containers so on your linux boxes right now you basically see a file etsy sub uid and etsy sub g id so the user ad command by default adds ranges of uids that are allocated for each user on the system and what podman does is it uses those uids to map you into a username space when you're running a container what that means is it can allow your range of uids to start at uid zero to uid 65 000 mapped to the uid defined in etsy sub uid and what the kernel does is it actually does the substitution so while in the container you will see your process running as uid zero if you leave the container that same process is probably running as your user id on the system so that's how podman takes advantage of username space but podman actually also can run rootful containers and rootless containers in different username spaces so if you can imagine you're running 10 containers on your system imagine if each one of those containers none of them were running as root even if they needed root inside of the container and they ran in a different username space so each one of them got allocated say 10 000 uids and then you just ran each one of them independently so podman dash dash username space auto will figure out the minimum amount of uids you need for your container and we'll launch each one of them so in the future i believe that if you're running your again edge device or service that's just running podman containers you should be using username space that equals auto even if you're running rootful containers and it will run them all in a different username space the reason i don't recommend that all the time for rootless mode is because we only allocate 65 000 uids so if you're running multiple containers in rootless mode you need to allocate a whole bunch more uids but anyways it's a really cool feature username space has been around a huge amount of time docker has never supported it fully okay docker supports one username space but doesn't allow you to run your containers in different username spaces kubernetes and cryo have just recently started to get username space capabilities into it and so kubernetes going forward will allow you to start to run containers on your system each one in a different username space which will be very very powerful from a security point of view so a couple of last commands we're going to talk about podman added capabilities to itself over the last year podman contain a checkpoint and restore allows you to take a container using the cryo system in the linux kernel and basically suspend it and then you can restore it back up so people when they usually think about uh suspended restore it's usually about i'm going to suspend it on one machine and copy some stuff over to another machine now you can do that with this system but i think a far more interesting thing is basically to take your container down say you're rebooting and you have a container that takes 10 minutes to start again that's a huge oracle database you can actually suspend it when the machine reboots and then when the machine comes back you can restore it and you know do an upgrade everything else and you can actually use suspend and resume for our individual containers on the system another tool we've added is podman image scp which allows you to copy container images between different services without going through a container registry so if i create a container image on my machine right now and i want to run on a different machine i have to push it to a container registry and then i got to get to the other machine and i have to pull from the container registry to the podman running on that service so podman can circumvent that and just say podman connect to the to a podman system that uh socket activated server or SSH connection and push the image directly to the service and you can push it from rootless mode to rootful so you can sit on your system you run it you create an image oh shoot i meant to run that rootful just direct run podman image scp and you can say run it as root on my machine and it'll copy the thing in that case it copies it over sudo podman secret support has been added and this is working with kubernetes secrets so we can actually put secrets into containers and container images but not have them saved to the image so this is full secret support and the last interesting one we've added is podman image mount so what we can do with podman is mount containers on your system so you can examine containers without having the container run and you can actually examine images without having a container run so if you would want to run scanning tools usually the scanning tools have to basically pull down you'll use darker to pull down the container and then in order to get the container mounted up they sort of have to run a container run and then enter well we think that running containers or processes on a on an image make the image at risk so we we basically allow you to just mount up the image get you a mount point and then you can run tools on the system to actually examine what's in an image so here is uh uh my grandson everett and he's asking uh what's next granddad and even though he doesn't call me that yet but i gotta love this picture so i have to stick it in my presentations uh so the podman future so one of the things we're looking at right now is the ability to remember i said that podman is not a runtime it is an engine there's differences that engines are things that go to container registries pull down images set them up set up storage and then generate an oci runtime specification and at that point podman darker container d trial will launch a runtime runtimes are things like c run run c kata g visor those are little tools that read an oci runtime translated into something in the kernel that actually runs the container so what podman is by default is moving away from run c to the tool called c run run c was written in go originally almost all containers in the world run with run c at this point but c run was a rewrite of the go code into c and it's much more efficient much better for running under kubernetes but also because it's using a standard c library we can easily add additional libraries to it so we've added a wasm engine so we're gonna have c run wasm we're gonna have c run k run and we're going to be able to run different workloads under the different types of oci runtimes and what the wasm one does it pulls in the entire wasm stack to the container engine and you don't have to embed your wasm run environment inside of your image so you can directly run your wasm based applications on the system anybody that went to Nathaniel's talk it was all about confidential computing and we're working heavily to get confidential computing into podman so that you can run basically container containerized virtual machines running oci runtimes so you can basically run these are light really lightweight virtual machines basically just using kvm to run your container on your system but imagine that you could run that in a confidential environment which basically means that root on your host cannot see what's going on inside of the container so again Nathan's talk previously talked a lot about confidential computing but the real goal here is that you can run an encrypted workload so again going back to edge devices imagine I'm in an edge device sitting at a windmill or on a train track somewhere and some random hacker comes up and rips the thing off the system gets to root on the system and then looks at what applications are running on the system well those applications are running in encrypted environments that even the person ripping apart the machine to get at it cannot see it imagine you're sitting in your car and you have a computerized system in your car and you have hackers to break into the car's operating system and get to root on it if you're running confidential computing you wouldn't be able to look at those so I think I just covered all of this stuff and then finally there's a shameless plug because I've written a book it's coming out yeah well right now you can get it early access at Manning Publishment the name of the book is Podman in Action so if you just google podman in action it'll come to this you cannot you can buy it and put about four dollars in my pocket so that will be very much appreciated so at this point I'll open it up to questions if I have time oh I didn't run out of time that's I usually like to run out of time so any questions yes let me get to you with the mic so everybody can hear does the kubu play support uh customize mode does it support customize this is when I reveal that I really don't know kubernetes all that well so I have no idea what a customize is so what I would do is I would open up an issue with podman and say it should support the dash k so and that's what we're doing we're looking for people to want who want to add kubernetes features to podman one thing I've been told is I can't make podman compete against open shifts so we will not make podman into a kubernetes competitor so what is the status of GPU acceleration in podman because that's the one feature that was in docker that prevented me from fully moving to podman so to what is it GPU acceleration cpu gpu like nvidia gpu yeah so for inference so gpu so this should it should fully work I don't know so the problem with the GPUs is the world of GPUs is probably the most hacked up universe so what probably is happening is you are comparing podman rootless mode to docker rootful mode and I will guarantee you that the nvidia and whoever is built at the gpu mode basically did not take user namespace into so if I think if you ran podman rootful mode it would work perfectly um what happens in so what the uh why don't we talk afterwards because I could spend 15 minutes talking about the insanity of GPUs and the way that the vendors are handling it so um let's let's move on other questions yes shout it out uh does it also install a hypervisor or do you expect a hypervisor to be so uh right now podman uses qmu so when you do a brew install podman podman requires qmu qmu comes down um apple is being very hostile to using qmu matter of fact that they've added in the m1 uh hypervisor now they've added really nice support in their native hypervisor to be able to run x86 applications on it so we think that apple's going to force podman to move towards supporting their native hypervisor so right now yes it's it forces use qmu in the future I would say it's very likely that we will also support uh apple's hypervisor just because that's the way apple is they're not very friendly at open source any other questions see it keeps on making them run back and forth across the any plans on supporting other uh init systems other init systems yeah podman is uh podman will work on any init system so whether or not it will automatically run and understand it the way another init program wants to be run or you're asking about like dominant one of those yeah well I mean right now if it's a fully open source project so if you had other types of generates that you wanted to generate from containers and you were nice enough to open up a pull request we would not block it so but yeah right now we've only done the init system that pretty much everybody uses any other questions oh same side uh the do podman still supports uh inheriting cpu weight and memory limits from the from a system d unit when you when you were not yeah so you can uh so if if you have set up in your unifile constraints then podman will run under those constraints so yeah podman has no podman has no special file special powers so if you say in a unifile that podman has to run underneath this c group and you're using c groups v2 then podman will use a subset so if you basically set put it into a c group that can only use a uh cpu2 on the system then when podman runs it cannot put the container on cpu1 it would be locked into cpu2 so yeah we will follow all the rules that the unifile tells us and that's just because podman doesn't have powers it's not it's not magical last question oh by the way i i do have podman stickers if anybody wants afterwards if you don't mind since you have extra credit for time because of the ab things i had three questions first one was um when you go in the cube play mode i'm wondering what the support is for persistent volumes does it so it will it will translate if you have a volume defined in your kubernetes yaml file it will create a local volume based on that that name so it will automatically create a persistent volume on the system okay but it's local volume it's local it's not going to go out or volume drivers or docker well docker volume and podman volume support services so you can create other types of volumes but by default all we create is a local volume for okay another quickie mentioning windows and max support how about bsd since this is so actually it's funny you should ask so um again since this is a fully open source project over the last month we've had bsd people coming in and opening up all sorts of pull requests for us to be able to support um podman on bsd using jails so it's like uh bsd machines don't have any concept of a of a container but they do have concept of jails so podman will be able to pull down an oci image container um for bsd and be able to run it on that i don't think he's completed his effort yet but it's pretty close he's got it pretty much into all the endo line libraries okay and final one from me i was just curious about the use case for why you did that image scp is that related to edge in some way that you thought it was valuable there or was that something else see the number one goal of enginez is to be lazy and we hated pushing to a ridge it just was it just it came up all the time people basically doing something root fold mode and then rootless mode and they wanted to copy the image from root fold to rootless without having to do a podman build again um and then the other use case was basically but it will work for edge devices you can imagine just scp-ing while plugging this all into say ansible scripts where you can basically pre-load pre-install all the machines um out there with images already ready to run um so you can it just was a handy tool and since we have ssh connections it would just make sense to use some kind and that's why it's called scp because it's using ssh under the covers okay with anybody else yeah i'll be around for uh i'll probably go downstairs but i can sit up here for i don't know about 10 minutes or so before anybody wants to walk up and ask me and i don't have the lovely podman shirts sorry and they're not available online but okay let's thank dam that was a great presentation and um we're gonna take a 30 minute break on the container track but since we went into edge the next speaker is actually going to be talking about edge um in relation to kubernetes so you might want to stick around for that thank you you want me to speak a bit this works yes thank you i've got six o'clock so let's get started oh i see i'll let somebody get seated here but um welcome to the last talk of saturday in the cloud native track at scale los angeles uh i've known our speaker today fredrik debian he comes all the way to talk to us from autoa canada and i've personally been seeing a lot of his presentations before and been in numerous meetings with him he's associated with the eclipse foundation as a shepherd of a number of edge related projects in the eclipse foundation and he's going to talk to us today about whether kubernetes might make sense for you in edge applications so with that said i'll let you take it away fredrik all right thank you steve uh for the introduction that's certainly the best hundred bucks i ever spent all right so i'm the only obstacle between you and the party tonight game night so thank you so much for being here we'll try to make that uh entertaining uh i think that that's the case so uh first let's start with the show of hands who in this room hates the eclipse ide okay great um who in this room knew that the eclipse foundation is much more than the eclipse ide okay excellent well if you didn't know now you know so now it's your job to let others know so we have been very bad at marketing right and many people conflate the eclipse foundation with development tools but we do much more than that in fact uh we celebrated last year the 10th anniversary of our iot working group and in 2019 we started the edge native working group and started tuning more serious around the edge and my job is really as the program manager for iot and edge computing to keep an eye on the 50 plus open source projects we have in the space iot and edge combined and and and those involved over 400 committers at this point in time so you know this is no small community and uh well it's a privilege to be uh the voice of that community in front of you today all right so uh our presentation today is really asking a fundamental question and you know this is game night so i'm i'm very kind to you the very first slide gives you the unconditional you know uncontested answer to that question so if there's one thing you should remember okay it's on the next slide and then you can go no problem okay so let's do this so should you bring kubernetes on this edge road trip and the answer to this unconditionally i'm i'm sure you will agree is maybe all right and the rest of the presentation explains you why is that okay uh but but really when you think about it you know uh building an edge computing project is literally like a road trip and if you had kids or maybe an annoying uh significant author or x significant other no i love my wife honey if you're watching this i love you but you know being stuck in a car for hours you know with somebody asking are we there yet or you know giving conflicting navigation advice or whatever is very painful and so if you're building iot or edge infrastructure and you make the wrong choice you know your experience on that particular project will be as bad as a bad road trip with that little lady there that's not in the mood okay so my goal today is to give you rule of thumbs advice you know coming not for unfortunately from my personal experience because i don't have the time to be in the field building stuff but this is what i heard from our community members and their partners and their customers and really um you know uh believe me when i report that but i'm more than happy to put you in touch directly with them if you don't believe me directly all right so uh the structure of presentation really simple first we'll take a step back and really um take the time to agree on what edge computing is and how it is different from cloud computing i think that's really important then uh we'll have a look at the kind of workflows you run at the edge and this is really critical because um a container is a container is a container yeah maybe but whatever you put in it or what you want to run outside of it is might be important so we'll take the time to really take a look at the edge landscape then uh we'll dig into options to run Kubernetes at the edge you can do it sometimes okay it's a good option sometimes not so we'll take the time to see what's available not exhaustively so i won't list every option but have a high level look at possible ways to achieve that and finally um we'll take some time uh we've been working on a number of open source projects at the eclipse foundation so i will tell you about edge ops the way that we see edge computing and how those projects very briefly fit into that and whether you should consider them with Kubernetes or instead of Kubernetes in some cases all right so um we talk about edge but edge is really you know in something that lives in parallel with the cloud and so when we talk about cloud computing this is really an environment where you have on demand availability of resources you know if we if we try to give the simplest definition of it you demand and you get resources now those resources typically they are homogeneous well of course you have different instance types but when you request a c4 or m2 instance you know what you're getting and yes you can configure it but you know it's fairly easy to have thousands of VMs with all of the same properties now um the thing is this is a large-scale environment right so you can yes request thousands of those but at the same time we forget this not limit less i have a friend in germany hello hans if you watch this i won't name him or his employer but the guy literally exhausted okay a specific instance type in europe in a specific geographical area so you couldn't create even one more of that because he had tens of thousands of that instance type running yes that's crazy it can happen okay so anyway so that's on a large scale but that's not undefined or unlimited and then of course this is centralized in the sense you connect to this handy dandy console so yes you have availability regions and servers all over the globe but you manage it in a centralized way and that's the whole point right you don't want to connect to the apac console or to the us console separately you want this centralized management and that's really important okay the edge is the complete opposite of all of that okay it's completely distributed the whole point is that you are outside the cloud outside the corporate data center and you are trying to deploy something there so that's a whole different environment and you will tell me but fred you can build a field data center and put servers in there yeah that could be mightily expensive so sometimes you will deal with whoever is doing that for you but then when we talk about the edge this is about putting a little edge device at the bottom of every sequoia in california to make sure the rlt okay that's the type of project we're talking about okay i exaggerate a bit maybe you could consolidate and have one box or 10 trees whatever but okay the point is you are in the wild literally which means you cannot go there easily which means it wouldn't be economically feasible to go there frequently and there are many many constraints there temperature humidity forest fires whatever right so that's a whole different ballgame so yeah you can build your own little field data center but that wouldn't be economical to put that throughout the mountains of california for example and we are operating at a small scale not in the sense that you you you cannot have 10 000s of edge nodes yes you can but each of those nodes has limited resources limited elasticity okay if i'm deploying little boxes in the wild maybe i don't have the budget to have you know terabytes of memory and you know petabytes of hard drives and unlimited or at least large scale resources like in the cloud okay so you need to think small so to speak when you are at the edge even if you may have powerful processors and and relatively lots of memory why you don't want to go back five years from now to replace the bugs because you deployed so much stuff on it that it doesn't deliver performance anymore okay you try to optimize early for size and you know a smaller footprint then there's the fact that it's heterogeneous you are doing things involving the real world involving more often than not a real-time mission critical type of application so you want hardware that really deliver you know the best performance possible in the smallest cost possible and typically lower power consumption and things like that which means you may not have a very heterogeneous environment you will have those little armor risk five processors that you will leverage in order to really have the best bang for the buck you will use AI accelerators in specific types of applications and all of that is something that if you work with containers you will have to tailor container images for all of those possible combinations that can be painful believe me okay but that's something that you need to take into account so ultimately if you want if you want to agree on let's say a kind of generic definition for the edge we could say that it provides compute storage networking capabilities closers closer to the edge of the network and optionally okay in the second half you maintain some level of elasticity and a consumption based pricing model if you are dealing let's say with some that someone who is an edge compute resource provider for you of course if you deploy yourself well maybe you will maintain some form of internal accounting that kind of stuff and then this really applies but this is optional really the core is that you bring okay all of those resources closer to the action now when we say we run on the cloud typically what we are running are cloud native applications right so when we say cloud native this is about a few things okay and and maybe you could add a few points to that but generally speaking we talk about you know doing in a DevOps fashion using agile methodologies microservices and containers and that's my pivotalness showing there repair repay rotate whatever I mean the whole point is that those containers are expendable right they are stateless and you want to scale taking advantage of that using a certain type application well edge native is a bit different as we will see now when you do edge computing there are a number of challenges you're trying to address and this slide really synthesized them well why would you put devices in the field and directly you know go to the cloud I mean you can do that but there are many situations where this is not wise okay and you see it on this slide there first is the problem of latency um you heard about the pig war no 1859 if I remember well so essentially when they established the border in the west between Canada and the US they decided that it would follow the 49th parallel you know through the ocean and then that the border will be in the channel between Vancouver Island and the mainland but in the channel that's a problem you know there is a set of islands smaller islands between Vancouver Island and mainland so it can be on one side in the middle or you know on the other side because of that uncertainty they were you know big problems in the area and then there's this genius us general you know hardball you know with a passion for um let's say violent action that decides hey I will solve the problem you know will occupy militarily those islands you know people were coexisting peacefully waiting for the border to be established but things were going well anyway the pig of a British settler eats the potatoes of a US farmer on the island and they can't agree on compensation you know one was asking $100 the other one was offering less than that um anyway it's been a whole mess the general sees that sends troops tension escalate and now you are this close to a war between the British and the US because a pig had eaten the potatoes of some farmer and the guy was running the show the general was running the show you know he sends a letter to Washington you know reporting about that and asking for instructions but he does his whole thing for a full month before you know an answer comes back and they send someone else to you know keep a lid on the potential conflict and finally you know that that war didn't happen but the thing is latency is a big problem you know if you have to go you know you have this oil and gas refinery you need to close this valve in 10 minutes seconds otherwise the whole complex blows up you don't have the time to go all the way to the cloud and back before getting an answer so edge is good to solve that particular problem of course because you bring the decision making logic closer to the action and you don't put into play the variable latencies of the public internet and your cloud provider and and you know you know our things can go badly there okay then there's a problem of bandwidth of course video analytics is a very trendy very sexy type of application but if you do video analytics let's say on a single hd feed that's roughly three gigabytes an hour worth of bandwidth so if you send that in real time when nothing is happening you know hour after hour after hour at nt and Verizon and whatnot will love you and your boss will hate you right so you want to do this locally and maybe start streaming live to the cloud when there's an incident you detect there's a robbery or something you know or whatever you need to detect with the analytics okay so edge helps you save on bandwidth for sure then there's the whole aspect of resiliency what happens if the network goes down maybe you heard in canada um two three weeks ago we had a massive outage one of the national wireless providers went down and it took them the better part of the date to recuperate and all of that was because they deployed a software update well our configuration update to the core routers in the network problem is you do that it went badly and they couldn't even access the data centers because the card readers were on the same network as everything else that's the core routers of everything so they had no way you know they had to force open those doors and you can imagine the mess okay so resiliency if you have something mission critical your network will go down and you need well you need to continue to operate because in the case of mission critical real-time things there are often lives at stake okay think air control think autonomous vehicles okay so you can't afford to have an outage and then while too bad we will be back in eight hours and finally data sovereignty that's really important the fact that because of regulations of or even personal preference you want data never to leave a specific physical location whether that is your home or maybe a state or maybe you know a region north america the european union whatever okay so this is especially important when you consider health care patient data is not supposed to exit that particular hospital and even if my doctor is flying let's say to japan okay is not supposed to access my patient data when he's outside the country for example things like that so it's really important and edge computing helps deliver on that all right so what makes edge native different from cloud native first you have to assume that the network will degrade or fail even if you have redundant networks you know people don't test for redundancy most of the time so oh yeah yeah we've got you know backup cellular connection there no problem did you test it even the cell network can go down as canada shown optimize for size and power that's really important power consumption is critical and you would tell me hey i don't operate over battery power no problem i don't need to optimize for power yes you do because the components that overheat because you put too much of a workload on it the lifespan will reduce you need more cooling it will throttle ask that to the owners of the new m2 uh mac book air for example you know that's a perfect case study why you should optimize for power and of course eat dissipation and finally zero trust you know when you put devices in the wild like that someone at some point will steal one or more okay and this means by default you shouldn't trust any device and then you have strategies in order to quarantine or maybe flag data that comes from devices that could have been compromised and that kind of stuff and that's really important okay so those things are not necessarily things that we think about when we do web apps that we deploy in the cloud okay but they are really really really important at the edge because we are outside the cloud in the wild okay now let's have a look at those edge computing workloads what are we running at the edge how do we package it and how it influences the kind of platform we will pick first okay when you talk about edge computing people will tell you oh there's the telco edge there are two edges three edges whatever you know and depending what they're trying to sell you okay the answer will vary our vision at the eclipse foundation there are at least in our edge native working group is this the space between the edge and the cloud is really a continuum and whatever you're trying to deploy the components for the solution will live in various physical places all the way up and including the cloud okay and that's true for all three planes management plane control plane data plane okay depending on what you're trying to build okay your components will literally be all over the place so the edge so to speak is anywhere and everywhere okay and that's really an important consideration when we say the edge is really distributed you see it right there okay now that's for people maybe less familiar with the planes just you know reminding you what this is about so the data plane are the actual software components right the control plane is really about controlling the application themselves and the real-time monitoring and finally the management plane manages the rest and does the device configuration so every year we at the eclipse foundation run an annual annuity and edge developer survey and we have a parallel commercial adoption survey for business types and those are the numbers from the 2021 edition but we'll publish the 2022 stuff for developers early october and the commercial adoption one in january by the way all of those surveys are on our website and they are all licensed under creative comments so feel free to pick a slide or pick a number and put that in your own deck okay that's something that we do on balf of the open source community you know as public service as part of our mission so we asked people in that 2021 edition hey what are you doing at the edge what are you deploying and in that particular version of the survey we were distinguishing between edge gateways and edge servers we removed that I mean the numbers are pretty equivalent on both sides as you will see but the interesting thing there is the variety of things okay yes AI is at the top but you have control logic you know for industrial automation and things like that so driving actual robots assembling a car and whatever then data analytics and sensor fusion the fact that you assemble and aggregate all of that data to push something maybe in real time maybe at regular intervals somewhere else okay and you do some of that mashing or preprocessing or merging on site closer to the data right then we asked people how do you deploy stuff and you would think that hey it's containers all the way no yes containers are part of the thing but there are native binaries especially for smaller microcontrollers you know you have a premium on your space and compute power so you keep things simple script files you know whether you know we didn't ask exactly what technologies people were using but you can imagine a number of answers there serverless functions you know some people do it and it's certainly a good model there and even virtual machines or virtual images on edge servers play a role as well so of course you want to maximize the utilization of the hardware that you want that you that you deploy of course you want to you know have the security benefits of isolation and things like that but there are many ways to achieve that and depending on the workload you don't necessarily always do it with containers so already when i said you know when i was asking do we need kubernetes on that road trip well not always because there are more efficient ways you know to to run some of those workloads without kubernetes involved right and i say i keep saying kubernetes that's my french showing up i apologize for that so that's kubernetes so please alias kubernetes okay sorry that's a bad tick i'm trying to correct it anyway um another important consideration you know you are at the edge and and keep people keep forgetting a whole half of the universe in the sense we've been using linux 3bsd macOS windows whatever all of those are you know going back to the roots of the unix time sharing system right okay time sharing type of operating systems where essentially you tiny slice your processor time and distribute that along all of the processors on the machine that's a great model especially to maximize utilization of the resources that you have the problem with that is that yes you maximize performance you maximize your resource utilization but the latencies are very variable right and the problem once again is if you have a very mission critical life-threatening type of application like let's say the abs breaks in your car you don't want to do time sharing okay you need some form of real-time system or operating system in there where you guarantee the latencies for every operation so that's a completely different mindset and that's really important because you know try to you know when I was coding I'm still coding a bit unfortunately not as much as I want but in my really coding days um you know we were um we were really focused on on on on on those types of applications that that's really um really take advantage of time sharing capabilities right the fact that you scale in a certain way and all of that but in real time it's a completely different day you know I'm true and true an IT guy information technology but if you go on the factory floor if you go on an oil field okay if you go in the wild where you deploy those edge solutions you will meet operational technology people you know the guys that support industrial processes process automation industrial automation and things like that and their mindset is completely different you crave for cutting edge right the presentation right before new stuff in podman and things like that those guys are obsessed with stability the fact that if this particular robot in the factory you know as downtime you will lose 16 million in production value this afternoon okay so this means you are very very careful about what you do right because that's directly related to the production capacities of an organization or even once again two lives you know if a pipeline fails if a nacoduct for water distribution fails this has massive consequences and you will be in the news quite fast right it can happen in IT as well but the mindset is different so the people around operational technology they are developers like you and me right because I was busy you know building web apps in my coding days and things like that like you are probably but their mindset is completely different okay and this is why you need to keep that into account take that into account when you do edge computing but because you are you know much closer to this gap between ot and it and you want to bridge it which means you need to to slow down on cutting edge sometimes and and listen to their specific requirements in order to deliver something that makes sense all right so kubernetes who got it at the edge so there are a number of options as I said I won't be exhaustive I will name some okay and but there are two now when you try to contrast the approaches there are essentially three approaches to run Kubernetes at the edge the first one is run plain kubernetes at the edge right you just deployed the open source version or they changed the name but open shift open source edition what's the name for that anyway whatever you you take that and and put a number of servers and then you fail and that's not a good approach so you have alternative approaches that have emerged in the last few years right like for example k3s from from rancher you have also microcades from from canonical and other alternatives and I just pick one you know there but you know there are two mini shift micro shift mini cube anyway plenty of stuff to love and like but the approach is always the same you try to reduce the number of dependencies and in the case of many of those lightweight kubernetes distribution you remove etcd right you don't want the thing to blow up I hope there's no big etcd fan there all right all of that to say that the approach is typically to reduce the footprint reduce dependencies and in the case of k3s they put everything in a single executable that roughly takes 40 megabytes to run so quite slim but not that slim in the grand scheme of things because many microcontrollers have less resources than that and then there's the cube edge approach okay which essentially requires kubernetes control plane somewhere but then as a set of different components the cloud core and the head job that you deploy in edge locations and those things are smaller footprint if I remember well one megabyte will be no four megabytes would be enough to to run the edge components in in in cube edge so that's better right certainly a much smaller footprint than 40 megabytes so we are getting there and all of that is fine and those projects and products even because there's a bunch of commercial products as well that I don't mention all of that that works well right people are doing actual deployments and no problem there but there's a thing you are doing containers at the edge and there are inherent limitations to what you can do with that okay now there are other projects that I didn't mention and and a few eclipse projects in the mix and all of them have a different focus most of them are cloud managed some of them can run just in edge mode without a cloud connection so for air gap environment in defense for example or things like that that's a good choice etc some of them integrate with kubernetes some of them don't anyway so that's a good reference okay and that's adapted from from a little paper you have the link to the original paper that I didn't write okay community members in our ecosystem wrote it as a scientific paper and compare some of those things anyway so you will have access to the slides and if not you know reach out to me and I will give you access to the deck and you can read the full paper there now let's have a look at specific use cases okay really to understand where kubernetes could fit or not in those use cases and the first one is the connected car we don't realize it but the modern car the car of today is really a data center this is not just about oh there's an onboard computer no there are several computers and they are connected to high-speed buses and things like that that are very specific to that industry and they're trying to standardize and go towards a more generic model there some people are even pushing for kubernetes in the car and things like that and and I suppose for some workloads that's specifically fine but when you think about it okay which in those connected car features are mission critical you know things that shouldn't fail whatever happens versus it's okay if I have to reboot the infotainment system it happens even in my car that doesn't have kubernetes I had to learn this weird control at the lead for my car because sometimes it freezes and yeah thank you Ford anyway all of that to say that when you think about driver assistance safety okay you need to have predictable latencies you cannot afford let's say for the little algorithm driving the brakes you know to be killed because oh we just restart the container or something right and and this shows that you know there are concerns or types of applications that really could be a fit for containerization inside the car and you have this little cluster and and maybe there's a need for a different approach for some of your components and that's really important right the modern factory is a data center as well right when you think about it all the level of automation robots not just for making things but for transporting things you know autonomous forklifts and things like that so once again should you run kubernetes in that particular context and it's too small it's ugly I know this comes from an eclipse project called basics that is in the space of industrial automation but what it illustrates is really the many layers that you have you know in the modern connected factory the fact that you have intelligence in the programmable logic controllers that drive the robots for example and then you have full subsystems let's say you are in a pie factory you have a full system that's really okay baking the pies and then you know that you have this huge conveyor that will bring them to the cooling tower and all of that so you have sensors to watch all of that and the packaging system at the other end so many of those things are not only machines but full subsystems you know with a various levels of components and all of that is automated by operational technology people they have their own networks they have their own set of standards protocol connectivity technologies all of that right and the level of complexity is staggering but once again some of those things can fail no problem let's say the system with little screens that remind employees that they should smile because they have a job and things like that that can fail right but whatever is at the core of the production for example shouldn't fail because you have lots of potential financial losses at play you can ruin you know 80 000 car 80 000 car very easily with a robot that doesn't complete its welding at the very precise moment it's supposed to do so once again you need more robustness than a typical container based system will provide you because you have so many levels of indirection in there so and we spoke about AI AI is not about one specific industry but there are certainly things that are very specific to AI types of applications at the edge that you need to take into account for example when you deploy AI at the edge you operate on the fragment locally of the full data set and that's really important because you need some ways to bring that back at some point to the clouds in a way that makes sense but you will make local decisions that will be a bit biased by this localization then you have always this concept of an error function if you have data that doesn't make sense or you need to validate something you will call up you know to the cloud and try to figure things out there but as we said you have to assume that the network will go down so once again you need to design around that fact and then optimization of that particular model in data set you know of those particular algorithms will depend on local conditions and there let's say you analyze humidity in the air and there's a water broken water main above your little sensor and it drips on it you have completely falsified whatever comes out of that particular node and you need to prevent well you cannot prevent such incidents you will detect them after the fact so how do you flag potentially suspicious data you know that's not in line with the trends and things like that right and and that when you are at the edge you need to be able to do it autonomously without a connection to the cloud because you never know when the network will go down okay so that's really an important design consideration there so in terms of architecture what kind of questions should you ask yourself on whether you know you will you will pick probabilities or not first there's the element of latency how predictable it is you know you're driving a nuclear powerpoint you're driving a very complex industrial process with precise timings um you're literally deciding when to break in a connected car once again you know those are things that cannot be interrupted and should operate on a predictable time frame okay then there's a question of whether you can afford to lose your data or not okay in many applications let's say a connected building a smart hotel let's say you put sensors all over the place to measure temperature humidity room occupancy can you drop 15 seconds worth of data probably it won't kill anyone you know we can maintain the temperature in this room without any problem of course if that's the international space station you cannot afford to drop any any microsecond of data right because at any time you may need to correct altitude and things like that so that's a whole different ballgame um and then there's the unicity of your instances so we said you know in a stateless model you know containers you know it's like in a in a big family you can lose a kid no problem you know you you don't hesitate okay maybe not um that example no all of that to say that in the case of typical kubernetes types of application any given container doesn't have any particular significance you'll try to be as stateless as possible well in the case of industrial automation in many processors like that or even you know real-time trading on wall street you cannot afford to lose anything and you cannot afford to lose what any specific container would be containing at that point in time at as a state so that's really an important consideration then our our constraint is your hardware and your overall infrastructure that's really important and as I said you have to assume that even if you have headroom you have to be conservative about the way you use those resources because you never know when someone at head office would have a bright idea and they will ask you oh you have those nodes put that on on them too you never know when that will happen so you need to be conservative about whatever you are using especially to prolong the lifespan of the equipment right so if you put too many things you will overeat if you overeat the equipment will fail faster and then of course you have little to no elasticity so you may over provision of do a point but since you are spreading those nodes thousands of them probably at some point you cannot afford to over provision that much you know we're not in the cloud and finally how far your control plane should be compared to the actual edge devices so once again the farther you are the more unpredictable latency you are introducing in the whole system so in some applications you can afford it in some others you can't and those are really important considerations in designing the whole solution and deciding whether you want Kubernetes in the mix or not and yeah you know one important thing is people have been doing or trying to do stateful containers or Kubernetes you know I'm ex pivotal we are trying to run databases there I wasn't involved in those efforts but really you know you are either the guy walking away from the explosion or you're the one stuck in it and most people that tried it are stuck in it okay so and and really this makes sense when you think about it Kubernetes from the very beginning has been engineered as a way to scale stateless workloads right that's the religion that you should have the mindset that you should have okay you can try to shoehorn something that states full that's not necessarily the best model especially at scale so you know that's one important thing if you have state and most real-time mission critical types of application have local state then you should be careful about this you don't want the guy to be stuck inside that fiery explosion okay so if we try to synthesize where is Kubernetes a bad fit real-time mission critical constraint devices right even if k3s let's say is only 40 megabytes you can go much lower you know with other types of solutions okay and finally there's heterogeneity you know yes people are starting to say yeah we mix and match x86 and an arm in our clusters and then you look at the documentation for that and it's scary it's not mature we're not there yet maybe five years from now it will be mature enough to do something mission critical that would be on a hybrid cluster we'll see you know I'm but don't do it don't do it yeah all right out of this to say that overall when you think about it you know edge is not about one particular technology but picking the right platforms and tools from the extensive open source toolkit that exists so this is from our white paper on the jobs and the logos in color are from eclipse the authors grayed out are from uh author provenance uh typically linux foundation and fh and things like that uh not that I don't like them we were just trying to illustrate our own footprint here and you may argue with me to know no end with about the placement of one specific icon you know a logo for a particular project you like on this the whole point is just to try to illustrate them in relation to each other and of course that's the typical deployment you could okay so we are among ourselves to know and on this so you but don't don't do this the whole point is to show that even though we are still known for the desktop ide and all of that the footprint that we have at eclipse is really you know spread all over you know the edge to cloud continuum and the DevOps continuum and the author dimension and what we don't have we integrate with so of course open stack of course Kubernetes of course things like podman or whatnot projects like fledge and edge x foundry all of that is highly relevant and interesting depending on what we're trying to build there's no one stop shop in edge and it's even tour in iot and edge because people will tell you hey I can provide you everything from the sensor to the data center yeah through their network of partners and all of that no one makes all of those things and tests them in advance right so it's more often than not an adventure and in building your open source architecture and all of that for any particular solution you have to keep that into account so please don't limit yourself explore what's available and there are plenty of mature well maintained open source projects in the space at the eclipse foundation at the linux foundation and elsewhere uh don't don't forget open stack and the open infrastructure foundation there all right now I will focus to conclude this talk about the way we see things at the eclipse foundation and when I say we I'm just the voice there right I'm speaking on the behalf of our community so um we may be right we may be wrong but that's what we agreed on and you are more than welcome to join and you know show us the right direction if you feel we are wrong uh first so the full story you know this could have been a three hour talk is in this handy dandy white paper so you have the QR code or the link to download it so what I will do is really synthesize it a bit in the context of our conversation today okay but really uh this is something that we came up together you know representatives for our major members and I sat down and we wrote that together that has been painful 30 plus pages of collective writing is something I don't want to relive again uh anyway so what is ed jobs exactly ed jobs is the fact that DevOps is fantastic but when you are at the edge you don't do naive DevOps okay think about it connected car everyone is stuck on the interstate right 430 pm you know horns blaring let's patch those cars oh yes right you don't do that you don't do that so yes you want to break the division between dev and operations that's good okay yes continuous integration but not continuous deployment you have to be a bit more intentional about that okay and and sometimes more incremental because you know you you spread over the updates on a subset of the nodes so that you have leftover capacity if something fails for example um and and of course microservices infrastructure record all of that is good but really you know I covered the challenges that the edge you know edge computing solves so edge jobs is really DevOps tweaked with those changes in mind with platforms that out of the box will help you with latency reduce bandwidth and things like that the characteristics of edge projects as well in iot in the sense that a typical edge or iot project has a longer lifespan you know you replace maybe your servers every two three three years maybe maybe even less than that right well in the case of edge you're placing those nodes for 10 years on every locomotive you know in Germany for example that's a different ball game okay um heterogeneity i talked a lot about that the constraints the physical constraints that you have vibration dust watering grass humidity etc there are so many things and you would tell me why i will use ruggedized hardware no problem there problem is that's not enough yes that's an important consideration but your data will be falsified by those vibrations for example and you need some level of correction and all of that so that's not just you know putting some code in there you have to be intentional about the way you write it and and then there's the connectivity the fact that the network will fail my brother's works for major network equipment company that rhymes with crisco and you know he hates me when i say that but you know it's always you only notice the network when it fails but when it fails you notice it right it's really so you have to assume that that's really a core assumption and of course we covered the workloads at the edge and the types of artifacts that people are deploying so you're getting more than just containers you have VMs straight binaries scripts serverless whatever and and this diversity will grow over time technologies don't die they just become niches that pay richly the old people maintaining them i'm getting to that point in my career i guess so all of that to say that edge ops is not only a philosophy but platforms that implement those things in a way that's different from the typical DevOps supporting platforms and so all of that action in the eclipse foundation happens in what we call the edge native working group and really there we are about three things we are called first so yes we care about blueprints but blueprints that come from our code okay we write the blueprint or the architecture after we implemented some stuff and add real pain with real customers okay the goal is really to simplify and streamline production deployments and really edge ops is you know this not turn star where our members whatever project they are involved are trying to implement in one way or another and when I say implement that's the four projects that you see there at this point so you have eclipse iofog eclipse fogos it's written zero five is pronounced fogos it what happened when you let engineers name a project right and trying to be clever there eclipse kanto and zeno so iofog is container orchestration at the edge okay so it will it will use an oci runtime to execute them but really this is about the fact that you have the central controller and the edge node spread all over the world and there's a built-in layer seven proxy in there that comes from redact projects copper but it's been fully integrated in it so without vpns or complex network configuration you have the central controller that can be in the cloud or completely at the edge and you control the nodes wherever they are okay so that's really one of the tools and and and that has been around you know it's so whole that it's got fog in the name so if you if you've been in that space for a while you know you know that it's a bit outdated as a name but it's got recognition so they kept it and the same for fogos it's been around for a while as well but it's a completely decentralized take so there's no central controller and really the principle is to have this unified compute fabric including microcontrollers to edge nodes to cloud and it supports virtual machines containers trade binaries and things like that so a different take on it and it's a really cool technology then there's eclipse canto which is a take on the edge more oriented towards software defined vehicle and things like that so it's an edge platform we run containers and it's got specific facilities to upload and download files and things like that so they are a bit more use case oriented in the case of that particular project and finally there's zeno zeno is a spin-off of fogos essentially they wrote a custom protocol to run fogos and then they realized hey it's got on its own so zeno is a pub sub protocol next generation one and it melds you know what dds could do with what something like mqtt could do and it's really powerful in the sense that it's built from the ground up for the edge so you start a few nodes they will also discover and connect and form a clique or a mesh or whatever but then you have a routing node that you can deploy in order to route over the public internet from long location to amateur but the routing node is also the point where you can declare distributed storage so you have a storage plugin that will support no sequel or sequel databases or even the file system or even in memory for caching and things like that and the routing node you know is the place where you have this rest api that you can you know access in order to create those storages and things like that and when you publish and subscribe data you can say let's say i publish slash us slash robot slash telemetry and that particular database will record all of that but then you could have let's say localized databases that would record data by state for example and keep the data inside that specific state if you deploy nodes you know in the corresponding physical locations okay so it's a very powerful concept and it includes support for evals and queries and things like that so this is not just pub sub this is literally a way to manage historical data built in into the protocol at the same time all of that particular work is done by our members you see LF edge there so we exchanged memberships because we support any and every open source project or community that really shares the open source way right but really you know there's always room for more i hope that particular slide to be full with your respective logos two three years from now um so my invitation to you pleas