Give it a minute. Continue your conversation. People just coming in, there are plenty of spaces up here in the front. Okay, welcome to the first session of the day on the container track. Elle came to give this talk all the way from San Antonio, Texas, and it's her first time in Los Angeles, so give her a hand. Howdy, y'all. All right, so thank you so much for joining me today in my talk, containers, what you need to know so you know what you need to know. I know that that's a bit of a verbose title when it comes to a conference talk, and I really think that it deserves a little bit of an explanation. This title came from when, a few months ago, it's probably been almost a year now, I decided I was going to learn containers. I was really excited and this is the new hotness, and I got all my courage up and I walked up to the Kubernetes team and I said, hey, guys, I want to learn containers. How do I get started? They were kind enough. They said, all right, Elle, what questions do you have? I just kind of stood there staring at them because I didn't know enough to know what I needed to know. So they gave me the advice that I'm sure a lot of you in this room have given before. They said, Elle, why don't you go read the friendly manual and then come back to us. So a little bit deflated. I went over to my computer and I pulled up the Kubernetes docs and I sat there really wishing I'd taken high school Greek and I had no idea what these docs were saying. It's been a long journey and I'd like to tell you guys that it was an easy one. Don't worry. Containers are so easy to learn. You'll fall right into it. But dear God, would I be lying to you guys? It's a hard road and it's a hard lesson to go and do by yourself. So I thought, why don't I take all the things that I've learned along the way, all the things that my mentors have taught me and give it to you guys so that you guys can have an easier journey into learning containers. If you came here today hoping for that zero to hero talk, you're going to walk out of here a container expert. Just go ahead and go. You're in the wrong room. But if you came here because you are genuinely interested in containers and you're just going like, how do I get started? What's that first step? And once I have that first step, where do I go next? This is the perfect talk for you. I'm not here to teach you everything you need to know about containers. I'm here to give you the roadmap so that you can go and develop your own journey. With that, I guess I should go ahead and introduce myself. My name is Elmar Kaz, and I'm a community architect for Linux Academy, one of the world's biggest e-learning platforms, as well as being a part of the Jupiter Broadcasting Team. This slide is really where I'm supposed to stand up here, you know, pull my shoulders back, look you in the eye and say, this is why I'm important. These are my certifications, the great things that I have done. That's just not me. Somewhere along the line, my mentors told me, El, don't ever talk about yourself. Let your work talk for you. So when people ask me what I do for a living, I say, I'm a professional noob. I love learning new things. I love breaking them and trying to glue them back together and making that, you know, two o'clock in the morning tweet to a friend going like, so the colonel says it's not there anymore, my bad. Yeah, it's happened. But that's why I love Linux, right?  And that's really what I've learned that containers give you the ability to do.   It's not just talking about myself. If you're going to learn containers, you're going to break things. And they're going to break all kinds of bad. And you're going to feel that it's your fault. You know what? It's not. Containers is a continuously developing project and actually a series of projects. So when you hit a point when you break something, you don't know what to do, reach out to me. More than likely, I've broken it. But one of the cool things about being a community architect is I build communities. Someone that I know is going to know what you did and be able to give you that next step. So whether that's two weeks from now, just reach out. You know what? I'm not going to tell you to go read that friendly manual. So the first resource I want to give you, though, is my website, LOPunk.com. I am not a web designer. Don't judge me based on it. It's basically a place for me to be able to share resources with you. You're going to find the slide deck there. You're going to find access to my GitHub, blog articles. One of the cool parts of my job is I get to give away free training. So when I do container-based training, it'll be on there as well. And you'll be able to email me and everything. These slides are not stagnant. I'm going to give them to scale. They're going to post them. But you know what? When the technology changes or I come back with a better demo, I'll update the slides and you'll be able to have access to it there. So just go ahead and write that down. And I keep saying it seriously, guys. Don't be afraid to reach out to me if you hit a stopping point. All right. So we've killed four minutes. Hopefully this is the last individual coming in because we're going to go ahead and get started. Today's agenda we're going to go over, what is a container? And I know that that seems like the most basic concept to be discussing. But I'm continuously amazed when I talk to people working with Kubernetes and OpenShift and all these container orchestration platforms. And I start talking about the actual container in the application that's being split up into processes. And they have no clue what I'm talking about. So I think it's a very important step. Then we're going to talk about what's the container made up of. We'll go old school and talk about tributed environments. We'll talk about namespaces and C groups. These are critical concepts to understand if you're going to be developing for containers to be able to understand if something goes wrong, your troubleshooting process. Then we'll talk about why containers. When you have access to AWS and Azure and Google Cloud and there's just so many, I think Oracle Cloud now, why are you bothering with containers? I think that's a really important thing to talk about because containers aren't the answer for everything. We're talking about using the best tool for what your actual needs are. Though I say that there are questions at the end, I'll be checking in with you throughout this talk. It's more important for me for you to walk out with an understanding than for me to get through the talk. Like, we can discuss all of this later. They'll be available to you. So let's get started. What's a container? When I first started learning containers, I did what I'm hoping most of you did, and I'm not alone. I went to Google and I typed in what's a container. And I read blog article and blog post and docker docs, and there's something I kept seeing over and over again. This guy. And the story is that we're supposed to think about Linux containers like these shipping containers, these big metal shipping containers that are put on boats, and they go from one side of the ocean to the other, carrying precious cargo.  Like, people build houses out of these things after they're retired from shipping. So the first thing I want you to do is every time that you see this image, replace it with this. And no, this isn't your mama's Tupperware because your mama's Tupperware is sacred. You lose that thing, you are dead.  A week later you pop open your trunk and there's your lunch from a week ago.  That's how I want you to think about containers. Now I know that there's a few of you who are like, all right, oh, cute story, but I'm here for the technical, I need to know what a container really is. So let's turn to what Docker says. Docker tells us that a container is a standard unit of software that packages up all of the code and all of the dependencies, allowing an application to run quickly and reliably from one computing environment to another. Some of you, especially some of my friends who came to support me here that's a good definition, that's really good. A few of you who are new are just like me going, high school Greek, I get it now. But that's why we're here today. So let's take a step back. When I run a computer, I'm pretty much just running processes. If you're really new, a process is a, the word process is a fancy way of saying I told the computer to do something, now I wanted to go do it. It's a task that the computer is performing. Now we're going to speak in kind of, we're not speaking in absolutes, there's always flaws to analogies, but let's say that one of these processes, process three, is a small Python script that I've been working to develop that someone has given me a review, changed the code a bit and gave it back to me. I want to run this, but I'm not 100% sure on what it's going to do when it starts changing system environments. Like it's going to change my global system environment, it could impact other applications that I'm running. But I still need to run the script. Some of you would be like, oh, that's easy, I'll go spin up a virtual machine. I'm not going to spin up a virtual machine that's going to be.  Really, you're going to go spend resources every time you want to test this application out. No, what you need is you need a sandbox. Now, those of you who've been around for a while know that we used to really use virtual environments, and you can create different security sandboxes, and really all that it is, it's a way to run an executable file and to restrict that software into a different operating system or to a different subset of the operating system, so it doesn't impact your entire, I said impact, can you tell me how I come from security background? Your entire operating system. The way that this started was actually old school, 1979, so container technology is older than I am, and it started with the charoot command. Now, charoot pretty much does what its name implies. It changes root, charoot, well, kind of. It actually changes the apparent root directory. So I'm going to tell you a little story, because that's how I learn, and the names have not been changed for the story, but I promise that nobody is the villain that I'm going to portray them to be. So, when I started at Linux Academy, I started with my mentor, Amy. We were working on learning containers based on OpenStack technology. My background is in OpenStack. And when I was hired, my manager, Terry, said, you know what, L, I really need you to work on containers, like OpenStack is your past, containers are what you're new to act. But it's really hard to give up what you love. So let's say, let's just pretend, this is just an example, that Amy and I had a server, and we spent a lot of time there teaching me how to develop code, how to work with containers. And Terry one day said, you know, L, we hired you to work on containers. I'm going to need access to that server so I can see what it is that you guys are doing. I could, in theory, go in and go into my servers. You see I have my root directory, and I've got your binaries you expect, I could create a charooted environment. That's your charooted directory. It doesn't have to be named charoot. Within it, I go in and I create new home directories. I create a bin, I create Etsy, I could even put Amy's files down there. You see them at the bottom, so I have new home files. And I give Terry access to the box. He can go in and he can see everything that we're working on. And those home directories would be full of interesting information about OpenShift and cota containers, containers on OpenStack. And my real work could be above. Now, every story sometimes needs an arch-villain. So let's say that Terry's like, no, you don't need to work on OpenStack, and he goes in and he deletes everything. All right, I pretend to be upset, but you know what? All my research is safe from above. So this is really when we begin to see the concept of real isolation. Now, I know that that explanation just kind of explains the what, but not the how. So I never said that I'm good at GUIs, so let me see if I can get my demo going here. But it'll give you an explanation to, where's my mouse? So this actually occurs. Now, hopefully it'll let me pause through it and I'll read through it in case you can't see some of it. So the first thing that I'm going to be doing is I go into my computer rabbit hole, because we go on adventures, and the home directory which I'm creating is Wonderland, as expected, right? I'm going to create two directories. One is going to be for bin, my binaries, and one for lib64. This is going to be because when I want to execute commands inside of home Wonderland, I won't have access to the actual system files. I need to bring them in. I am intruded to that environment. I copy over the files from binls to home Wonderland. If you don't know what to copy over, the which command is going to be your friend. I will show you, though, that there is a little bit of variance and a little bit. So I do an LD binls, and what that enables me to do is actually see what libraries I'm going to need in order to make those binaries work. If I'm kind of talking over your head because you're new to Linux, don't worry. I have a homework assignment that I'm going to assign to you that is going to help you be able to fully understand this. So when you go in, and I will show you, I copied binls over into home Wonderland. However, when I did a whichls, you see that it's actually pulling from user binls. This is something that generally isn't the biggest issue, is whether it's using your user binaries or your system binaries. But occasionally, there is some difference in what libraries it's using. So that LDD command is going to be your best friend because it's going to tell you exactly what to pull. You'll notice I was using Ubuntu for this example. At least I think I was young for sure I was. And so I only use bin and lib64. If you're using a CentOS system, you'll actually need to put in binlib64 and lib, but you'll be able to see that all through when it tells you your exact files here. So the next thing I do is I do an LDD for user binls just to show you guys that it is the exact same libraries.  I won't tell you the first time I did this. I made people sit and watch me create a Trudit Apache environment. It took 25 minutes, and I could see people's eyes just twitching every single time I made a typo. So we decided to go with an easier example here. Now I will remind you that these are all privileged commands, so you're going to need that sudo command. So you've seen me pull in libraries. I can create a Trudit environment now, right? Not exactly. That's because when you try to charoute home Wonderland or whatever it is, you can't. You don't have access to a shell. You will literally only have access to the things that you've pulled. So you have to go in and you have to give it access to binbash or another shell if you're using. Now I know a few of you are kind of like, oh my god, this is taking forever. Imagine being a system administrator before containers and having to create these charuted environments. When people are talking about when you build an application, you only pull in the things that you need. This is what they're referring to. You make your script in what people have access to as small as possible. All right, so we've got everything copied over now and I'm able to create my Trudit environment. Oh wait, one last thing I was going to do, I forgot. I need something to be able to show what we're doing. So I create a file called cookie.text and in this example, what else can that cookie be but other than eat me? So we go ahead and we create our Trudit environment and we remember that it's a privileged command. I leave these typos in so you guys can see that they're not perfect. All of us forget what we're doing halfway through. We can do an LS, then we make some typos, we do who am I, I don't have access to it. That's a command that's accessible in every Linux environment. So I try to cat the cookie, that's it. This is a perfect example, well it's perfect because I did it right, but it's a great example of what we mean when we talk about isolated environments and the mindset that you need to have when we talk about building your scripts and building your environments having the least amount of information available to it. So you only have access to what you need. All right, so if you were thinking, okay, but I can't what you're saying Al but I don't really fully understand like you're somebody that really loves reading white papers and man pages because you help develop that image. I want to encourage you to go and read my favorite white paper. And yes, I'm that kind of nerd that I have a favorite white paper. This was written by Bill Cheswick, it's linked to onelopunk.com so you don't have to go look for it. It's a weird endeared and studied. This is one of my favorite people in the world, I follow him religiously and he was one of the first security analyst for Bell Labs. And at the time, he was having to deal with a cracker. It's a term that's not used very frequently but it's basically a hacker with negative intent. And this hacker was just coming in and he was just owning Bell's systems. He was just tearing them up and Cheswick wanted to understand like what did this guy want? He wanted to be able to study him. He wanted to understand the concepts behind Charoot and he created what became the BSD gel command or there was other people that helped develop it I'm not going to leave it on him but he essentially created one of the first punny pots that we've ever used. So this is a great example of how container technology slowly became developed. This was in the 1990s, long before Docker really became its thing, long before Kubernetes even existed. So once you understand that, here's your first homework. I want you to explain to me or to an 8-year-old what the Linux file system hierarchy really is and how it works. If you're old-score, that's easy. I want you to go find your nephew, your niece, your kids, explain it to them and explain it in such a way that they can turn around and explain to you what you said. If you can do that, I promise you containers become easy because that's really what you're dealing with. The next thing is that you go and I want you to be able to have the same explanation to the process tree. But really be able to explain it to someone who's new. Once you can do that, go create the Truded Apache environment. Seriously, I'll even let you go Google it and pull up a doc that lets you follow command-by-command, but understand what you're actually doing, why you're bringing in the files that you're bringing, why you're bringing in the commands that you're actually bringing in. That's really the first step to getting started with containers. Once you understand that, then we can start discussing the isolation a little bit further. And we talk about namespaces and C groups. Now, I wish I had all the time in the world, but really I could spend an hour teaching you about namespaces and C groups. So I'm just going to give you a little bit of blurb on it, and once you understand it, I'll give you additional resources to be able to delve into further. So we have our username space. Our username space is actually kind of a new concept to be brought into Docker containers. So if you go out and you start looking for blog posts and articles, you're going to find a lot that were probably written with probably a year ago to six months ago and think, well, username space isn't used. Like I said, container technology is continuously developing, so now it's actually used. So make sure that you're looking at the dates that these blog articles were posted in. Anything older than six months may no longer be relevant. That's how quickly things are changing. But your username space basically allows you to have users that appear in your containers that are your root user, that are pseudo privileged users that are nobody users on your system. You have your IPC namespace. IPC is really what controls your message queues in system V. The network namespace, pretty self-explanatory, right? It controls your network. Mount namespace is what allows you to have volumes that are only visible to that container and not to other containers. You've got your process ID namespace, which controls ability for you to have privileged processes. So process one actually would be process 5,000 within your system. Your user namespace, which control, I already did that one, sorry, I'm going out of order here.  This is the one that I think people just kind of brush off, but it's really important. It allows you to have your own dedicated host name within the container. So you can actually save. This is my app one container. This means that when you're scripting things out, you're not having to go through and change your script every single time you change your container to reflect that UUID. I know I went through that really quick and it was probably like, I have no idea what you said L. So go to my website. I have a talk called containers. No, not your mom's Tupperware. That is the 100-level version of this talk. And it really does spend time going and educating you through every one of those as well as going through C groups. The big thing that I tell people to remember is namespaces reflect what you actually have access to on the box. So what you can actually see. Whereas C groups reflect what you have access to actually do on the box. That's really the split between them. This is the one and only time that I will tell people to go read the man pages. I don't know who wrote them, but wow, did they spend time on them. They seriously go down and break into what system calls are occurring on the kernel level to explain how these namespaces work. It's a beautiful work of art in my opinion. All right, so in order to kind of give you a little bit of a visual of it, and before I give you your next homework assignment, we're going to do a little bit of a demo on the network namespace. So let's see this. Okay, so I go in and I'm showing you what box I'm using. So we're in a boon-to-base system. I show you the IP table rules for the box. Simple enough, right? I got nothing here.  So IP net and s add db1. I'm using a database as an example. I go in, I can list out my databases. Now I'm going to show you the contents of var run net and s network namespace. And you'll see that there is a file handler that is created. This is actually where the work that you're doing is going to live. I know these are really quick. I promise you they're available on my site so you can go watch, but we have a lot of information to go through. So I go in and in my network net and s, I exec execute inside of db1 bin bash. Hey, I've got a bash shell. I'm the root user in here, even though I wasn't before. ID table s shows that nothing configured here. So let me go ahead and add a rule. I'm adding accepting traffic for 3306, which is my sql or any db, really. Here you go. I can see it. There it is, right? I'm accepting traffic for my sql. I exit the container. I do my ip table stash l. And my host sys, well, pseudo, remember I'm no longer that root user that I was in the container. And I can see that that rule hasn't affected my sqlable system environment. Why does this matter? We actually use this in open stack a lot because it allows us to have one server that has created all of our instances and we can give access to instant one to the db that's on instant two so that they can actually be able to communicate with one another without having to give access to the outside world a lot. It really does benefit you to understand how this works because it can really help you implement security policies for your application, hands-on learning. Then I want you to create two networks. At this point, you're laying down the networking that you would be using within a container. And once you're able to do that, you'll kind of understand the next topic. And the next topic is why containers. At this point, if you've done all the homework that I've assigned to you, you have now spent the time and you've created your own charooted Apache environment. You've spent the time and you've written your own demo for a process ID and you've set up networking. You've spent the time and you're thinking why in the bad place would I ever do this on myself and you'll realize that this is what system admins were having to do every time they went to work. So why containers? Oops, that slide did not change over there. Sorry, guys. Like I said, I never said I was good at the GUI. All right, let's see if that works. Okay. So why containers? Because they're easier. So we've gone through that whole process and I'm going to click. Come on, we can do it. No. All right. We're going to just do it small. Okay. So let's say I want to use LXC to create everything that I just told you guys about. I do an LXC list. I have no containers. You can tell I'm not cheating. LXC launch and image. I'm going to use CentOS. I'm going to name it MySent. Now, I did not have this. I did not speed it up because I wanted you all to see it. I haven't pulled this image before. It's reaching out to the web and pulling down the image from the LXC repositories. And so here we go. All right. As simple as that, my container is up and running. That took, what, maybe 15 seconds? It seems long when you're watching it. But look, the network's already done. I don't have to sit there and mess with the networking in it. If I do an LXC launch image CentOS just to show you, okay, it's done. Everything I went and assigned for you guys to do is done with you for a matter of, I don't know, five seconds. That one was up and running. The networking wasn't up when I ran the command, but that's okay. The next thing I'm going to do that you cannot see because of how small it is, but I do clear the screen in a little bit. That didn't work. If I do an LXC exec, MySent, which is the name of the container, tell it I do it. And if I, there you go, I can do a cat etsy red hat release so you can see, even though I was running Ubuntu earlier, I am now inside of a CentOS environment. And this is when I realize that it's an old CentOS image that we're going to go old school here. Cool. I have installed HD2PD and I don't have to create a charuto environment. My isolation is already happening because I'm having this happen inside of a container. What more than likely will take you a little bit longer? So I'm using docs that are already written out to show you how to create charuted environments and namespaces can be done in 30 seconds if I'm slow and my connection takes a while. That's why containers became such a big deal. Is system admins, we were tired of doing all the work by hand. All right. So let's go into, there you go. What is it like, why containers? What's in a name? So when you hear the term Linux containers, now a lot of people are really meaning Docker. But I think that there is a real use case for LXC and understanding what LXC is. So LXC stands for the L and X are Linux, the C is for container, not a lot of mystery there. And it's really kind of a supercharged charute. It allows you to isolate applications or more importantly, entire operating system distros within one system. With a lot of the work that I've been doing with the it's okay to be new concept, we tell people how do you get started in Linux. And one of the things people say, go spin up, just re-kick your computer.  A lot of times when people say they want to get started in Linux, they're talking about wanting to get started server side, wanting to understand commands. Go spin up a container, it's 15 seconds of time. And if you're doing random images down, there are always use cases. People can do some interesting things. But if you're doing basic security concepts, you're not really going to do anything to your system. So containers, especially LXD, allow you to go in and try Alpine Linux, and do it in the way that it's safe for learning. So one of things is the concept of between having an application container. This is a distinction that isn't always made for people. What I want you to understand is a container like LXD is going to enable you to run that isolated environment, but you're going to have everything you need to have Debian up and running. It's not the same as that charooted environment where I'm only bringing in specific things. That's the biggest difference between LXD and Docker is really how much do you need. If I'm just starting out developing an application and I don't need exactly that, I'm going to be writing it in Python. But what libraries are I'm actually going to be calling. LXD containers are great because it's all going to be available. Once I've got it written, once I know everything that I need, then I can move on to Docker or Kubernetes or wherever you're going to go, not limiting it to those containers. But the concepts all work together. So people say, why do I need to bother with LXD? I've got Docker. One of the things I want to tell you about LXD is that even Docker on their website says we were never written to. We were passed using LXD for their background, but the relationship is very friendly and they understand that there's used cases for both of them. So when it comes to having both LXD and Docker, what Docker gives you is the ability to have portable deployments across machines. LXD is very much rooted with what you have on your system. So if you write your application and you give it over to a friend and he runs it on his environment, if the resources aren't the same, if they aren't able to call upon everything that you had on your box, there's no guarantee that application is going to work. Docker takes that away and says, if you have Docker installed machine A and you have it installed machine B, it's going to work. And if you pay for their support, they're going to be there to make sure that it works. I'm surprised for everything, right? So I'm going to talk about how you can use LXD as an operating system containers and application-based containers. Developing both. People are always like, really, what should I use? When you're first starting out, I tell people to use LXD because you aren't going to know what you need. As you become more comfortable with the environment, expand your knowledge and move on to Docker. As part of being app centric, they also offer the ability to have automatic access to your application. You can create the process of your containers coming up and running. There's no logging in to LXD and installing Apache. All of that can be scripted out inside of your Docker files. And like I said, I'll show you that in just a bit. It allows you to have versioning. Not a lot of developers are working on applications completely by themselves. A lot of us are working in teams. And a lot of times, if something happens, if something happens to you, you would have in GitHub within your Docker files, which can be critical if you're trouble shooting something that's really affecting your uptime. It also enables you to be able to have component reuse. And we'll really explore that when we look at our Docker files. But it means being able to make very small images. So let's say that I have version one of my site. And I make a change that just pulls in maybe a different module for images. I promise there are visuals to this later, so if it's kind of going over your head, you'll be able to see a visual of it. But it's allowing my system and my containers to be able to share file system space so that I'm getting the most out of my resources. That goes with the ability to share. Let's say I write a really cool app and I want it reviewed and I wanted to be able to just give it to the world or I want to be able to give feedback. I just push it up to the Docker hub and say, hey, can you go ahead and upload your file here? I wanted students to come in and skimmy the most fun way to be able to say hello world using a Docker container. And it was really cool because I didn't have to say, okay, then go to this Google form and upload your file here and I'll download it to my system and pray that it runs. No, I just did a Docker pool, the name of their file, and watched it go by. I really wish I would have put some of those in my slides because they came up with some really cool stuff. It's because it's easy. So I do Docker container run dash D is going to enable it to run detached.  I used to get this out of it and it's what I know. I'm going to say I'm going to pull down that Apache image latest. It wasn't on my system so it knows, hey, she wants this image, I'm going to pull it for her. Then we go down, I do a Docker inspect, which is like a command line based browser. There you go, Apache, up and running and isolated. Everything that would have taken you that long to get to in a matter of, I mean, what was that? Oh, look right here. It's 38 seconds from the time I started talking to the time that I had an isolated web server up and running. So let's talk about Docker because a lot of black magic just happened, I know. So when I did that Docker run and the container image wasn't found, it was the same as running that Docker pull command.  So we're going to go in and we're going to talk specifically about Docker images, but when you write your Docker file, any time you use a run command, a copy command or an add command, it's going to create a different layer. That's what you see here with the 1.7.7, 6.5. These are different layers. When you do that, you're going to change into layer 3. Layer 1 and 2 will stay the same and only layer 3, 4, and 5 will change. That's when we talk about being able to have shared resources across containers. If I have one container running off of this version of it and another container running off a version where I made the third change, they're still sharing the bottom two file layers. Let's see that in action, though. So if I do a Docker history, it's a Docker history for HDPD. I'm not going to expect you to be able to read this. I'll show you the Docker file a little bit. When I started out with containers, I was like, file system layers. If I make a change, it's going to make a change in the size of the image. That seems so logical to me. So I went in and I started looking and this one has 55.3 megabytes and it added a file. I can tell you just from history because of how much I use this, this is just pulling in the OS layer based on a .3 and I'm like, okay, so I executed and I ran app get update. That's going to make some changes. That makes sense to me. The next thing, okay, 43.1, I go in, I can see that I have some added data and I can see 133, okay, so I copied over file. This isn't making sense. There were five files when I pulled it down. There are only four things that made a change. The change is right here.  This was a tiny change. I made a change so it created a new file system layer. This is really important when you're looking to make the smallest images possible. If you write a docker file that's just add this, make this directory, change this and I'm pulling 57 layers to get your image, that's not going to save time. I probably could have gone and written it by the amount of time that I get your file into my system. So something to keep an eye out. So if you look at the docker hub, this is what it actually looks like. I know some people prefer to look at it through docker history. I'm a visual person so I actually prefer to look at it here. And that's exactly what we were saying. So file system one, I'm sorry, layer one, comes in and it's pulling Debbie in stretch. That's what Apache is based off of for a docker container. Then we go in for that run make dir-p, hey, that's similar to Cherook, and it says what's going on. If you'll notice, there are other like this big section here that didn't cause layer changes. That's because we're just changing environmental variables. We didn't run one of those three commands that we talked about. As part of your homework, go in and look at some of these docker files and see if you can see why file system layers are being made. The last one is a copy command here. So that was your fifth layer. Interestingly though, once we look at our docker history, you see this last number it's the very last layer that's given to us. It's going to be the same as your image ID. This has finished building. It now lives on your system. So that is the image ID that you'll be using to be able to interact with your docker file. I know it says missing here. All it means is that basically this wasn't done on your system. This was all done on somebody else's. It was pulled from the docker hub. It's missing because you didn't do it somebody else did. That's something that just nobody ever explained to me. So it was important. I promised I would leave at least 10 minutes for questions. Let me show you really what docker does for us. So I do my docker list and I see that I have my container. So let me run a new one. Docker container run Apache 2. It takes me longer to run this command than anything else. We're up and running again. It's as simple as that. It was 2, 3 seconds and we have an Apache container isolated up and running. So that's what I want you to take away from is the fact that the reason that we move to containers is that it saves time and allows these things to be automated. So with that what questions can I answer? So if you've got a question, raise your hand and I'll bring you in the mic. Am I that good? Like no questions? All right. There you go. And if you are somebody that doesn't want to ask questions like in person, I'm definitely available outside. How does this relate to virtual machines and that connection does it create a virtual machine? So when you're running Docker or LXE on Linux, it does not because it's able to communicate with the Linux kernel. All Linux distributions share that Linux kernel so it allows for the communication. If you're running Docker on Windows like Windows 10 not Windows Server, a small virtual machine is actually created that runs Linux that enables you to be able to make those system calls because at this time, the Windows kernel cannot communicate with the Linux kernel. Does that address your question? Yeah. I have an extension of that same question pretty much. So if the underlying Linux machine is running on one kernel and you want to spin up an LXE running on a different kernel, would that now be able to work or would it just fail or would it create the actual virtual machine or how would that work? I totally put you on the spot. Do you know because I've never actually messed with it. I phone a friend right here. You only get one kernel for all of your containers that are on a shared host and that's part of the kind of blessing and the curse is usually you don't need another scheduler, another kernel, another scheduler per process. Containers give you a lot of the isolation of a virtual machine without introducing a new kernel, a new scheduler and all the extra overhead that goes along with it. So the process is boot faster and share the low level system internals that are capable of being reused. This is my aha moment. This is the moment where in all of a sense, so the secret sauce, the real magic behind all of this is the shared kernel. So it's like how does containers work? Why is this able to work? Why don't I need to have a virtualization here, shared kernel? That's the one that's like the magic little secret sauce behind everything. This is where you give talks right now I know to add that in here. And the way that we achieve that is by on your host machine, you need a very, very new Linux kernel and the rule with the Linux kernel has been that you try to always have the APIs be backwards compatible. And since that has always been true, if you have a very new kernel on the host, you can emulate the kernel APIs of basically any older Linux distro. And so you can have a system appear to be Ubuntu, appear to be Gen2, because all the kernel APIs are all identical. The only thing that really matters is the CH root that you mount underneath. Right. That's it. Thank you. So to piggyback off of that, so if you're trying to, essentially I guess, run a container as a virtual machine, wouldn't that be LexD at that point? There are a lot of comparisons and a lot of people who say that LexD really is kind of like a virtual machine. The argument to me is no, because you're not emulating hardware.  I'm emulating system parameters and I'm emulating OSs. With virtual machines, I'm emulating and limited to that hardware. So maybe you have some guidance as to documents about design and conceptualizing your containerized workloads. And specifically also, I'm really interested in about persistence.  You kind of go back to the whatever you install, the binaries, but you also maybe have workloads that may want to go away. So I'm kind of curious as to how you do that. I'm sorry, that mic was not really, can you rephrase? So it's kind of two questions rolled into one. One is what guidance do you have for designing container-based apps in general? And more specific version of that question is how do you persist data in a containerized environment? Okay, so I'm going to answer them backwards. Persisting data is really going to take a lot of time. I tend to specialize in Docker. So you're going to be using Docker volumes. And you can use them as mount points or you can use them as part of the image. That gets a little complicated more, your 102 level. If you go to loconk.com and you click on that Docker quick start one, I think it's still free. If it's not, get back to me and I'll make sure that you can watch the video. But it really goes into actually creating mount points and pulling information in and how when you destroy that container, that mount point, you put it in a different container or put it in your script. Going back to actual documentation. Sorry Ryan, I'm going to throw you in here. But go and play with the playground for OpenShift. Because it's where I learned how breaking apart applications actually works. Ryan did a great job and his whole team did a great job of actually writing applications that allow you to be able to make configuration changes and see it happening in your browser as you're doing it. My favorite thing about this is that you're doing a run called, was it like US maps or parks. And you're going in and you're making changes to your config, you're making changes to your application and you're seeing it happen real time. So that's definitely something that would go for. Play with Docker is a great resource and obviously I have to give us a plug. If you want to, you know, visual based learning and getting a chance to do it by yourself, I'd really encourage you to join Linux Academy and our free training. I'm not trying to sell you the branding and possibly that of others. I've heard snaps thrown around kind of similar to Docker. Specific instances would be like Next Cloud. I've spun up Next Cloud in the Docker container. I've done it as a snap. Would Snap just be a packaging utility or is it somehow used separately like an image would be? Or is Pofi when I need him? He's not in my talk. This is so wrong. Alright. So he gets, he can introduce you to someone who can really delve you into Snap. But as far as I understand it, you're just looking at a packaging. You know, it really isn't a containerized. You can use it in your containers. I quite often do. Ryan, share with the rest of the class. Alright. So basically if you have questions on that, get with us. I know Wes and Chris right here know Pofi really well and he's the one that does the Snap magic so he can get you. Alright, we good? Alright. Last one. Last one. I will answer questions afterwards. I'm not trying to tell, but I want to be respectful of the next speaker. So, for the layers, what dictates a layer is when you make a commit, is it anytime you run a command, is it outside of your control? The OS layer, or the image layers? The image layers are run, add, and execute, I believe. Are the three that will build your image layer anytime they use this?  Does that make sense? Alright. Okay, thanks. We're going to take a lunch break and then the container track is going to resume in this room with a panel session of Kubernetes users. Thank you guys. Thank you, Ryan. Thanks, Al. That was interesting. Okay. Check. Check. Check. Check. Check. Check. Testing. We'll get started in about two minutes. Did any of you guys actually see that cannon thing last night? Had never I left the cannon thing? I don't know. Okay. It's 1.30. I guess we'll get started. Welcome to the resumption for the afternoon on the container track third day. This is a session where we've got a number of Kubernetes users who are going to speak as to the experience they've had. Our panelists, I'll let them introduce themselves. We'll start with you, Ilan. Hi. Hi. I'm Ilan. I'm one of the conference organizers here at Scale, but when I'm not doing Scale, I work at Datadog where we're big users of Kubernetes. I run our product and community teams. Hi. I'm Guinevere. I'm a software engineer at GitHub. I'm on the infrastructure team, and we do a lot of things with Kubernetes. Also, I see that we're here in the corner sitting over here, so if any of you are more comfortable with actually seeing our faces, it's totally cool to get to the top of the room. Go ahead and do that. We're not going to be weirded out. Yeah. Hello. My name is Michael Goodness. I am systems architect and tech lead on the Kubernetes cloud native team at Ticketmaster. And we are also doing things with Kubernetes, which probably explains why I'm here. Yeah. So all of you are obviously using Kubernetes. You wouldn't have gotten the invite if not, but let's get into, I want to give you a little bit more detail on how to describe the details. First of all, what form are you using Kubernetes? It can range from you taking the pure open source, building it yourself to using some form of distribution or a Kubernetes as a service hosted in a public cloud. Second, when did you start? How far along the path percentage wise you think you are? Are you like 50% into Kubernetes on the way to 100 or is it 90% and then finally, please describe your workloads. Were these new apps that you lifted and shifted? Things, Greenfield things that you specifically started new for Kubernetes or some mix of the two? And where are you running Kubernetes? Is it on prem in a public cloud or some mixture of the above? Why don't you start it, Lon? No worries. So we use the open source upstream Kubernetes ourselves. We haven't used any of the cloud providers managed service offerings at Datadog around Kubernetes specifically. The challenge we had is when we started, we were starting to use Kubernetes at Datadog was that most those cloud service offerings were fairly, fairly young. So like EKS was in a private beta that we had heard about and had access to, but I think you could run maybe 50 nodes or 100 nodes in there or something like that. So we had not even a percentage of the size of the clusters that we're talking about. So we had to roll our own. Rob, one of my colleagues, Rob Ball, gave a talk in this room about 24 hours ago about a lot of this journey in the container day. And what drove us to do this is actually was moving existing apps. We had a business requirement to spin up a new point of presence in Europe, which we had never, we always had just a variety of requirements where things could not leave the European region. So we needed to spin up something in Europe. And then for business reasons, we were also asked from the business to do this on a different cloud provider. So it's like, let's change all of the things, but keep the application code the same. And so we had to port all that over and yeah. Are you saying you're on a mixture of public cloud providers? Correct. Not running your own on-premise point. And then I think your other question. None of this is, well, there's always some subset of our stuff that is new in Greenfield and being developed from scratch, but the reality is that there's, you know, the company's been around for eight, nine years now. There's eight, nine years of code that was not going to go away and get rewritten, so it was lifted and shifted into Kubernetes, which we can talk a bit more about, but was quite a bit of effort. I think the one thing that's different about our workloads than most people I talked to is that we are significantly connected to the cloud. We have a lot of folks at conferences like this, and they have a lot of stateless workloads, right, where it's just, it's their API servers, it's their websites, it's things that maybe that users interact with but don't store data. We're a time series as a service and an event storage and log storage as a service. So then are you running the state full on Kubernetes? Because some people are considered that to be leading edge. So Kubernetes is there. And then finally, the last thing I wanted you to cover was what percentage of the progress are you towards Kubernetes? Are you halfway there or are you thinking there and in full production use about the same percentage you'll ever be? So it's always increasing. We have regions that are 100% Kubernetes, and we have regions that are, like our original data center region in the U.S. fully intended to be there. It's just sort of a convinced engineers to, next time they do the deploy, do it over here rather than over here. So it's a lot of education, a lot of evangelism within the company, but it's coming very soon. Okay, thanks. Let's pass the baton. Are we taking questions? We'll get into that. We'll have some questions. Yeah, so first I want to start with I a bit of a disclaimer because I started working at GitHub very recently, but I am on one of the teams that runs our Kubernetes infrastructure and one of the really awesome things about GitHub is that we changed to Kubernetes sometime in 2016 and took about a year to gradually migrate to Kubernetes. Maybe some of you know this, but we actually started with github.com the website that many of you use day in, day out and moved all of that to Kubernetes as the very first transition to Kubernetes. One of the main reasons for this is the super supportive and vibrant community that is the Kubernetes open source community. Obviously, being GitHub we support open source. That's sort of our deal. And so in a way we like to keep friendly relations and remain each other's poster children. We at GitHub like to point at Kubernetes as one of the biggest, if not the biggest open source community that uses GitHub and we in the Kubernetes community like to collaborate with GitHub and have a really close working relationship to make sure that we align features work and that sort of thing. So that's just kind of a really cool collaboration right there. We run Kubernetes on bare metal on our own servers, which is super awesome as was already mentioned. We do not run any kind of stateful on Kubernetes. We have a data center team that does a lot of really cool work and we currently don't have any plans of migrating any of that to Kubernetes. As far as this percentage of adoption we have a lot of internal apps that basically almost every new application inside of GitHub is going to start running on Kubernetes. And a lot of our legacy apps are migrating and as far as I would say we have pretty much reached the level of Kubernetes adoption that we want and it's pretty pervasive throughout the company. On my team specifically we are building deployment tooling that makes deploying your app onto Kubernetes, onto our infrastructure as easy as typing in a chat ops command. Okay, thank you. Michael if you want to go next. So we've been running Kubernetes in production for about a year and a half now and while I personally and I think a good chunk of my team would be comfortable running kind of pure unadulterated upstream Kubernetes. We like having we like having somebody behind us. So we have been using for as long as we've been using Kubernetes we've been using CoreOS Tectonic which of course is now has been or is in the process of migrating to OpenShift. Our adoption hasn't been quite as full as I think my team certainly would like. I think we're at maybe a quarter of all of our applications have moved to the cluster. We are the default landing zone at Ticketmaster. We have reached a level of maturity and stability where that's being our platform is being promoted as go there unless you have some extraordinary extraordinary reason not to. Finally it was where is it running? Public Cloud? Sure, sure. When we started we had just made a grand announcement that we were going all in on the cloud and AWS in particular shortly thereafter we got our first quarter bill. Yeah. It was kind of everybody back into the data center. So we've been running hybrid cloud approach for about a year now. The data center clusters are right now bare metal however we are in the process of spinning up new clusters on cloud stack to at least gain the advantages of a virtualized infrastructure. Okay, thanks Michael. Maybe we'll bring it back to Elan but pretend you could go back in time and tell your newbie self one nugget of advice. If you had a do over in your path to adopting Kubernetes what would you tell yourself? Probably better education of end users of Kubernetes within the organization. There's a lot of we ended up finding a lot of teams that were looking for answers on stack overflow or placed on the internet where they thought that's clearly the right answer and it turns out that it might have been the right answer for that person or maybe it's the wrong answer for everybody depends on what they found but it probably wasn't the right answer for how we were doing things at Datadog. So basically doing more of that education more of that training and pairing with teams as they were moving. We just had a very aggressive timeline so it was like Kubernetes there go and while we partnered with the teams we should have done more and so the teams going back and doing a lot of that. I think a lot of the types of things that Gunnevere mentioned earlier around building more into our deployment process into our release process to make it a more standard path to get there rather than like here's Kubernetes APIs enjoy. That tooling and that education would have helped us move a lot more quickly. Thank you. Same question so we're going to do it differently knowing what you know now. Yeah, so I would say the main thing that I would want to do over on is the contributor integration into the upstream community. I think at GitHub we've developed a lot of really valuable tools around our existing infrastructure and we could have definitely done a much better job earlier and sharing them with the wider community at large. I have sort of made it my personal mission to rectify this which is why I kind of bring this up here. We have a couple of really nice lightweight tools that can make the transition from your existing infrastructure to Kubernetes a little bit easier perhaps depending on your use case as always but that would basically be my main regret if I could go back and do that over. Okay, thank you. And Michael, same thing, what's your do over? Sure, so it kept going in Austin in 2017 I gave a talk titled One Chart to Rule Them All the idea behind it being that Ticketmaster had created one helm chart if you're familiar with Kubernetes you would have at least heard of helm. One helm chart that would be provide a common deployment pattern for every team that was deploying to Kubernetes if I had things to do over again I wouldn't submit that talk because what we found in practice is that at least the approach that we took to creating that kind of one size fits all chart in particular just it doesn't work. What ends up happening is teams end up having different requirements to be able to auto scale someone want to be able to you know this or that exploit different parts of the Kubernetes API and by kind of locking them into that helm chart we only have two ways forward for them they either walk away from the helm chart or they add to it and stick in a whole bunch of extra templating and ends up being you demand in helm it's just no not necessarily I think right I think what we would have embraced from the outset is the idea of forking and extending. Next question back to you Elon Kubernetes is of course a traditional open source project that's governed by a steering committee but let's pretend for a moment that instead of the steering committee you're the benevolent dictator and you can put the one feature you want into the next release of Kubernetes what would that one feature on your wish list or extension be? Yeah So I'll answer this more as a person that builds products for Kubernetes unless it's the user of Kubernetes because I think that's where I'm I would there's always a very long sort of roadmap and vision instag instrumentation for metrics and telemetry within Kubernetes will go and if I had my wish we would just sort of like fast forward a year to get to the end of that so that I can then build something on that API it just right now things change constantly and so when people come to me with like Kubernetes one three which yes people do still run I assure you I have to look at one thing and then when they're with one four it's another thing and like every revision there is something different it just makes it hard for me as a person building products for that to deal with I asked my engineering team what feature they would like on Kubernetes and they said they sort of laughed and said could you actually talk to them about removing features there's too much in there so that was actually something that came up at the contributor summit and you're fair to answer this that quit adding new features and work on stability for a few cycles and that would be a fair answer but let's see what others are you kind of took a little bit of the words out of my mouth I also don't really I can't really think of a new feature that I am desperate for what I would really like is a little bit more support around cron jobs and jobs because I think half of the questions that I answer as a first responder for our Kubernetes infrastructures like hi I'm trying to delete a namespace but it won't delete why not and it's usually because there's some hanging jobs and then you have to go in and manually delete the jobs but if then there's a cron job running then you have to get rid of the cron job and so I think at least half of our headaches come around jobs and cron jobs and I would just kind of like a little bit more support around that okay thanks so my feature request would be around the metrics which is a relatively new API to begin with so it's not really an incredible ask I don't think specifically I'd be looking to decentralize it but custom metrics autoscaling is a thing in Kubernetes and more recent versions of Kubernetes however there has to be a place for Kubernetes to pull those custom metrics from right now the metrics API is centralized you have one data source for those metrics at Ticketmaster we run one instance of Prometheus per namespace or per application per product team and they all have instrumented their applications their own way so if they want to scale based on request per second and they've used a slightly different naming a slightly different name for that metric then team B they can't there's no easy way for us to kind of pull in all of those metrics into a central Prometheus and then use that for scaling that would be my kind of yes my selfish request okay thank you next question and we'll pass it back to Elan I kind of know your answer so I'm going to give you a warning here there's a huge number of supporting tools and projects listed in this CNCF landscape it's getting to the point where they're giving them away in the CNCF booth but they need to give away a magnifying glass with it to look at all of these and I know your favorite tool is going to be Datadog so for you only what's your second favorite tool so yes obviously Datadog but I think it's hard to pick one we integrate with we use many of those tools we also have to support almost all of them as our customers may not pick our favorite tools I realize this is a non-answer but honestly I'm sorry I just realized that you have partnerships probably with almost every one on there if you were a king maker you could potentially have to deal with that afterwards you can give a category or whatever or just a recommendation of where people ought to look in this chart for things that will help them out I'll say things like we've internally made the shift from Docker to container D we found that as we scaled up that was a very important ability wise and so I recommend that as folks are looking at what to power their cluster they talk to some other folks that have been running at interesting scales and see what makes sense for them similarly I think on the networking side like we found the lift C&I plug-in is quite awesome for AWS okay thank you yeah so this is a little bit of a shameless plug I have noticed that when so because of the quick release fairly quick release cycle for Kubernetes versions I think a lot of the trend currently is to instead of upgrading clusters in place to decommission older versions and bringing up new clusters and in order to do so what we do internally at GitHub we have multiple clusters and the newest clusters always the latest version of Kubernetes and then the first one we decommission eventually and bring up a new one with a new version and in order to do so and this was also probably part of how we moved over to Kubernetes in the first place we needed to do something to export cluster wide services to an external key value store that then our load balancer could access so we actually have we just recently open sourced one of those tools and it will help you bring your own load balancer to a multicluster Kubernetes infrastructure I think it's really helpful because it avoids having to use ingress for cluster access and management so okay thank you I would have to say our next favorite CNCF project would be Prometheus we're using Prometheus pretty extensively for monitoring so Prometheus yeah that's the short answer I would throw Fluent D in there as well Fluent D is great for log aggregation and shipping super flexible and then we're starting to devil with container D which I also I would agree is kind of feels like the next logical step after Docker since container D is a foundational component of the Docker daemon so yeah okay thank you what are the best sources you found for training newcomers in your organization and then after people are done pass through the newcomer stage what's your advice for getting successful outcomes with Kubernetes so we we sort of took two paths one is because we have so many customers that run Kubernetes we called all of them up and said hey what are you doing what scale are you doing it at let's chat and that was both great from a product perspective I got to know what they needed us to build but from an engineering perspective we got to exchange a lot of notes so I know you're looking for public resources on the internet but honestly a lot of our education came from just talking to peers and industry at conferences like this or one on one meetings and then we built a lot of internal educational tooling around all of that which opens sort of like open the specific educational documents we built inside but instead what we've been encouraging our team to do is share all of that knowledge at community events so Rob spoke here we had Laurent speaking at KubeCon our goal is to make sure that anything that any sort of the lessons hard lessons that we learn we're passing on to others in the community similar to what one of you was saying and so hoping to continue to do that but I don't know that there was great and I know not everybody here is in your position of being able to talk to partners the Kubernetes community in my experience is pretty open to ask your questions in Slack or Zoom meetings yeah so this is this is a little bit of a hard question for me to answer because I started Kubernetes as an intern straight out of web development bootcamp so that was interesting I think that I might be the main resource internally for people who are looking into getting good at Kubernetes because I mean she has run a contributor some of the events so you're free to plug those yeah so at the the past couple of coupons I have helped run the Kubernetes contributor workshop which is an introduction to the workflow in the Kubernetes community and where to get started where to find your first issues how we work how the project is structured how to meet people how to become unblocked how to overcome that initial fear of becoming an open source contributor where everyone can look at your code but also I maintain a list of beginner friendly Kubernetes resources that I send out to various people on a regular basis the other thing that it feels weird to recommend to people but you don't have to take the exam the recommended preparation for the Kubernetes certified Kubernetes administrator exam are actually really good preparatory tools on how to use Kubernetes as a developer and as an operator so the materials that are provided are quite extensive and good you don't have to take the exam though we can talk about that another time and I believe your events were recorded and posted on YouTube correct and the materials are on GitHub so I think gosh it's again a little bit spread out but if you look for the KubeCon Kubernetes contributor workshop on YouTube you'll probably find a series of instructional videos that will walk you through the workshop step by step and the slides live in the contributor playground repository on Kubernetes SIGS and I think there's also a link from the Kubernetes community let's see what else is there so I think the certification classes as well as they have both free and paid training classes associated with those if you go to the CNCF booth that's in the Expo Hall they have a sheet of paper that has those links on it so I'd encourage you to go there if you want to follow up on what was suggesting sure and if you can always follow up with me talk to me, find me I try to wear the SOTY at all times my name's on it then you won't even have to remember my name or peer at my badge or something yeah this yeah this is a tough one I found myself pointing more and more people more and more often to the official DAX actually has specific sections of a given documentation page the DAX open source projects don't always have the best reputation for documentation but Kubernetes DAX have gotten much much better certainly over the last three years that I've been involved with the project to the point that I do actually point some users to those DAX when they have questions I don't just send a link or try not to just send a link that's not a great way to bring somebody up to speed but at least get them started there and then maybe expound a little bit on that I think we've also found that it helps to have a set of what I call Kubernetes whispers in the organization who are available who know the API who know the specifics of our implementation of Kubernetes, how we run it how we offer it as a platform to be able to do some of that translation for people until they you know have kind of gotten up to speed and then of course trying to create tooling and processes and pipelines that make it easier for folks to get up to speed right away Okay at this point this is the end of the prepared questions that the audience members have some this is going to be difficult since we have one hand he'll die okay so let's start with you what's your question okay so I'll repeat the question observe that these people have been successful with Kubernetes the question is now that you've moved was the fact that you went to Kubernetes worth it short answer yes absolutely especially given in GitHub's position of where we are and how we interact with our contributor communities even just from like a like a visual standpoint it's just really beneficial thing for us to even just have that technology readily available running on our platform and then using our platform and then using Kubernetes to run our platform the other thing is that our uptime has significantly improved even just in the last few months it's just gotten much much much more reliable in fact if anyone remembers our last big outage in October of 2018 that wasn't that was not a Kubernetes based outage at all which actually runs into your second question about storage oh did we skip that never mind let's do that later yeah I'd say it was it was worth it for us there's one is you know just those business requirements that we had to meet in terms of being in different regions of different cloud providers I don't know that we could have done it in a timelines we had without Kubernetes it would have been a lot of manual and painful efforts but more importantly we have Kubernetes is one of the fastest growing technologies we see our customers adopting and so as an engineering organization and a product organization that wants to sympathize with our customers and understand their challenges it's really hard to reason about things like Kubernetes and containers if you're still running everything in VMs yourselves that doesn't mean that every time one of our customers wants to adopt the technology and for us to monitor it that I have to go to our engineering team and say you must adopt I don't know Oracle as a database because we have customers that use Oracle as a database you know there's but things where there's like that big of a shift in the way you think about it and the type of data that you collect and the frequency data that you collect and it just it's made our product so much better and it's made our relationship with our customers so much better to be able to say we run in the same way that you do here's how we do it here's the challenges that we've had let us help you and in the same way they've been able to help us so like we've just gotten better as a company as we do provide it to our customers because of it so I regretfully have to say that it would depend on who in our organization you ask that question of and how up how high up they are I think most of our developers would say that it was definitely worth the pain of kind of migrating to containers and learning the Kubernetes API and processes they get things like fast deployments and really high scalability but it hasn't been without as I said it hasn't been without its pains and and I think it also depends on what really I think it depends on what you expect out of it if you were embracing it because it was the next thing then you may be disappointed but if you've done the research and I think those of us who understand what Kubernetes was meant to do it wasn't simply intended to cut costs you know then it's yeah I think you're more you're going to be in a better place to be able to be successful with it thank you and I think his second part of the question was what are you doing for storage how are you accommodating stateful so we do not have a lot of stateful sets deployed in Kubernetes those that we do are EBS in AWS and we have yet to pick a persistent volume solution for the on-prem data centers we're kind of leaving that to the infrastructure team but our requirement is that whatever solution they choose has to have a storage driver available for Kubernetes with regard to that as of this point we do not have any plans to move any of our database stuff over to Kubernetes I don't know if that's in storage for the future no pun intended actually but yeah currently that's not a goal for us at all it's a mix of things a lot of the data stores that we run are actually not off the shelf data stores that you pick up off the internet but rather things we've written ourselves and so they don't always lend themselves to the off the shelf storage technologies that are out there but we do use a you know in AWS we use EBS where it makes sense in some places we use our own tools that replicate the data where it needs to be replicated it's unfortunately not a particularly useful answer for the rest of you all because you don't nobody else runs the data.datastores so okay thank you somebody here had a question okay he's asking what are the prerequisites that people should have to get into Kubernetes you mean like personal knowledge of people or hardware resources okay recommendations for a 101 class on what you have to know before you're going to be prepared to move on to Kubernetes I honestly recommend starting on kubernetes.io the docs website there is there are 101 tutorials they'll walk you through a super basic like this is what it means to run a Kubernetes cluster you don't even have to install anything because it's set up with Kata Kota it's really really nice and very instructive okay we'll go here next and I see somebody in the back your question okay so your question anticipates that you're going to run a big Kubernetes cluster and you're asking how you might take advantage of namespaces for oh okay so his question I believe is would you recommend having a Kubernetes cluster per tenant or per user group or going with namespaces it kind of depends where you want your pain points to be right at github we have multiple clusters that sort of act as one and the same infrastructure to our users our internal users but we do namespace everything but we don't differentiate between where you run stuff not really we're actually starting to group clusters less by use case and more by the properties they have as we're moving out into a multi-region um I it really depends it might be a little bit easier to not mess with namespaces and rbacks as much if you have a per cluster but then you have to manage multiple clusters so there's that I think we'll just answer these one person because I saw a lot of hands that were running short of time and maybe your second part if I I would actually like this one because it's something we're in the process of doing I have been, since I've been using Kubernetes I have been a big proponent of fewer bigger clusters multi-tenancy and then controlling that tendency through network policies and rback however again you know three years in you get some experience it's inevitable and we have some workloads that we are thinking deserve their own clusters their own sets of clusters we have embraced kind of the in place upgrade model updating the control plane in place and when you have big workloads and that's in terms of scale and in terms of resource requirements that causes quite a bit of churn that in place upgrade process creates quite a bit of node churn and when you have these big big sets of big pods it slows things down considerably and actually I think it makes the whole upgrade process a little bit a little bit sketchier than you might like but really it's very situational yeah there is definitely not one answer okay if there's time we'll get to your second question because I see multiple hands would somebody here had a question go ahead and then I'll repeat it for the mic yes so the request is that those resources you referenced can we publish the list so that he can save the search effort yeah we can get that out I'm not sure that we have a vector to post this associated with the session at scale but if we can add them to your slide deck then I'm more than happy to post them on the scale website so we'll get them and I'll I'll send out the slide deck via some means I'll put it out on twitter at least and my handles can't be wrong any other questions okay the question is have you standardized on any tools for updating your clusters kubadm or whatever you might be using for upgrade so tectonic and open shift are built around a technology called bootcube it's self-hosted kubernetes so that's what we've been relying on so far I have worked with kubadm and for building clusters that's really come a long way it's a lot easier than here again than it was a few years ago but day to day we rely on the we rely on the vendor anybody else want to describe what you do to upgrade I think somebody said that you just stand up new ones and port the things over don't we move all of our traffic over to a new cluster and then kill the old one that is currently our workflow we do patch upgrades in place actually so the downside of that moveover do you find I think you need a little extra resource to be able to host that during the time period where you've got old and new living at the same time since we have our own data center we already have the machines it's a little it wouldn't save any AWS bills to not use them yes you always have to have a little bit of extra so that you have room to move sometimes when we move over a larger cluster we have to do so gradually and only use some of the machines and decommission the old ones re what's that called like re-provision thank you I was missing that word re-provision those for the new Kubernetes version and then add them to the new cluster and then slowly move traffic so yes sometimes it's a multi-step process but we have a lot of work dedicated into making this process as easy as possible for the cluster maintainers it's as easy as a chat-ups command to provision a new node at this point for my team thank you and Elan do you want to describe what you guys are doing for upgrades still evolving I'm gonna say no yeah I mean I think we're sort of between sort of which of the two paths to take given some of the stateful decisions we've made I kind of want to ask a question though of the folks here what are you all doing around things like service meshes and inter-application communications and where is that going for you so far I have played the typical ops card of just saying no I'm actually not that proud of it are you saying no because you don't think it's ready or because you don't think it's useful or some third yes I think you know service mesh has tons of potential tons of potential I just I'm not sure if it's there yet especially when you and you know in full disclosure you know I've made these kind of decisions based on you know third party articles and conversations they just you know right now they seem to add quite a bit of operational overhead and we're just not in a place to take that on and you know the benefit to our workloads we're not running a lot of microservices a lot of our applications are you know monolithic and so the benefits of a service mesh I haven't become clear to me and when I ask our dev teams they haven't really become clear to them so that yeah that's why I don't really feel bad about the out of and no your choice you can make him answer his own question about service mesh or throw him another one really quick we actually have a dedicated team that is currently in the exploratory stage of how to do services internally they're called service communications they are also hiring so if service meshes are your thing come talk to me because there's some opportunity there and it actually looks really really exciting so I think that was a question in the back there couple we're no no I'm not we've been exploring Istio pretty pretty aggressively there's been some challenges around performance especially you know on the control plane side and not on the you know not on the the envoy piece is pretty solid so I was just curious what other folks are doing it's a thing we have to do we just haven't figured out how to do it at the scale that we need it yet so okay somebody way back here had a question I'll listen and I'll repeat it okay so the request is that he's attempting to integrate in Prometheus with alert rules and having some difficulties doing that and wants to know if anyone's got advice on how you take care of that I think so we were big users of the Prometheus operator which is you know Coros originated project the operator introduces or somewhat recently introduced an object called the Prometheus rules resource definition we're not using that very extensively however it will work very well for us once you know we've kind of made that migration I'm not sure though if you know I'm not sure that that would work very well for your specific use case as soon as you start throwing red jacks and different environments into the mix that you know it gives me some pause there is no the cube Prometheus sub-project from Coros so if you go into the Prometheus operator Git repo there's a sub-project called cube Prometheus that is using Jsonnet to do a lot of templating and in fact that's how that project is generating their Prometheus rules it's all kind of templated out using the Jsonnet DSL and then you run it through the Jsonnet without a Prometheus rules object that might work but you know we run our clusters we try to keep our clusters as similar to each other as possible so we don't have a lot of variations so I can't really speak to that specific kind of case so I think we might have time for one last question here and I'm listening go ahead okay so he's asking what does the developer workflow look like in your company do developers have direct access to deploy onto a Kubernetes cluster themselves or does it go through some kind of preliminary layer to accomplish that there's there are some that are interacting with the API directly it's not necessarily a desirable approach a lot of our deployments we're just we're using spinnaker and helm at the moment to do that and so that's the encouraged path my company builds a tool called moda which is basically an overlay configuration into our Kubernetes infrastructure so with our tooling you can create a new GitHub repository that already comes preconfigured with a deployment environment and has both the Kubernetes configs and the moda configs and the moda configs has a very very short stanza that is easy to duplicate and edit and bend to your particular use case and then the actual deployment happens via chat ops in Slack let's see yeah so this this kind of gives everyone kind of the paved road of this is how we want you to deploy apps and then moda gives you all sorts of information of how your app is performing it sends links out to our data dog and and you can you can look at your containers both via chat ops and via this the moda website interface most people do not have direct production access but the chat ops spot does we do allow direct access to the clusters the end user access however we strongly encourage deployment via CICD pipelines and this is actually something that we're constantly re-evaluating that decision because we keep getting called into incidents where if they had just made their change through a CD pipeline they wouldn't have gotten themselves into that trouble keep couple edit is pretty cool but not in prod okay thanks I'll give the speakers thanks for coming this was a great session if you want to leave any parting remarks or throw a question out last minute go for it thanks for coming I hope it's been helpful if you're interested in continuing conversation we had taken master pretty open about the way we're trying to be open about the way we run our Kubernetes clusters so get in touch with me Twitter handles everything else I'm always happy to chat any questions regarding Kubernetes how to contribute send them my way I'm Gwen Codes on Twitter I follow my Twitter account pretty closely especially around conferences thanks for coming I'll throw that out there as well where you are hiring a data dog for all things but also people with Kubernetes related stuff so yeah I would check out things I'll put out there maybe a little bit of a shameless plug but we've written a couple Kubernetes related guides that I think if you're looking to know how to monitor Kubernetes whether you do a data dog or not there are some great guides up on our blog that you can share please check out Rob Ball's talk it's already up on YouTube he goes into it in way more depth than I did on this panel and I think it's worth checking out if you didn't see yesterday thanks to the audience for coming the container crack is going to resume in a little while but we might have some of three o'clock containerized desktops thank you thanks everybody one, two okay I'll give the last few time to find a seed and we'll get started here so if you came for the containerized desktops session on the container track you're in the right room Matt came to us from Boston Massachusetts and is working on the containerized desktop at Red Hat and Josh if you've been here for other sessions this day you've probably seen him before he's an organizer of the container track event works for Red Hat and is a long time scale attendee and I like using containerized desktops so that's why I'm co-presenting here so welcome everybody so containerized desktops how many of you know who this woman is anybody? yeah I got a few in here Jesse Frizel one of my technology heroes so Dr. Conant 2015 she showed all of us how she had containerized her whole desktop how she'd put everything into containers and that sort of thing she demonstrated using Spotify in a container she demonstrated using the Chrome browser and routing all of its traffic through another container running tour so she could do anonymous browsing and that sort of thing put up a bunch of stuff on github with all of these .files etc and a lot of us felt that this was a pretty awesome concept it involved an awful lot of hackery and so a lot of people who work for different companies that do container stuff and some people in their own went away from that wanting to actually implement something similar to that only make it a lot more of something that anybody could install and use and not just Jesse and so there were some dockerish things for the U-Know Rancher OS they've got a thing in the pavilion for a while they had a desktop edition I don't know if they still do the us folks at Red Hat created this thing in the Fedora project called Atomic Workstation that was also this containerized desktop concept there was a core OS also somebody spun up an unofficial core OS desktop edition all of these sort of containerized desktop concepts all around having a desktop that was immutable running your applications in containers and installing and removing them sort of in container form I used a couple of these myself core OS desktop and also the Atomic Workstation a whole bunch alright I hope you can all hear me and this mic works seems to be okay alright so yes Atomic Workstation is one of the many experiments or attempts that people have done to do containerized desktops over the years about a year from today like last February at DevCon from Peno a few people who were interested in Atomic Workstation got together and we decided that we want to take this and push it the last mile to make it actually consumable for everyday Linux desktop users so we set out to create a project around it we formed a special interest group and over the past year we've I guess established a project and we set out and found a new name that's Silverblue that's because of us really actually and we got a new logo that you can see there over on the left to get us off the ground and I wrote a bunch of blog posts over the past year about Atomic Workstation and now Silverblue is really interested and I guess there was a little bit of concern on the Fedora side that we were ramping this up as a new thing outside of Fedora and maybe in competition to Fedora but that's why I like to stress that this is actually something that we do inside the Fedora project it's part of Fedora just another variant of the Fedora Workstation and it's actually built from Fedora content like what we use to to create Silverblue is just RPMs that you could also use in a traditional Fedora Workstation that's an image based OS and I'd like to maybe explain a little bit what we mean by that Silverblue before at Atomic Workstation is using a tool called RPMOstree to create an image and operating system files to an image taking a step back if you're running a traditional Linux distribution on your laptop if you use a package manager to update your system what's happening is that the package manager is downloading a thousand packages each contains a few files and then the package manager unwraps each package and puts the files where they belong and slash user so at the end of the day when that is all done and nothing went wrong hopefully you'll end up with a file system image on your system slash user that was created on your system and it's very unique for you what's actually in there depends on the history of your past updates and what exact version of each package every user at the end of the day has its own slash user and a traditional scenario which makes it sometimes hard to understand if a bug that a user sees is actually something that just comes to be because of the history of what's on his system or something that can be reproduced elsewhere so it makes QA a lot harder and makes things less reproducible for that reason RPMOstree takes the approach that yes we want to still use those packages they are very useful, they have been built by trusted packages over the years and that's good content but we move the composition of the image to the server side and create a single OS image with a single version number that's against the important bit here I'll check some and then every silver blue machine will update and they all have really bitwise identical content because they download the same image and in particular we can also do QA and QA on this one image so that things are in good quality so that's how the OS itself works but obviously nobody is installing a laptop just because the OS is beautifully done or updates reliably you actually want to use your system to run applications on it so we need to have an answer for applications because in silver blue we treat the OS images immutable so you cannot just take a say Firefox RPM and install it at runtime and start using the application that's not how it works so we need to have an answer for applications and that's flatback which is just another container tool I would say it's taking all the usual container technologies you know from Docker like namespaces, cgroups, sitcom, bind mounts and applies them to make a system that works well for desktop applications and basically one of the problems that we ran into with the early containerized desktops is desktop applications actually need access to a bunch of things that are not services that are available over at TCPIP port or file handles exactly the biggest one of which being debuts for all of the OS and desktop events and that sort of thing and so flatbacks are containers implemented a different way by the GNOME project in order to have access to the things that GNOME applications need to have access to and have everything else be names based off right so important aspect of being in a container is that you're independent of the OS so flatbacks when you install them and run them they are really not using much of your OS image they have their own slash user tree that matches what the application expects and can be different from what you have on your OS and each application can have a different version even so there's naturally some redundancy built into that you have to just download if in doubt you have to download more file system trees to give each application what it needs but it pays off in the sense that you can actually use the same flatback that I'm running on my raw system and can run it just the same on say Retail Enterprise Linux version that might be much older and might have very different content in its OS image but the flatback does not really get affected by that because it brings itself another aspect of container applications is that we want to have them sent box so they are isolated from your home directory and they cannot necessarily snoop around and steal your secrets and your vacation photos right and where do you get those we have an answer for that flathub which is simply an app store for flatbacks this is how it looks if you go there in your web browser and it's fairly simple but it recently reached a milestone of 500 applications so we are getting there in terms of having some critical mass please it's also nice because it's made for a better experience of when you have to install proprietary applications on your Linux laptop I would rather use open source applications but there are some things, particularly video conferencing these days where I have to install proprietary applications actually I started using this laptop primarily as my video conferencing laptop because on my native fedora laptop the other native fedora laptop zoom video conferencing platform we use a lot for kubernetes not keeping up with the various fedora additions and so they were actually putting me in a position where I couldn't upgrade fedora because I was waiting for the new version of zoom put zoom in a flatback that's not a problem anymore they have one flatback on here it runs on whatever addition of fedora that I needed on right so we have a good answer for desktop applications or at least a somewhat working answer and make some people happy like Josh but the fedora workstation is kind of targeting developers as the primary audience and of course the developers want to use Spotify too but what you really need as a developer is tools you spend most of your day typically in the terminal and you want to run compilers and debuggers and all those things and that's the workstation so flatback is not necessarily the best answer for those kinds of use cases so we have a different answer for that which is called Toolbox which is at the end of the day it's just a shell script that gives you easy access to pet containers that are ready made for developer use cases it's essentially a quick way to get a shell prompt with DNF so basically give you access to working the way that you're used to in the traditional fedora system and underneath it uses podman and scopio and other modern container technologies to do this thing and it's actually something that we kind of inherited from CoreOS the CoreOS project before it was acquired already had something called Toolbox which was basically the same idea it's a shell script that gives you a very quick pet container so yeah oops what happened there nope so so like I said I'm actually really more of a user a silver blue than I am a developer other than just reporting a lot of bugs and so the question is like why why bother, why not just use my traditional fedora laptop for everything there's a few reasons let me go over some of this one is the application isolation the second one is some security and privacy improvements over your standard stuff the concept of fearless updates which is backed by rollbacks and then the other thing is that I am a Kubernetes developer I work on container technology all the time and we really aim silver blue at the new cloud developer so in terms of isolation let me give you actually an example for something that worked out really badly on my traditional fedora laptop anybody here at python dev yes you remember in pythons or requests is like one of the most popular non-core libraries in python and for requests 211 they had a security fix that broke backwards compatibility and if you're like me and you have a lot of python based tools installed on your desktop when you went ahead and updated one of those that required the updated secure requests it broke all the ones that hadn't updated yet at least I went through that and instead of having instead of having all of your sort of tool change shared the flat packs actually bundle a lot of their dependencies into the flat pack itself right which means a bunch of redundancy in terms of storage but it means that also as a software developer I don't have to worry about this one environment that I'm installing here clobbering this other environment that I'm installing there for apps that I use rather than apps that I'm developing right apps that I'm developing run them in a virtual and run them in a container that I don't run into but for apps that I use on the desktop having them in flat packs gives me that same isolation that I get in containers for the apps that I'm developing the other thing here is that for for the flat packs we've taken sort of a key from the sort of iOS world and the phone apps world we really ought to actually have some controls and user discretion in terms of what applications require right because right now the way that it is is if you want to install an application on your laptop and Linux you sudo and you install it as an RPM and that installation process has full root access and it will do a bunch of stuff and you don't know what any of that stuff is and so that means that all of your privacy and security has to resolve around trusting the provider you know so I mean and that's fine when you're getting that RPM from the Fedora project you're getting it from CentOS you're getting it from Suzy or whatever but sometimes you actually need to install packages that aren't coming from as trusted of a source because you need new and cutting edge software and so one of the things that is built into the flat pack installer is actually giving you information about what specific resources the new application is going to have access to and which files on your system it's going to touch so you can actually take a look and you can say hey I thought this was a code scanning tool why does it need access to my audio driver and maybe actually before you move on actually if you look at this in detail not sure if you can really read this all the way in the back but it also is a nice example for showing that that's always difficult to explain to end users these kind of permissions because if you look at the list here there's stuff in there like IPC and network maybe most people will have it and have a notion of what network access means but if anything is there or SSH it's hard to explain and we're still trying to work out the best way of doing that this is the first attempt here's the graphical version of the same thing that has a little bit more clear in terms of the way the permissions again you see a lot of this in your phones these days and it feels like it's time for desktops to actually catch up to this in terms of limiting the permissions and because the flat pack is an isolated environment it means that they really can't access anything that's not on this list which is not something that you can do in a native environment that is you could have those kinds of alerts but then you couldn't actually enforce them right so as I mentioned already that one of the primary audiences for the federal workstation is developers so silver blue fits right in and it provides access to the expected set of tools here like I guess Josh working on containers knows these better than me but like docker, podman, builder all these are available out of the box on silver blue and I mentioned the toolbox already and I think I'll try to show some live demos of the toolbox later on and further down the road we look at doing a little further improving things and maybe adding actual support to the terminal application for easily giving to visualize which container you're currently in like if you're running a toolbox you'll see that later we try to like give you some hint that you are in a different environment but we could make that maybe better by actually making it part of the terminal application which is something that we'll get to eventually so one of the things and so right now it's we mainly focus on a developer because honestly the way that this started was that a lot of us who were in fedora were using the atomic workstation for our own development and so we wanted it to continue in a better you know in a new improved form so it's really aimed at developers right now but we're also thinking in the longer term you know actually and by longer term I mean like next year of the admin use case so one of the other things is that actually this is a new this replaces my previous think pad I got it replaced two months ago for corporate lease right and so I had to reinstall one of the things that I immediately noticed and remembered that I like so much about in image OS is how much faster it is to install like easily you know 10 to 20% of the time that you're used to for installing a brand new traditional Linux on a desktop or server and that's because we're not installing a whole bunch of individual RPMs we are basically just writing bits from the installation media to the disk which is a lot faster for that matter the update process is also a lot faster for the same reason because we are just applying a binary diff to the disk so it is really like an order of magnitude faster updates are also faster because we can actually download them in the background so what happens if you run APM OS 3 update on a server blue system is that APM OS 3 starts like downloading the new version of the image somewhere in the staging area in the background and you can keep using your web browser or whatever applications you have running because downloading the update does not actually touch your running OS image at all and when the download is complete you can just reboot into the new image so essentially the only downtime you have is the time that it takes to reboot the system the other thing is that if you're an administrator and you've got six dozen people so how many people here would identify mostly software developers how many people would identify as administrators what is like one of the number one things that you would like to stop your administrators you'd like to stop your users from doing installing random stuff on their desktops and see the nice thing about in image OS is most of those executable directories are read only to the user so they don't actually have the opportunity to install stuff via an unapproved mechanism and if you limit their access to downloading things like you can lock down other stuff to make it read only so that they really cannot install stuff without your approval not just via security tools and demons running on the system but actually as part of the OS the other thing is one of the tools that's in development is to have right now there's a bunch of hackery tools to do this but working on making an actual knowledge tool for you to develop your customized image right because if you're installing fedora, sentos, rel something for your users you don't want the general upstream version with just it's base level applications you're going to have a bunch of applications that you want in your company that need to be there so the idea is that you pull down the base image from the upstream add your applications as what's known as a binary layer and then redistribute that binary layer out to your users who just see it as an OS image to install so that's I mean you can do it now if you're willing to get into the depths of it in the future we will have a tool that does this so I guess we're switching gears a little bit again and this will be the fun part where we try to do some live demos of course something will go wrong and you'll have to be here with me do we want to see if we can download and upgrade in the background we can try doing that so right now this is running fedora silver blue version 29 the stable version let me actually just start with that up here most is the command line I mentioned that up here most is the tool that we're using but there's actually a command line utility by that name and it gives you some information about the system it shows you things that probably don't make much sense for most of you but it tells you where the image was put from which exact commit it is that we have here and if there was an update we could install it I'm not sure that we can actually we'll try that we'd have to rebase we can't do that can you do the command of the top of your head? I can I'll look it up while you're working on the other things I can do the command I just need to look up the source we'll get back to this at the end of this we're going to rebase this in raw height right so actually I wanted to really start with something else here so this is a silver blue system and this is a shell a host shell I would say on the whole system so I could run dnf install something because I wanted to show you maybe a gdk demo application and oops there's no dnf on this system so this is pretty shocking if you're a fidora user what am I going to do? my hands are tied behind my back we'll see so the next thing I guess to see how the system actually looks like would be to run some other commands and see what we find out at least the basic command line tools are there that's a relief and I was just looking here for how slash user is mounted and if you look over on the right you can see it's mounted read only so there's no way to make changes to the system at runtime that's why we call it immutable right but all is not lost if I run the next command here I get something back so that at least tells me that there's still in this immutable slash user file system image there's still a traditional rpm database that gives me information and it shows that the content is still composed from rpm so that's why the rpm database is there and gives us information that can be useful at times right but what do we do now? we don't have dmf so we cannot install anything here but I did mention the toolbox command earlier so I'll introduce that now there's something called toolbox and if you just say toolbox enter what happens here is that I already created a toolbox container before this talk so I don't have to spend 15 minutes watching the download prompt here with you and the toolbox enter command just basically runs this this container and inside the container is a traditional fedora system more or less and just to point that out here before I move on we try to give some little hint that you're in a different context now by adding a little unicode blue diamond character there at the very left we don't have a logo for the toolbox yet but maybe this blue diamond can become the logo it's just a little hint to tell you in which context you are maybe eventually we'll find a way to integrate that more into the surrounding terminal application and maybe give you a hint in the header bar about what context you are in but for now we just have this little blue diamond there and I'm now in the toolbox and I can try to run that command again that I was going to show you earlier okay so that's unfortunate but again so there's a DNF in this toolbox it's really more like a traditional system and I have my familiar tools that my fingers want to type so I'm feeling a lot more safe already and this will take a little bit to install and did you find that thing that we need yeah we can maybe alright so now comes the totally unprepared part of the demo this will go wrong so part of the idea behind how we treat the OS images is based on the semantics of get so what we have for each image is that we have an original source so do we need to change anything else I don't think so I think I actually just need to change the version yep yep okay hold on so Josh was trying to do yes on the fly reconfiguring the remote for the Fedora workstation image it's basically working around the unsolved problem in the repository setup where we have a different GPT key for each release and there's no existing mechanism to conveniently upgrade them without you manually entering the right URL here so that's why this is a little cumbersome yeah now we are importing the new GPT key here into the OS tree configuration and I'll be amazed that this works okay okay and I think that should whoops whoa what happened there hold on yep I think it's work now so I showed you earlier the uphemost tree upgrade command which is just basically to install the latest updates in an existing release stream and the rebase command that Josh is using here is for switching from one release stream to the next basically what we are trying to do here is take this Fedora 29 silver blue and turn it into a Fedora 30 by rebasing we'll see if this works Fedora dash workstation before that I mean to actually add mirrors here and this is why we love live demos yep it's always fun let me actually try the other URL so what I'm doing here is I'm switching it to a different server switching it to a different server within the Fedora infrastructure I'll give it one more try and then we'll switch back to working demos wait we'll probably switch it back to the same server the previous server what do you think looks like that isn't the same you don't mind alright so we'll try that again in the end switching back to my previous demo where I was installing an application a package in the toolbox you can see the install now completed and I installed this because I wanted to actually run a binary that's in the package and yeah it works so what this shows is that the toolbox is also properly set up to run graphical applications so like the display connection and everything gets set up so that desktop apps can easily run inside a toolbox and display on your host system which is otherwise something that Flatpak would do for us and another part of a demo that I'm skipping here for now is would be to try the same thing with the manual created container like basically I could use Builder to create a container for a Fedora base image and install the GTK 3D build package again and then we'll try to run GTK 3D demo it would not find the display server because Builder doesn't know how to forward the display connection and other things that desktop applications need so I'm going to skip that here because it takes a while to install but this shows that the toolbox is actually very useful even though it's mainly focused for developer use cases where you want to have access to GCC or GDB or whatever toolchain you need but it also works for desktop applications so we switch back to the day for demos then before once that's good Do you have more to demo? No I think we're skipping that I think we're skipping that I talked about this so we skip this as well and I showed this Builder what? Let's go back one I was going to just briefly in passing mention that the toolbox that I showcased here and Flatpak is not the only way to get development done on an immutable system there's other things like if you want to use IDEs as well one IDE that I mentioned here is GNOME Builder which is an IDE that's very focused on writing desktop applications and it's also very well equipped to support Flatpak for development and Richard Hargert who may be here somewhere I don't see him right now is getting a talk about GNOME Builder and the latest developments there tomorrow which you might want to look at it's a fun IDE but you can also install command line tools and build things in container wise ways right so switching gears again I wanted to finish by telling a little story that shows that while immutable OS's are really cool that we want them because they are reliable and they give us fearless updates at the end of the day it's still just software and you can still manage to get into trouble if you're not careful and you don't know how the system works and I wanted to tell a quick story of what happened to me with Silverloo on my day-to-day laptop last February when we decided to start pushing this as a project and so I could write all those blog posts based on my own experiences and one of these experiences is that I one morning was supposed to get on a video call and I still had an hour of time so I figured well with Silverloo and APMOS3 updates are fearless so I can just quickly update my system what could go wrong and now I'm using Rohite which is like the current development stream like the latest bits in Fedora and stuff does break every now and then like it has been like that forever and we're trying to make it better but occasionally there are broken composes so it doesn't boot and in fact that's what happened to me I updated my system that morning and when I rebooted to get into the new OS image I didn't get to my desktop and instead was greeted with this error message that talked about some GBM renderer problem and yeah that was not great because I needed the web browser to get on to my video call 45 minutes from then and I could have just done the right thing and said oh well I just rolled back to the previous image I could have rebooted and it would have been in the grab menu as one other item to choose and everything would have been great but I figured oh well I'm a developer I want to help people make things better so I asked somebody from the graphics team what was up with this and they asked me to provide a little utility called EGL Info and now EGL Info is not in the silver blue image as I found out when I tried to run it so I said oh well RPMOS 3 actually I did not mention that earlier but it actually lets you install packages on top of the OS image it has something called package layering so I figured oh I can just use this package layering utility and quickly install this EGL Info utility so I can get this output for the graphics team and then I'll be in my control in no time so I ran RPMOS to install EGL Utils this command here which actually worked and I got this thing layered on top and then I rebooted because to get into a new OS image with silver blue requires you to reboot into the image and well then I got to the grab prompt again and was suddenly faced with the choice between this broken image that I wanted to debug and the broken image plus EGL Utils installing EGL Utils created a new image version and RPMOS 3 was at that time at least configured to always keep two image versions around so I can always roll back to the previous version under the assumption that the previous version is a well working version but now I was already booted into the broken image when I created the new one by installing EGL Utils and so now I had two definitely broken images to choose from which I guess goes to show that I had not really fully internalized what was going on under the covers and I managed to get myself into trouble here and I had only 30 minutes until my video call so what to do I googled around a little bit and another thing that we say about RPMOS 3 apart from that it provides here is upgrades updates is that it's like git for operating system so I figured well if it's like git there has to be some history and there has to be order commits and what not that there's a command called OSTree log which I could run give it the right ref here for my system and it would tell me which commits are actually available and unfortunately it told me that there's only a single commit around only the one that I had my two broken images were based on because file system trees are big and OSTree decides to not keep too much of it around but default it only keeps the ones that it needs so I guess one or two and I googled some more and I found that there is another command that is very similar to git which is called OSTree pull which actually lets you manually download more commits of an existing ref so I ran that with the option dash dash equals five dash dash depth equals five and made OSTree pull the five most recent commits for my image and then I could manually run OSTree admin deploy to switch back to an older commit and I could reboot again and my system was working and I got onto my video call just in time so that was a I guess near death experience with an image based OS just a quick story I think we are almost at the end here there's a lot of future plans that won't take questions while I actually try to get it rebased or higher again alright so yes future plans many of these things have been mentioned before so federal authority when we set out last spring we started pushing Silverblue as a project we made a little roadmap and we said well Fedora 28 that was coming out that spring we'll have a preview release Fedora 29 is when we have the first release and then Fedora 30 it should be ready for prime time so all the major issues we identified at the time should be solved Fedora 30 is coming out in May and I guess these are all the major issues we identified back then we need to have easy support for installing the NVIDIA NVIDIA driver which is a little difficult for image based systems like OS 3 because I mean slash users we don't need to you cannot easily have a different color or things like that so we needed to have some extra support for this thankfully that has recently landed in Fedora so for Fedora 30 it should be very easy to get the NVIDIA driver working and we also have we also have pretty much complete support for Flatpak and RPMOS 3 in the graphical application that we used for installing and updating applications we saw one screen told of that earlier where we showed how flatpak permissions are actually reflected in the UI now and similarly we also have support for RPMOS 3 in there now so it should be a pretty seamless experience from that perspective in Fedora 30 and we'll have the first release two box that I demoed a little bit that will be included in Fedora 30 and it'll be useful and usable on a traditional Fedora system just as much as on Silverblue so I would encourage everybody to try it out just to get familiar with container based development workflows I guess it's a useful tool to play with and yes we have some Flatpaks available out of the box in the Fedora release we did show Flathub we talked about Flathub here earlier one aspect of Flathub is that we cannot easily enable it by default in Fedora because it's a mix of things that are free software not free software that's something that is always a little controversial in Fedora to enable by default so as an alternative we're actually building we're taking a bunch of the RPMs that we have for desktop applications and we are converting them into Flatpaks that we can make available inside Fedora out of the box we can actually use an OCI registry to serve the content it's something that we did not mention before but Flatpak can actually not only use content in OCI form but it can also pull applications out of an OCI registry yeah those were the plans for Fedora 30 these are the more long term plans for things that might happen after this spring for example integration for the toolbox and one thing we also want to do is provide some ready made toolboxes that are solving interesting problems we are working together with students at Boston University and work as interns at Red Hat and one thing they're probably going to do is work on GPU compute stuff so they need things like tensorflow so one idea was that we want to provide a toolbox that is ready made and set up with all the necessary tools to do that kind of development but we want to make more container-wise make more IDEs work well in this container-wise environment, I mentioned Cologne Builder earlier which is already the first container native ID I guess like it already works very well for Flatpaks and also Christian is working on making it work very well with other kinds of containers like potman containers but there are other IDEs out there that are very popular obviously so we should make sure that those all work well in this environment and then there's a whole category of applications that are kind of difficult to stuff in a container because they either because they run privileges or they set up their own sandboxes like Chrome has its own sandboxing built in and then it kind of interferes with the sandboxing that Flatpaks does so we need to find solutions to make those go operate nicely and VirtualBox is another example of an application that has proposed challenges on kernel module and wants to insert that into the kernel which is obviously not very compatible with being on privileged and sandbox so those are problems that we have to solve for the download okay so there's some links but why don't we go ahead and take some questions while we make another stab at re-basing this to Rahide so if you want to hear this alright I'll start with you will it be possible to provision Flatpaks out of a service catalog in a particularly open shift but other Kubernetes based environment I don't know exactly what a service catalog is to be honest but what I mentioned earlier is that we do have support for putting Flatpaks in a registry like a Docker registry essentially and ship them in that form and I'm not sure what a service catalog is like a registry something similar to that yeah I assume that should be possible it's not something that we have solved yet but I'll look into that so you mentioned customized images do the OS tree composes for those happen on Fedora infrastructure or is it like the layering that RBM OS tree does or how do the customized images work I think Josh talked a bit about that currently you can build your own images using the same tooling like APM OS tree the command line tool can also do composes and can build images if you feed it the right config files it's a little obscure and not very polished as Josh mentioned so you have to dive deep and figure out how to do it but then you can run that on your own system and create your own OS tree repository that contains the image that you built then you have to worry about where the updates come from that is very true you probably have to build updated images and also upload them into your own registry just like we do for the Fedora infrastructure any other questions how's it going up there are you making progress there's one more could you tell us a little bit about how the containers work with the local video presentation is it through you think about things like blue jeans and like sharing the screen and things like that so yeah so there are different ways to do it many test applications still use X as their display connections so what you need to do is make the X socket available inside the container and we do that if the application asks for that so the way Flatback works is it has sandboxing built in and by default you get a sandbox that's kind of tight and does not leak stuff into it but then there's a way to poke holes into it like the application ships with the manifest that says I need access to the X server or I need access to the network and one way is to like have an X connection and pass the X server socket into the sandbox or you use Wayland and then we'll pass the Wayland socket into the sandbox no not really it's really just the same way as like if you run a unsetbox desktop application it would also talk to the display server over the same socket it's just that if you're inside the container normally the socket would not be available to you unless we explicitly pass it in and that's part of the stuff that Flatback does to make applications work in containers if you run like a regular docker container that would not happen for you and you would get a couldn't find display connection error message or something from the application and the intro slide that Josh had with Jesse Frazier I mean she has manually like made the right things happen right by manually poking a hole in her docker container and passing the X server display connection in there you can of course do that but there's a lot of manual work to do that for everything that you use it's not just the display connection it's also like the debus and it's the key ring and it's like a bunch of other things that you expect to work and Flatback kinda is specialized for making all these things work alright any other questions? any progress Josh? well people start leaving I forgot that we don't have a lot of silver blue swag but I managed to bring one stack of silver blue stickers if anybody wants one I have the stack up here just come and get one it's like as a little to turn it off alright if you all come back in two hours we can always roll back alright thank you all very much sorry we moved this part of we implementing the faster downloads we moved around where everything is and that was like two weeks ago so stickers oh good oh hey you are bigger stickers I don't have one for display yeah so by the way when I was talking about faster download and install that 3,000 number right there that is a package count to give you an idea of how much faster this is than updating RPM packages in the normal realm in my current yeah installing a mainline then installing after layer with what it does build the binary so part of the idea is to actually the binary is the layer so it goes my directory so user and min and and that's the one that we're going to have to break something away and change it's still going to be a big deal yeah but some of it's like well and that's actually one of the reasons libraries are libraries are a layer and the idea of it is a lot better and part of this is all stuff that I know we're a player against that yeah with the capability in the well yeah there we go once we decided to go with discourse yeah if you just log into discourse get an account and then just enable everything as you know that's the reason why we use discourse we wanted something that was a little bit more phone friendly and discourse actually had to be okay wait wait we're rebooting into raw hide wait for another 30 seconds here oh yeah well until it reboots I can't figure out where I went and looked at the savings down registration desk check the next door at least it's on the premises okay and there we go yay there we go 31 my wife raw hide reporting is 31 indeed how well does this work with regard to the hour hand testing testing we'll be back in the afternoon going out at like 7 yeah welcome everybody to our next talk if any of you here were we're here for the panel after lunch there were a bunch of questions about storage and so we flew in probably the singular expert on storage in kubernetes solidly who's the are you the chair of six storage right your chair of six storage co-creator of the container storage interface which he is going to tell you all about so take it away right thank you very much Josh so like Josh said my name is Saad I am co-author of CSI and co-lead of SIG storage for kubernetes and I work at Google at the moment I want to get to know you guys first a little bit what is your familiarity with kubernetes how many of you know what kubernetes is how many of you consider yourself app developers to deploy applications that you deploy on kubernetes cluster developers who manage kubernetes clusters okay and storage vendors who are interested in connecting to a kubernetes cluster okay it's a little bit of a mix of everything um so today what I want to talk about is the container storage interface the reason that you would want to pay attention to this talk is if you fall into one of those three