e there's too many microservices because it's very easy to go to extremes. So you can have too many microservices. How do you manage all of them? It's, I see this. Yeah, I guess the true question is how you, I have a question about testing it. So you can have a bunch of microservices and they all kind of- I can't hear you, could you please hold on? Oh, sorry. All right, so my question is that how do you test microservices that are overly decoupled? That are over? Overly decoupled? Overly decoupled, I see. So from my experience, the testing is in phases. So the first thing that I do when I build a microservice is write unit tests. So that makes sure I'm making sure the code that I have written is doing what it is intended to do. That's the first step. So I write unit tests. The second thing I do is can I spin up this microservice locally and then run tests against it?  So when you start the container, you get a REST API. So then I test it. That's the second step. The third step is I deploy the whole application together and test it again. When you test it again, do you, so for the tests, the unit tests, are you combining, are you grouping it based on that microservices or do you like, let's say you have three, okay? You have the identification service, like a database and application microservice, right? Together as a whole, do you test them for the test to test the whole? Do you maintain it in a separate package or do you kind of have each microservice kind of being aware of like a central testing thing? So the question is like, so when we have these microservices, how do we test that? Like do we test the same microservice or do we test it along with other stuff, right? So like, so the next step that we were doing in my experience is once we deploy the whole microservice, the first step in that would be, I would test the specific functionality that I'm expecting out of this microservice. Then I would test the whole application together, like usually we have test suites, we have automation suites. So we just run through them and make sure the rest of the application is working fine. So when you have a new microservice, it's hard to come up with this automation test suite because you don't know the functionality yet. So at that time it would be more manual. Sure. From your experience in terms of you're deploying microservices and also containers, it doesn't make sense for you for each microservice, say for each image to have its own specific Docker registry endpoints or would you actually try to do different like images in the one registry or each microservice on its own registry endpoints? So when you say registry, you mean like the repository and the tag? The repository. Okay, okay. So we prefer to keep them separate. Most of the microservices that we have, we keep them separate. The way we optimize the pool time, like you saw we are running 10 different microservices. So if we have 10 different images, when we were bringing up all of them, it was taking a long time. So the way we optimized it was we have one base image which has all the needed stuff in the base one and then each microservice, whatever is specific to that, we added as like a couple of other layers. So we start with one base image and then kind of extend it. So thank you very much. We're actually gonna finish up now. You can ask the question in person, like, because we're gonna finish up now and take a break for lunch and we will be back in this room at 1.15 for patterns and anti-patterns in Docker image lifecycle. So thank you. Thank you. Thanks, everybody. One, one, one, this is loud. All right. Yes. Okay, well, I guess it doesn't matter then. Howdy, everybody. I hope you had a good lunch. How many people here went out for tacos? Tacos? There we go. How many people went out for lunch? Same people. Same people. Yeah. Sushi, sushi. Oh, there we go. Awesome. Okay. Anyway, our next speaker is Baruch, who is a developer advocate for JFrog and he's going to tell us all about how to screw up our Docker images and shut everything down. Big time. Yes. Take it away, Baruch. Thank you. Hello, everybody. Thank you for coming. That's my first time at scale and it looks like a great show. So thank you for having me, the organizers. I know why it makes noise. I hope it's not me. Yeah. Okay, now it's better. And today we're going to talk about patterns and anti-patterns in Docker image lifecycle. So as I was already introduced, this is me, Baruch Sadogurski, developer advocate at JFrog. This is how I usually describe myself. It's a little bit too early, but I guess soon enough I will get to the first part of the statement. Show notes. As of now, you can go to jeffrog.com slash show notes and under scale 15 you'll find the slides and all the links, reference material for this talk, feedback form and a small raffle just to say thank you for being here. The video will be uploaded, I guess, later today. Somewhere between this talk and the part when I am already drunk. Someone in between it will upload the video. So come back later if you want to use it as a reference or something. So, yeah, I'm with JFrog. We sponsor this great event. So tomorrow we will have a booth in the expo hall where we will give away those amazing t-shirts. So, please don't forget to stop by and also hear about some software that we do which is Artifactory, Bintray, Mission Control X-ray. I won't get, of course, into details on any of those. I will use Artifactory as an example here for the stuff that I speak about for docker images for obvious reasons. But other than that, please stop by our booth to hear more about JFrog and its products. And now it's the poll time.  And show your hands, and when you vote, you keep your hand up until the option is not relevant for you anymore, right? Who heard about docker? Just for the reference, all the hands, all the hands? Yeah, all the hands are up. Who can do the tutorial? Tried or think they can do the tutorial? Well, a couple of hands were down, but pretty much all the hands are still up. Who did some proof of concept for work, played with it for work? All the hands are up, almost. And now, who has it in production? Yeah, take it down, that's okay, that's okay. You can take your head down now. Okay, so we left with, I guess, in what, like 20% of what we started with and of course it wasn't real poor, it was just manipulation. I knew exactly what was going to happen because it happens every time, right? And the real question is why? There are a lot of reasons, of course, why people hesitate to take this amazing technology to production and I'm going to speak about one of them and this is the lack of trust. And it's natural because a docker is kind of another layer of very obscure wrapper around what we already do. So when we have docker-containing running in production, how sure you are that what's run in it is what you intended for it to run. There is some process, continuous integration, continuous delivery of the docker image that we create the container out of it and the question is, is there enough integrity into this process so you can be absolutely sure that what you have in production is what you intended to have there when the developer first crafted the docker file, right? So we're going to talk about that. Any employees of docker-ink here? Anyone? Okay, I can run my jokes. Okay, whatever you see here is a hug. Of course, that's frog hugging the whale and nothing else and that's because we really love docker and that answers the question why I'm here speaking about the docker stuff and that's just because of very long history with it. We in JFrog do binary management or binary files for more than 10 years and when docker first come out with their layers and images and all this stuff, we of course were very interested and we actually implemented the first docker registry before it was docker registry, right? So we know at least this part of docker pretty well and this is why I'm speaking with you now. Another interesting aspect of what we do compared to what docker does, this is of course, Mr. Solomon Hikes, the creator of docker is speaking at the keynote at dockercon 2015, two and a half years already by now in San Francisco asking this question, who is using docker and nothing else? And the question is, okay, now you have a bunch of technology in your company, is there someone that the only technology they use in their company is docker? Anyone? No. Thousands of people in dockercon the same answer exactly and that's because the analogy of containers works here as well. No one, no company in the world are in business for shipping empty containers. It just doesn't make any sense. It's the same as here. We always have something in docker inside the container. We have Java, we have Node.js, we have .NET, we have applications that we ship in docker containers but there is other technology inside.  So let's now talk about patterns and 90 patterns in managing docker images. We are all software engineers and this is how we operate. When we have a new technology, first we look for a pattern that we already know and we try to apply it and sometimes we also think about are there any changes that we need to do? But usually we just try to apply it, right? And continuous integration delivery, those are all news. We do those pipelines for years, if not for decades. Next couple of slides are a very brief recap and I really hope I won't tell anything here to any of you. So this is a promotion pyramid and here what you can see is in builds that are passing through tests. At first we have a lot of builds and very fast tests, unit tests. Next we have less builds because some of them failed but longer tests integration. And this is how it goes. In some stage we have manual QA which is like long process but we have very little amount of builds to test and eventually we have couple of them that go to production and everything is great. Same picture, just put it on the side, is this. I really love this diagram. It's for a great book called Agile M by Michael Hutterman and it describes exactly the same. You have here binaries that are goal from one repository to another, that's your promotion, by passing what are called the quality gates. Quality gates are very important because they guarantee that only the right quality of builds reside in specific area in your pipeline which means in specific repository in your pipeline. And we promote here the binaries that we build from repository to repository all the way from development to production. Again, I really hope this is no news for you. Now let's talk about what's happening in Docker and why Docker forces you to do the wrong thing and stands in a way when you try to do the right thing. So this is why. Docker build, as you all know, is extremely powerful thing. Docker file as a build descriptor is more powerful than any build descriptor that we knew previously. We can do everything in there. We can type in any command and then we just run and we get an operational image that we can instantiate a Docker container out of it. That's absolutely awesome stuff. The problem is that once we have this great hammer in our hands, everything look like nails. So instead, that's almost the same picture. That's almost the same picture, but look at the difference. Instead of promoting binaries now, we trend to promote the build file. Promoting a text file, comparing to promoting binaries is very easy. All you need to do is re-tag it differently in your version control, or maybe just move it to another branch, or even change the name. It's very easy to deal with one source file. And then all you need to do is you promote this source file to another stage, from dev to integration, from integration to staging. And what do you do in each stage? You run Docker build, and it works. Everything's great, well, not so much. That's a home that was built and then felt. Because fast and cheap builds are not always the way to go. What do I mean by that? Why, where is the problem? The problem is that. Now, when I prepared this slide, I wanted to craft something ridiculously wrong. I said, I will do something that no reasonable man will do. And then I will explain that, you know, it's just to make my point. And then I went to the internet. And I didn't need to craft anything. In the show notes, there is a link to this file published on GitHub, extremely popular, tons of stars, tons of clones, and this stuff even is full of that. Now, you understand what the problem is, right? If I run this build file with Docker build more than once, chances are, I will end up with different image, right? So if I run this one in development, I will get one result. When I run it in integration, staging production, I might end up with different results. And that's exactly the root of the problem. This is why I am not sure. Do I have the same in production that I intended in development? Ah, I'm not sure. And now all of you guys are, there is a good number of people here that have their hand up when I asked about Docker in production. And this is where you should tell me, come on, Baruch.   We can actually do better. We can fix that, right? How do we fix that? How do we fix that? Version. Version, thank you very much. Let's fix it. Here we go. Let's start from the first statement, from from. And now we fixed it. Now it's Ubuntu 14.04, and we know exactly what it refers to. Do we? Why not? Thank you very much. Now, here is the thing. Ubuntu 14.04 was released when? 14.04. Apple 14, right? It was three years ago, and since then something very bad happened. Remember? What happened? How did it, thank you very much. How did it happen between 14.04 and now? And here is Canonical, who are great company sponsors of this event. Anyone from Canonical? Here? No? Okay, there are some here. Well, this is annoying. There are some of you, and they had a dilemma. Their dilemma is, from one side release versions, like 14.04, means it's an ironclad, immutable version. This is what release version means. It means that every time in the future, when we try to take this 14.04, we will get the same file. But, many of you guys come from the Ops background. Who comes from the Ops background? Okay, half of you, that's great. And what do you hate?  Why you hate upgrading servers? Because things break. And because they screw with your uptime. Right? We don't like that. So, here is the dilemma. From one side, we cannot let people stay on the same version that was released on April 14. On the other side, there is no way to convince people to upgrade to the next Ubuntu version. So what Canonical did, and that's a perfectly good choice. This is the right thing to do. They incorporate severe security updates into existing version. Right? Any Windows users or maintainers here? Couple of hands. All right, you don't need to be shy, that's fine. But you definitely experienced Windows update that ended up in blue screen of death? Right? Exactly. Here we go, and I will raise my hand. Not lately, but there were times. What it means is, we can definitely can get an update. Incorporated into this, not immutable, but mutable version, and still end up with a different docker image when you run docker build multiple times on this docker file. Agreed? Can we fix that? How can we fix that? We will get to that, don't spoil my talk. Use a shot, thank you very much. I didn't pay this gentleman, but I should. Here we go. Now, this is great. This is ironclad. SHA2 refers to a set of bytes in the file. Every time I use that, I know for sure that I get the same version of Ubuntu. By the way, which version is that? I just sat on the keyboard when I did that. It's not a real SHA. Right? This is great, but completely useless. Okay, now what about those? Yes. You want more? All right, yeah. Well, let's say I have update got, but that's a good point. For the next version, I will do an update, but okay, let's say I do the update. No, I don't, I can fix it. How do I fix it? Easy, come on, guys, huh? Version numbers, of course. Of course, I can refer to Python with certain version and Node.js for certain version, and that will work. That's great. How do we know that we can put a versions there? Because we know how app get works. This is scale conference, of course we do. How many of you have Java background knowledge? How many of you know Maven? Maven? Okay, a couple of hands, good, good, good, good. Most of you don't. So how about now? What do you say? Is it reproducible? Every time I will run that, will it get the same results? You have no idea. The guys that know Java, what do you say? Will it produce the same results? Well, it depends on the form. Not only depends on the form, it depends on all the transitive forms that this form uses, which are, of course, outside of our control, and basically the answer is we have no idea. If you know Maven really well, like unfortunately I do, I know exactly how to mail the versions. Well, I know how to fill the build if there are some kind of non-final versions. But that's knowledge I don't wish anyone to gain because there is no normal way to gain it, but through extreme pain. So you don't know, and that's fine. How about that? Well, you got my point. There is no way we can escape that when we build multiple time the same build file. There is no way. Every time we run a Docker promotion, this is our nightmare. We cannot be sure that this container and this container are the same. And this is why we don't trust Docker. So what we should do? We should look at another pattern. Inputable server pattern? Anyone? Okay, not enough. So here you go, useful talk. You will learn immutable server pattern. Mr. Martin Fowler, the one and only, came with a great name for something we know how to do for many years. So if you remember when all of us were younger and the servers were actually hardware, if we needed to reconfigure a server, we actually logged into it like a physical machine. It was a box. Not many of you remember, but it was like a box. And we actually logged into it and made changes, right? And it made sense because this box, it has a value X thousands of dollars. So, you know, if we need to be reconfigured, it will be reconfigured. But those days are on gun. Now our servers are virtual, which means they don't cost us any money. And if we want to reconfigure a server, what do we do? We throw it away and create a new one. And that's because managing state is painful. There is absolutely no way to reconfigure a hundred servers in exactly the same way. You will end up screwing somewhere. But now our servers are code, infrastructure, right? Infrastructure is code. So we code this configuration and then we instantiate as many servers as we like. If the requirements changed and now we need to reconfigure, we change the code that configures the servers, new old ones, create new ones, and new ones are all the same in exactly the same way. This is the immutable server pattern. We never change the existing server. Instead, we kill it and create a new one. So this notion of immutability comes from a lot of different sides in computer science today, right? A lot of languages put a great emphasis on immutability, especially in functional ones. Immutability is good. And we should take this immutability to our CI CD pipeline as well, right? Instead of rebuilding this image on every given stage, although doger file kind of invites you to do it, you need to release this urge and instead promote a file that will, binary file that will build only once. So that's exactly the quality gates image from Michael's Wutterman book. You build the image once and then you promote your image through different quality gates that are right here from development to staging to production, right? And now the gates. I keep talking about the quality gates and you might say like, what is this problem with gates? Well, I do have a good reason. And those gates are very important because those gates give us the guarantee that a right image won't get to the wrong place. They guarantee that the QA won't waste their time testing what they shouldn't test or that will stage something that is not, shouldn't be staged. And of course, that we won't end up in production with images that are not ready to be in production, in terms of quality. This is why there are quality gates. So they are extremely important and picking a tool that knows how to provide you with the most in terms of quality gates is important. And this is where we go to the part when we now know what not to do. Let's talk about what we can do and why it's not that easy with Docker. So as I mentioned promotion, the hardest quality gates we can get are separate Docker registries, right? Because they have unique URLs. And if our provisioning tools only see registry that it allows to take the artifacts from, we are guaranteed that there won't be any glitches in terms of our quality gates. So what we actually want is implement a continuous integration pipeline when every stage is a different Docker registry. Because, you know, let's take a moment to speak about, I just saw just a couple of skeptical faces here. And this is good, you know, your Docker. So let's talk about what are the other options. We can use different tags. Can we set permissions on tags? No. We can use different repositories in Docker terminology. Docker repositories are directories inside your registry. Can we set permissions on repositories? We can. But how do you manage repositories? We create new set of repositories, development staging production for every image. And then for every image, we need to go through those repositories and set up new permissions. What are the changes that we won't forget to do that? Not strong enough. When we are talking about strong quality gates. So having isolated space, isolated registry, per stage of your pipeline is the hard guarantee. Did I convince you? Did I convince you? Good. All right. Now let's talk about why we cannot do that. Or it's not easy to do. You only remember that, right? I mean, it's fun to make fun of Bill Gates and Linux conference. What can be better than that? We have something similar in Docker. Our Trump implementation in Docker is Docker tag. Now, as you all know, Docker tag includes the registry host inside the name of the tag. It's kind of innovative decision. It has pros and cons. The biggest advantage of doing that is that it's very clear where this image belongs to. If we just see Ubuntu as a tag name, we know it's the official real deal. If you see here JFrog slash Ubuntu, you know that it's our flavor of Ubuntu or whatever reason we have our own image. Right? But you definitely know that that's not the official one. So it's very in your face where this comes from. And that's a good thing. The downside of having the registry host in this format, in this syntax, in the tag is the slash that they use as a separator between different types of, between different parts of tag. And the problem is that is, well, slash is already taken by the URL format and standard. So when you see that, you realize that how do you implement multiple registers per host? If you have host and then the tag name, how do you do that? So this is what we would do with any other technology inside your artifact repository. You will maintain more than one repository or registry. So here, that's one for development. That's one for QA, one for staging and one for production. But you cannot do, you cannot express that with the syntax of Docker tag. You cannot, because after the host, what need to come directly after the slash is the tag name. So what do you do? That's our trump up limitation. Well, there is a workaround. It's a heavy, it's not a very pretty one, but that's the only way you can do it. And interestingly enough, this standard of Docker tag being there for four years now since Docker first emerged, but the understanding of we need multiple registers per host hit Docker guys much later. And if now you will ask their support of how to do that, they will point you in exactly that direction that we are going to show now. Because there is no magic here. There is only one reasonable way to do it. That. No. All right, so let's talk about what we need to achieve. This is the tag. We have host port and then slash the name of the image. This is how the request gets out of the Docker demon to the registry. Right, so it's pretty much the same. We have the host port and then the protocol of Docker, current protocol of Docker, and then the name of the tag. What we want to get is from here to here, because that's the real URL that we want to achieve. And it has, of course, the host, the port, the context name, and then the virtual, the repository name that the repository that we want to get. And then eventually the tag name. How do we do that? It's right here. So not very intriguing question. We can do that with virtual hosts and virtual ports. There are a lot of tools that allow you to do that, and basically any HTTP server. That's an example of NGNX, and Apache HTTPD is another option, HAProxy, I think there are a bunch of others, but they all can do the same. They can listen to a request that comes to a virtual port or virtual host. That's example of virtual port, 5001, and then they can say whatever comes to here redirected to here. Right? And we use here this port as a unique identifier for this repository. So Docker staging, for example, will have another port, 5002, and we will have the same block for the new repository. And then for production 5003, et cetera, et cetera. Right? So, and that's the way to do it. It's not pretty, it demands babysitting. We do our best to make it easier for you. We have generator for all this stuff that actually updates NGNX automatically. So it's kind of behind the scenes.  But I don't think there is another way around it. Now there is another problem. Since we have the host and the port inside the tag, the question is how we promote the Docker image from one repository to another? How we take it from localhost 5001 and get it to localhost 5002? We know that behind the scenes, it's the same server because all these virtual ports is just solving the problem of the tag. But from Docker's perspective, those are different hosts. And that means that we need to pull the image, retag it locally, and then push it back. Which is not fun. Especially considering the size of Docker images. They are huge, right? So what we actually need is some way to minimize all these dancing around different registries, but still keep the quality gates up. So our solution to that problem is what we call, and again, this is example of artifactory, obviously, but there are other tools that give you the same, maybe in different names, different terminology, but the idea is the same. Minimize the interfaces between your repository manager and Docker and pretend there is only one registry. When actually underneath, keep as many registries as you need to build continuous integration pipeline that respects quality gates. So this is what we did, right? We have a developer here that does Docker push to one URL. That does Docker push to one URL. This URL is what we call a virtual repository. There is no local storage of files there, and all the files are actually backed up by a local repository, first in our continuous integration pipeline. In our example, this is Docker Dev Local. Now, all those are fully pledged Docker registries in terms of the APIs as they expose, and each one of them use different virtual host or different port, and tools that are in charge of doing the testing can pull the images directly from each endeavor stage by using the virtual host and port. And of course, they only see their registry. This is our quality gates. They don't know about the existing of any other. But when eventually this image is promoted all the way to production, it can be again retrieved by using this public virtual repository, which looks like the only registry for the rest of the world. Does it make sense? I hope it does. Now, just an example of building one of those, and how you implement the pipeline on top of what we spoke about. So let's talk about a typical container or typical image, a typical stack, and you will do the adjustments for your stack, of course, appropriately. So this is our layers. We will use here CentOS as example, and our framework that we run, our code is Java EE, just to show you that even in Java EE, we can do good job, right? So we have JDK8 and JBoss application server, Wildfly, whatever. And then we have some raw file on top of it that actually runs in this application server. Those are the layers of our Docker image. Now, we can separate it to two pieces based on how frequently they change. We have those two, which are the framework. They change very rarely, only when we have a new version of CentOS that we want to upgrade. As I mentioned, we don't, if we have a new version of Java that we want to upgrade or a new version of application server. And we have our raw file, which is constantly in motion because this is what developers write. So those are the two, and we need two different builds for them. One will be the framework build and the other will be the application build. Framework build used a verified base image, the one that we played with, and we are sure that it's good, some version of operating system that we can trust or even a base image that includes Java and maybe includes this application server. And then we can add additional system dependencies from the source that we trust. And it should be in house repository that gives you those dependencies. So your up-get server source or your RPM source or whatever. Now, for example, all those can be implemented in artifactory, of course. Stuff like JDK, like Tomcat, you install in this image. And the most important part is you own it and that's back to the answer that the gentleman gave me. You own the base image. Once you own the base image, the problem is how can I guarantee that my image won't change under my hands? Of course, go away. This is a minimal framework build Docker file. It does absolutely nothing. It just nails down the version of CentOS and guarantees that until I build this again, I don't have a base image that I can trust and it won't change without my permission or without my action for this regard. Application build. In our example, it's a Java build that we use our framework as our base and then we run this Java build and we end up with a world file. And again, it can be anything, right? It can be Python build, it can be a node build. It can be whatever your software is and then you end up with the file that you need to end to your base, to add to your framework, to deploy to your framework. Right, so this is our example and here we use our own trusted base and we add one file. We take the world file from our factory and we deploy it to wherever place, oh, this is annoying, to wherever place the deployments go in your framework.  First, why the hell do you use a release repository for your file? This is an interesting question that can be answered differently depending on your process. And there are two types of processes. Sometimes you check your wherever goes in the container always with the container. And then you have matching steps. When your application is in dev, your container will be in dev. When your application is staging, your container will be staging, et cetera, et cetera. But sometimes you can run a continuous integration pipeline only for the contents of your Docker container without actually bringing the container all along. So if you have a continuous integration pipeline that brings this world file separately from development to release, you can take it from release. It will save you additional testing. And now you should go like, well, you just told us not to rely on the latest version and this is what you do. The difference is, this is my latest version. It is guaranteed to be good because it is in release repository. And we just spoke about how strong our quality gates are and everything that is in release repository is actually good. So I can take the latest one with confidence.  And the answer is for the same reason. This is verified base image. I built it. I want the latest because I know that it's good. There won't be any magic in the world that will make the latest framework bad. It might be broken. That can happen. And we are going to talk about how we can we find this problem very soon, but it's definitely not malicious. It comes from a verified source and I can definitely rely on the latest framework. And now let's talk about testing. Who heard about sandwich testing? This is a very useful talk. Here you learn something new every slide. Sandwich testing is a flavor of integration testing when you actually do bottom up and top bottom testing in the same time. This applies perfectly to our two builds. This is our two pipelines. This is the framework and this is application. And when you want to test your application, you take the latest good known framework to run your application on. That guarantees that you only test the application without testing the framework. And the other way around, when you want to test your new framework because you updated your Java version or your base image, you test it against the latest known good application. And if it works, that means that you didn't break anything. What can be the problem? That. When actually the changes are mutually breaking and you did some change that breaks in application that breaks it and some change in framework that breaks it as well. And what you can do to prevent that is setting up a good triggering system for your continuous integration. For example, you make changes separately for application and for framework. And by the time the CI server kicks in, it will only run on what's changed. So if you change the application, you will end up with good known latest application build before you change the framework. And if you change the framework, the framework build will end up and test it against the latest known application. So as long as you don't have non-backwards compatible, dependent breaking changes in both frameworks, you're good. And you shouldn't do that anyhow. So this is all. And what I want to end up is the reason why we actually need that. We spoke about trust. We spoke about this feeling that can I be sure that what I have in production is what I have in development? And how can we live with that? Except of raising the level of stress, it also slows us down. Because when we are not sure, we will add more tests. Those tests will probably be ad hoc, will probably be manual, will probably take a lot of time. When we build this trust, it allows us to go faster. It allows us to release faster. It allows us to do our work better because you can go faster when you are not afraid. Now it's time for Q&A. We have good five minutes at least. I'm at J Baloch on Twitter. That's good time to follow me. Scale 15x, that's the hashtag. Use that. Show notes, video by tomorrow or by today even. Slides, links, feedback, and raffle should come early. You can double or triple your chances for coming early. Thank you very much. Yeah, questions. You should have tons of questions by now. Like for example, what the hell did I just see? That's a good question. I'm not sure I can answer it, but it's a good question. Here you go. That promotion model that you showed? Yep. Is that only in Artifactory? No, no. So generally what we spoke today is not some kind of unique feature in Artifactory. If you want to do it, you can just set up multiple Docker trusted registries in your host and do that. It will be more painful, but you can do it. There are other alternatives, competitors of ours, that provide kind of similar functionality. I can now explain you why we do it better, but the more appropriate place for that is tomorrow at the booth. So for the promotion, why don't we use the Docker manifest retail scheme material, multiple tagging of the image to do the promotion? Oh, that's a great question. Now, Docker have a very neat feature which is different layers of metadata. One of the metadata, and then there is not the only one, is the ability to add properties to Docker image. And then we can say, you know what, screw that, I don't need that. All I need to do is annotate those images with their current state, right? I can say this image now is in dev, this image now is in staging, and this image now in the production, which we can do, but as you probably remember, I am very much obsessed with security gates and their quality. When you cannot add permissions on this metadata, there is no way to guarantee that a test that should only take image which is in development, which is in staging, won't take an image which is in development. The only way to strongly guarantee it is completely different registry as long as Docker concerned. At least for now, maybe it will change. Maybe there will be some other models, but now this is the strongest way to guarantee you want and test what you shouldn't test, but more importantly, want to end up with production with something that shouldn't be in production. And that's scary.  Yeah. You could, for example, restrict what you're allowed, use some other layer to restrict, say, pushing to the Docker repo or something like that, so your only your CI host is actually able to push and tag to your Docker repository. Yes, you can add permissions on registries, on repositories, which are directories in Docker registry, and you can do it on the level of Docker itself. You can do it on level of NGX, even, right, or Apache. But the problem is that you create new set of repositories for every tag, sorry, for every image you work with. Repositories in Docker are tied to an image. Yes, then, yes, what you can do, yeah. All right, so what you can do is, for example, you know what you say, screw it. I will use one set of repositories. I will have repository for them, for staging, for production, and then all my images will go into this one repository. Right? That will work, but that's the opposite of Docker way of this concept of repository, which means you can have multiple repositories per image. I don't say there is no other way to do it. There is, it's just one, you know, you will fight Docker to achieve that even more. Yeah. Sure. You mentioned pinning the base image. You did an example with CentOS 7. What guarantees that the CentOS 7 image is not updated in the public repo? What guarantees that this CentOS 7 not updated in the public repo is, of course, nothing. But what guarantees that I won't get this update is I don't run this Docker build until I wanna run it. Thank you very much. Okay, thank you very much, guys. Thank you. Thank you for every other support aspect that we have. Yeah. So, yeah, see what I mean, comment. That's not you. Oh, okay. You just have to live with it, if it's on your body. But I'm gonna go ahead and introduce you and I'll be back in a Q&A. Would you want me to use that instead? It would really help if I turned this on, wouldn't it? Good afternoon, everybody. Our next speaker is Lee Calcote. Calcote? Yeah, that's right. Lee Calcote of SolarWinds, who's going to talk to us about container networking. So, welcome. Very good. Thank you, Josh. Excellent. It's been a while since I've been in a room of this many like-minded folks. I can see it. I've been to some other conferences and there's a variety of folks there, but I'm getting a good vibe today other than the microphone. So, yeah. So, a little bit of a play on words about the over-under, the overlay underlay on container networking. So, we're going to dig in today. Networking is a complicated thing. People make entire careers out of it. Actually, that's kind of an interesting question, maybe, is how many of you guys and gals consider yourselves to be sort of all of the network, more of a network engineer, network systems administrator, network-oriented folks? Nice. Very good. Boy, you guys have just learned to tune that out, haven't you? Okay, I'll get there. Quick note, we will no doubt get through all the slides today, but to the extent that I skip over something or there's something you guys want to look back up, take note of the URL, the slides are there. A little bit about me. Feel free to harass me online if you don't do it today or tomorrow the next day. Reach out, poke me. I like to chat, nerd shop whenever I get a chance. I'm into a couple of things. By the way, I'm coming to you guys from Austin, Texas as of this morning. It was an early rise for me. Word of advice. I've just got some issue with time zone calculation and then the 24 hour time that's listed on the schedule. I thought I was speaking about two hours from now, so I'm nice and alert right now. Anyway, I make myself a nuisance in a couple of different areas. I'm a cloud native ambassador with the CNCF, so I spend a lot of time hanging out with in that general area. I'm an advisor to container security startup, Twistlock. Anybody know of Twistlock? They're kind of, yeah, very good. Excellent, yes, I'm sure you do, Mr. J, for God's sake, yes. Fanta, I do some writing for various places. I'm an O'Reilly ambassador. I organize a couple of meetups at Austin. So anybody coming out to DockerCon for Austin? Anybody come out to OzCon for Austin? There's a ton of great tech conferences coming there, so that and barbecue, so. We don't need to talk about my favorite subject, me anymore, but like I said, reach out. I organize a couple of local meetups, the Docker Austin one, another one that's affiliated with the CNCF, so it's on microservices and containers, and so we just cram the two buzzwords into one title. Currently, in sort of full-time, I'm with SolarWinds. I'm gonna do another raise of hands. Who's familiar with SolarWinds software? All right, so I'll be honest and say that's more hands than I've seen the last time I asked that question. That's good, and since you guys are here, you're obviously interested in containers, doing container-y things, no doubt. How many of you guys would like to see SolarWinds do, brings the monitoring software for containers?  You're looking to get off of it and run away fast? Yes, everyone, yep, okay. Nice, okay, well, like I said, so this doesn't, this goes without me saying, but container networking, well, networking in general is complicated, right? And I think I've made a friend, Justin, if you're out there, I don't know if, if container networking, if everyone agrees that container networking is a thing, I think most of us can agree that virtual networking is a thing, right? When VMs came around, and we used to just have things that we could touch and put our hands on, physical networks with cables, and we got into some VLANs, we brought in VMs, and we got virtual switches, and to use, you know, VMware parlance, what distributed V-switches, and SSX came around, and things like VXLAN got, you know, established and more popular. Anyway, there's, there's sort of virtual networking as a thing, and if it's a thing, then I'll contend that it's always container networking. Like, hey, it works a little bit different than the container land, and that's actually one of the things I wanna try to point out today is that for those of you who are familiar with physical networking, and I think you're gonna find yourself maybe leaning into different types of your, different types of choices you can make in terms of how you set up your container networks. So, we'll see. I think the other aspect of the fact that container networking is complicated is that things have only gotten worse for us in this sense, in the sense that physical networking is tough with just, you know, routing and spanning tree and layer two stuff, and just, like I said, people have got expert certifications in this, but it's gotten worse than that. We used to just have, you know, one or two physical network interfaces on a server, right, or maybe a couple of virtual sub-interfaces, and we might bond them together, or team them, or depending upon whose server you bought, you'd use a different, you know, vernacular for that. But then, you know, then we loaded up VMs on there. We got to, what, you know, 10, 15, 20 VMs on a host, and each of those VMs might have had an interface or a couple of interfaces, and so, what, you know, 15 or 20 times two, we're starting to get up into 40-ish interfaces per host, and this is just like, just interfaces, right? So, if we keep going on that count, and we say, well, now a host, that same host, might be able to hold 50, maybe 100 containers, just depends on, you know, how large your containers are. If you consider that they've got an interface, or maybe a couple of interfaces per container, if you take 100 containers times two, anyway, we're just getting into, it's just getting worse, right? And then we bring in microservices, and we make the network front and center to how an application functions. As if that wasn't already true with, you know, enterprise-architected applications, the, you know, your classic three-tier apps, like, hey, networking was fairly front and center to how the health of those applications, but as you move into microservices, and you do things in a distributed way, you guys, some of you've heard of CAP Theorem, and anyway, the point being is like, a lot of times for a given transaction, for a given function behavior of your application to execute, that, you know, microservices land in a distributed systems land that doesn't necessarily always happen on a singular host. A lot of times that's happening, you know, at execution across hosts. Anyway, point being is, not only has it gotten more complicated, but network has got a little more front and center. So for those of you that raised your hands earlier, I sort of identifying as being a network nerd, and I use that in the best way.  In the case is, as people move into this area of networking and using containers, there's some preset expectations. People already have an understanding of how networking works, how virtual networking works. They're probably gonna go in expecting the same amount of, particularly if they're gonna do things in production, they're certainly gonna want these to be, the networking there to be reliable, to be performant. They're gonna have a lot of the same expectations that they've had in the past. And they're gonna want that experience, that performance, that reliability to be on par with what they've been using today. Moreover, they're probably gonna inch up, or begin to expect a little bit more as they move into this more, you know, SDD, this more software-defined networking, software-defined data centerware. Now you can just express things as code and you define your infrastructure as such. You begin to get into, you know, another excellent, I don't know if this is necessarily a buzzword, but if it isn't yet, I suspect it'll hit you guys at some point, what is intent-based networking? It's, you know, in a nutshell, it's a lot like being declarative with the network. So to the extent that you've seen a Docker compose file or Kubernetes pod specification and you're declaring your infrastructure state, that that type of a, you know, that essentially is what intent-based networking is. There's some declarativeness to it. There's some reconciliation loop type of a concept to what intent-based networking is. It's the notion that you can go and define that in code. Let SDN controllers go and go and affect and implement the network as you've described it. So anyway, point being is not only do people expect to network to continue to function as it has in the past, they're gonna have the same expectations, they're likely to expect more. They're gonna want the ability to define how your network works in a developer-friendly way, in an application-oriented way. However you guys have heard of mode two and mode one, kind of Gartner's bimodal IT definition. Okay, good. So this is not, that goes back to I think what I said earlier which is there's some like-minded folks in here. This is not the Gartner crowd. So we won't go there. This probably used the crowd hopefully that where this image resonates with you, this is probably my favorite image of the deck. So we'll just share that. There are a couple of container network specifications. We're gonna walk through the, really the two that are there. And I just wanna disclaim this by saying it's some very interesting kind of thought-provoking things in here, some models that we can walk through and then as it goes to application, unless you're actually implementing a network driver as a plug-in to one of the container orchestrators, it's probably not necessarily a key that you digest all of this but I do think it's pertinent background for understanding really the, how container networking has been shaped and it's kind of between these two models. The one on the left-hand side is the container network model at C&M. It's really not necessarily a specification as much as it is an object model for how it is that Docker, the project has defined how networking functions within the Docker game and within the Docker environment. There's a singular canonical implementation of this model. It's called libnetwork. There are various vendors of written plugins for or vendors or projects have written plugins to libnetwork. We may touch on some of those today. The other specification, this is probably really is a specification is the container network interface. So specification proposed by CoreOS. It's been, the specification's been implemented a few times over and there's a list up there. Not only has that spec been implemented a few times over, but so have some plugins been written, that plugin to C&I. So the container network model, Docker's libnetwork. What does that model look like? They're both relatively simple in terms of how you diagram them out. In essence, what this describes is the hexagram, I think, if I've got my geometry right, is your Docker daemon, your runtime. The libnetwork or this container network model is really just an API specification that sits between the Docker daemon, the Docker engine, and network drivers. Out of the box, Docker daemon comes with three network drivers. That said, actually, I think that that's true as of Docker 111. I think with Docker 113, I believe it was, there's a fourth network driver. We're gonna talk about that one. Anyway, the point being is you get some, you get various types of container networking out of the box. Those are fully supported by Docker. Of the vendors that I've listed out before that have written those plugins, those are the remote drivers that plug into this. If we look at the actual topology of the container network model, we're seeing a few things. We're seeing that the model allows for more than one container to be connected to the same network. I think that's probably a fairly obvious thing already. One network driver, it really is responsible for managing a singular network, so one network can't be managed by multiple drivers. Multiple drivers can be running concurrently, so you could be running bridge networking on one network controlled by one driver. You could be running host networking on a different network controlled by a different network driver, so there's separation of concerns there. There's a lot of, some of you might have heard the term micro segmentation with respect to network, so there's some fairly granular ways in which you can interconnect these containers in this model. One of the things maybe to note is that even though the container in the middle is conceptually, I should say, the container on one of the ends, even though it's conceptually interconnected all the way to the container on the other side, by default it can't necessarily use that middle container as a bridge, so there is that micro segmentation. The CNI, the container network interface, the one that's more in Kubernetes land and was proposed by CoreOS, this is what that looks like. It is simple as well and it has a lot of the same characteristics that we just saw for CNM. So the flow of how it is that when you go to provision a container and it spins up and that container engine needs to assign it in a network and needs to assign it an IP address, this is the process that happens, so the container engine allocates, puts that container into a network namespace, it ends up passing along the config information to the network driver. The network driver then picks up that request, ends up attaching the container to a network as well as then assigning an IP address. So the network driver, the network plugin, not only is it responsible for the network and the services that are there but also, I guess I should say, that includes services like IPAM, IP address manager. This is just a quick JSON schema for what a network looks like as you describe it in CNI. So really straightforward. Both of these models are similar in that they're both driver based so we saw the plugins along the bottom there. That democratizes some of the selection of what types of container network you wanna run. Like I said, you can run multiple concurrently. Containers, like I said before, can join one or more networks. And the nice thing about the way that these models are written is that some of the application logic or the business logic as to how those networks are managed, that's dedicated to the network driver so vendors can come in and differentiate a little bit. These two are different in that. CNM really only supports the Docker daemon, the Docker engine. CNI supports the engines that I'd listed up there before, Docker, Rocket, Kerma. So I'd said that CNI is a little more of a specification because it's been adopted beyond its creator. All right, so good. So let's get into, I guess, some of the basics of how these different types of networks function. It's unfortunate that the screen is cut off there. But I wanna walk you through each of these types. I wanna start off with none and see if I can't give you guys quick examples as well. Okay, so as you go to, and we're gonna use Docker, the Docker CLI and the Docker engine to walk through this.  Okay, so we're talking about what's the behavior when you go to spin up a container and you tell it that it has a network type of none. Any guesses? Let me see if I would get this correct, I'll go around. Thank you. Turns out you actually do get some networking. You get local host. You just don't get external connectivity. So the container when you spin it up, it gets assigned to a network that doesn't have a driver. Or rather, I'm sorry, it does have a driver. The driver is called none or null. It does get a full network stack. It just gets loopback connectivity. Could be potentially useful for maybe batch jobs that don't need connectivity to the network. Maybe it's just writing things to the local file system when it's done. You can spin up a container on a network like this that isn't connected. And while it's still running, come back to it and take it and attach it to other networks. So just because it starts out with nothing doesn't mean you can't come back and reassign it to networks. So speaking of that, may be good to just do a quick review. So in terms of just how Docker works, I'd said before that out of the box you get three types of networks. Bridge, host, and null. Null is what we just looked at. Okay. So another type of container networking is links. And links were often used with the ambassador pattern. Really, this is a type of container tech networking that has come and gone. This was, I think if Jerome were here, I think he would agree this is probably a poor man's attempt at facilitating service discovery before there was kind of full-born multi-host networking inside of Docker. And so as such, we'll skip past it because it's just not necessarily that interesting. I mean, it's interesting to note that even this space has evolved just in the short time that it's been around. So another type of networking is bridge networking. And this is probably the one that most of you are familiar with. So if you go to spin up a Docker container and you don't specify the type of networking, this is the default network that you'll get. A lot of you have seen maybe Docker zero. So if we go out and just take a, do a quick run of that. Okay. So we see in this case, we do get the loop back zero. We also get E zero. It's attached to a Linux bridge, the bridge that we see out there. How many is it? How many of you is this familiar? This Docker zero bridge, the Linux bridge? So what's happening here is netting, right? And the use of IP tables. There's some, even though there's some cost to how Linux bridges work, there's some cost to the translation of that. We're gonna see if I can't show you guys some of that. Another type of container networking is container mapped. So this is, this type of networking is used very frequently in, it's essentially the default type of networking that gets used within Rocket and within Kubernetes pods. In Docker land, this doesn't necessarily get used that much. But let's take a look at some of its behaviors. So the, I guess I should explain what it is. It's container mapped is when you've got a container up and running and it has networking, maybe you're wanting to spin up a second container, have it attached to the same network name space as that first container. This is what you can use container map to do. So maybe you're having an issue with this container that's running. You're wanting to diagnose what that issue is. Maybe it doesn't have the right tools. So you spin up a separate container, drop it into that first container's network name space, and they both share the same IP address, the same MAC address. So let's see if we can show that. So if we start up a container, or just maybe this is what you were saying.  Okay, so we just spun up a container. Take a look, we're gonna have some fun with the formatting here I suspect. Here's our container. If we just take a look at what's in this container, so we didn't explicitly tell it what type of network to use, so it chose the default, the fridge networking. If we'd spin up another container, and in this case tell it to be container mapped or to use that type of networking, the syntax looks like this. So we're just saying the network type is, use the network of another container, and here's the ID of that other container. So normally as we would spin up additional containers on that same bridge, we would get an E0 and it'd get 172, 17, 0, 3, 0, 4, 0, 5 as we just kept spinning up containers on that network. If this container is gonna drop in to the network namespace of that first one, we should see that it really has the same MAC address, I'm sorry, same IP address and even same MAC address. So I think that's what we're seeing. Kind of interesting. So the two are actually sharing the same network namespace, kind of an interesting thing. Okay, host, there's host networking. So that's one of the three network types that supported out of the box with Docker Damage. And there's some pros and cons to using this. This essentially says that the container that you spin up is dropped into your node, your host's physical network namespace and it begins to share that same network namespace. So for some of the costs that you would have with NAT or IP tables, you'll avoid some of that because that container will be using the same IP address, same MAC address as your host. You'll get some better performance for, I think for many of us that either scratch our heads around network or are accustomed to physical networking, you may find this type of networking a lot easier. There are some question marks around whether or not it's more or less secure. Another thing you kind of suffer from here is potential port conflicts with the host. Since you're using the same IP address, you'll be sharing the same port range with the host. So let's take another quick look. Docker run. This time it's network equals host, alpine. And so what's happened here, so okay. Let me describe this a little bit. What's happened here is the, okay. So I'm running, I have a Mac. I'm running Docker for Mac. Docker for Mac uses a virtual machine running an XIV. The name of that virtual machine is called MOBI. That's just its host name. So if I were to go into that VM and take a look at that VM's networking. So by the way, so in order to get into that VM, what I did here was just spin up a privileged container and now that that privileged container has access to that VM. So if we type host name, we're actually in that VM and that MOBI VM. We take a look at our container hosts networking. We're seeing Docker zero, E zero, loop back zero. As we look at that container that we spun up and that we told to use host based networking, it in fact is sharing the same Docker zero, E zero, back zero. So kind of interesting. And like I said, can resonate with network engineers a little bit more. It's the default networking style used in Mesos at least last time I checked. Can avoid some of the complexities and kind of the costs of convenience that you might have with an overlay. So overlays are really convenient, pretty popular, particularly with developers who may have gotten a couple of hosts from their IT department or a couple of hosts out in a public cloud and now they need for those two hosts to talk to one another. One of the ways for those, well I should say they need for the containers running on those hosts to talk to one another. And a convenient way for them to do that and maybe do some shadow IT without actually having to go to IT and ask IT to make sure that these two VMs are in communication and give me another network so that I can use that network. Instead they might just set up an overlay between the two hosts. So very useful there. Very popular, a lot of the vendor plugins use overlays. They do oftentimes, are they almost always require a key value store? Whether or not that key value store is built into the solution or whether or not you've got to bring your own key value store has been changing over time. So over time what's happened is, it's bad, isn't it? There we are. Yes, here I am. All right. Okay. Boy I thought it was because maybe I didn't know what I was talking about. Okay, we're gonna have a much different conversation. It's gonna be much more personal. We're gonna, okay, good. Fantastic, so the train of thought that I was on was this notion of some of the way in which networking is working is getting easier with multi-host networking. As you go to track, oh boy, neighbor tables and names of hosts across different hosts, like hey, you need a centralized key value store to be able to keep track of that stuff. And it wasn't too long ago, a year or more ago, that essentially all of these offerings required a key value store, you to bring your own key value store. Some of these offerings now build it in. And many of you are familiar with Docker 112, the inclusion of Swarmkit inside of the Docker daemon itself inside of the engine. Part of that inclusion and part of that massive leap forward that they took was not just the built-in orchestration, but what was a lot of built-in, a lot of advancements on their networking as well. They have leveraged the RAF implementation from Ed CD and the GoMDB library from HashiCorp as well as the SERF project from HashiCorp, a gossip protocol. Just kind of a neat tool. But anyway, some of the other, essentially all of the other offerings out there do require a key value store. It's just how easy is it to spin up? Do they bring it or do you need to bring it? All right, so I just talked about overlays and the notion that they're convenient, the notion that developers can set them up. They don't necessarily have to go to their IT to ask them to set up a new network that the developer can do it themselves. But there's also a cost to that overlay, just in terms of adding on packets, tearing off packets. There's a kind of a category of container networking referred to as underlays. This is much more, you know, I think folks that are of the physical networking world will probably just resonate with those a lot better because that's in essence how these work. And by the way, these are not necessarily cloud-friendly either, so these are probably much more oriented towards, you know, running things in your own data center. The first on the list, and actually I'm recognizing that we probably blew past the time here, so I'm just gonna skip past this, like I said, the slides are available. This is Rocket's default networking, and it's fairly simple, it's point-to-point. MacVlan is, I mentioned before, like the latest addition to the Docker networking suite, sort of what were the types of networking that you get out of the box. There's actually two types of upcoming networking, of network types, I guess we just said it. One is MacVlan, the other one is IPVlan. They both end up, they're both physical networking oriented. They both end up utilizing, essentially, your host's network interface. Both end up creating sub-interfaces off that host network interface. The difference between the two is whether or not, and so as such, as we just saw with host-based networking a second ago, whenever you deploy a container onto like a MacVlan network, or an IPVlan network, you'll, that container will use the same network namespace as the host. They'll get the same IP address, and really the difference between MacVlan and IPVlan is whether or not they use the same Mac address. So with MacVlan, and I would try to demo this if I didn't have to hold the mic, maybe, but the, so MacVlan assigns a unique IP address and a unique Mac address for each of those containers that you put into that network. The issue can become in a physical network environment, a lot of times you'll have security configuration on your switches that will shut down a port if it sees too many Mac addresses pop up on that port. It's just kind of a common security best practice. We'd mentioned before about how many new types of container network, or how many new, or how many container interfaces you could have on a host, way more than a regular switch would necessarily expect. Anyway, the difference being with IPVlan, it doesn't use, it doesn't assign a unique Mac address to each individual container that you spin up on the network. It reuses the same Mac address. Can be good and bad. It's like, yeah, so there's some differences, some pros and cons of using IPVlan, MacVlan. IPVlan works in either Layer 2 or Layer 3 mode. In the Layer 3 mode, it doesn't allow any broadcast traffic in, but with Layer 3, you might head towards that if you're looking towards massive scale. And actually speaking of Layer 3 and massive scale, this is where you might also get into a project like Calico, which does direct routing, it uses Layer 3, it uses BGP. It gives you more granular control over the filtering, the firewalling of where traffic goes, kind of the isolating of that network traffic. It's also just kind of resonates with network engineers that have been doing Layer 3 routing. Okay, this last one, this last one actually, I'm not gonna describe. Who's heard of fan networking before from Canonical? Fantastic. Has anyone used it just out in the cloud, kind of for its most common use case? The use case here being you've got a VM that you've provisioned in a public cloud. It's got a single IP address that's been assigned to it, but in fact, you need a lot more of those IP addresses and you want those exposed. Fan networking is a clever way of doing some subnetting in order to essentially assign yourself a whole lot more IP addresses on the host and have those netted out or subnetted out. Anyway, it's an interesting one. I haven't seen it necessarily broadly adopted. Okay, so for the most part, what we kind of walked through today were just the different types of network drivers, the different types of the way that they behave, how it is that whether you get an IP address or a MAC address that's the same as your host or what network names they basically go into, but there's a whole genre of a class of other capabilities and services here as well. So we were looking at IPv4 addresses. IPv6 is still kind of an issue for us. As many of the top clouds don't support IPv6, has this changed in the time that I've written these slides? Has any, I don't think so. Okay, yeah, so none of the public clouds have announced IPv6 support just yet then. So we were saying that as we were spinning up new containers, at least for the Docker engine, that we were getting new IP addresses assigned, right? And those were, so IPAM is kind of in the box there for the Docker engine. IPAM and how that works is pluggable in Docker engine. If you plug in a different vendor's network driver, you may get, you'll get different behavior actually. One of the drivers is from info blocks. So this is where you shouldn't try to edit your slides on the plane. I guess what I was just gonna call out here is that the other thing that we've been talking about mostly is how it is that these different types of, different types of container networks function in their most basic form. And I said that Docker 112 took a massive leap forward. They did, and unfortunately, you know, I'm gonna have to try to describe that to you. Some of that's been around the leveraging of IPvS. So it had been about 10 years since I'd used IPvS. People familiar with that just in the kernel build balancing, highly efficient. Pretty cool things that the Docker networking team has done here around creating a routing mesh. It was in 112 that the Docker service construct came out and that service lets you essentially assign a virtual IP address to that service. The service may represent multiple, really kind of begins to represent your application and your application being comprised of multiple containers. Some pretty neat things that they've done with mesh routing. And so that's the ability to spin up a new Docker service, a new application offering. Maybe you're exposing a given port. And for your swarm kit or your swarm mode environment, for you to deploy a Docker service, it to come up and essentially for an external client to make a request of that application, it can send in that request to any node in the cluster. That client doesn't necessarily have to know which node to send it to. Because of the way that mesh routing works, the client can send it to any one of those nodes in the cluster and each of those nodes will reroute that traffic to the appropriate node where that container is running. Okay, another kind of high strain slide, but just some research around the various container networking solutions that are out there. Most of these, I think actually, well, all of these except for NSX are open source. Again, slide's spilling off there, but very popular, the use of VXLAN and overlays. If you're interested in more research there, these are links, so pull them back up. As we wind down here, I thought I would go back to the original question that I'd ask of the group. That was whether or not you'd use SolarWind software and if it would be helpful if SolarWinds kind of did some things around container networking. Well, we're beginning to, certainly giving some thought toward it and to learn more, go to the GitHub link there. I was gonna try to show you guys a demo, a preview of what that looks like. A tool, really, it's a free tool that gives visibility into your multi-host network.  Right now, it's a very nascent capability. It's an early free tool. The corporate VPN that I'm running to connect back to this thing is not functioning right now, so this is kind of, this is not, this is disconnected from the environment, but this environment was a Kubernetes, three node Kubernetes cluster running VXLAN as its network overlay. Anyway, a horrific demo, mostly because we're disconnected. The point of the tool is really to help give you visibility into what's happening on your network. It's really an educational tool. It's intended to help you make some kind of day zero choices to just better understand what's happening in your growing environment. And then to even let you do things like prove out what I was saying before about there being a cost to the convenience of an overlay, so the tool will ultimately let you select a given container, select another container, and do a bandwidth throughput test. So if you're familiar with IPERF, this is kind of basically the same concept. IPERF isn't being used here. This is on the back end is written in Go and is using a Go library to just generate traffic and slam as much network traffic as it can through there to just perform a test. So what ends up happening is you can run this in multiple environments and this will facilitate really just flexing that network driver or that type of network, the ones that we've stepped through before. So you can really begin to see the difference in performance. So unfortunately the tool is separated from the back end but I did get a screenshot of kind of what one of those results look like. And I guess this is a different environment. I believe this one is a two node Docker swarm environment. Anyway, just, yeah, you know, it's a horrifically generic name. It's container network performance tool. It's extremely descriptive of what it does. It is, the link to it is just GitHub under SolarWinds. There's a container section. So yeah, bad demo. I don't know that the tool isn't clearly not at a V1 but I did want to take the opportunity to expose it to you guys because ultimately it would be a free thing and hopefully you'd be able to take and learn and maybe self-instruct better than I have for you today. So if you do try it out, let me know. Let me know too if you're running SolarWinds software and you've got use cases for us. Always open to feedback. So questions. I know that all of what I said was crystal clear. Both, half of that's probably just my issue with talking. Yes sir. Yeah, great question. So the question is, hey, where does the 172.17.20, I think it was .20, or .00, you know, address space come from and that's a great question. So in this case, we were walking through the how Docker networking works. We were taking a look at, right then, we were taking a look at how does bridge networking, the default network type inside of the Docker daemon work and whenever you, whenever you spin up a new container on a bridge network in Docker, it will have assigned a subnet that it, so it has built in IPAM and it will have assigned a subnet to that network and it by default starts with 172.Jerome17 or is it 16? 17 and then just kind of keeps going from there. So you can create new networks as well, new Docker network, new Docker networks of type bridge and each time you create one, it will assign a new subnet and it just has its own internal IPAM so it'll go 172.18, 172.19 and it'll have a, it'll track the IP addresses for that subnet. Every time that you spin up a new container and assign it to that network, it'll just dole out a new IP. Oh yeah, sorry. So the second part of your question was, hey, where does that come from and can I change it? So it comes from RFC1918 if I recollect correctly which is private, yeah, which is extremely helpful. It comes from, it's a private network. Oh yeah, yeah. So there's, I think, I don't know if it's an IEEE standard that says, hey, that particular network space, 172.16 slash 22, help me out, slash 16, I guess. Anyway, there's a private, sort of globally worldwide. Oh, okay, yeah, sorry. You know, I don't know. There may be others in the audience that know, sorry. Oh, okay. Yep, so it's part of the, so I'm slaughtering the question. It's part of the question. Hey, I don't want to use bridge networking and I don't want to use, no, okay. It's, hey, I want to use bridge networking but I don't want to use that address range. I want to use a different address range. Fantastic question. It is configurable. I forget specifically where it is, that was the problem. So yeah, there. Fifth time's a charm. Yes, sir. Ooh, say the last part, the first part was if you're using shared networking, oh, I see. Maybe I'll take the example of the, like the container maps networking that we looked at where one container joined the network namespace of the first one. They literally are sharing the same, yes, that second container does suffer as the first one does. I think two use cases come to mind for me. One is that, and I don't know how often it's necessarily used in Docker land, but one is, hey, you've got a container running. It's performing a particular function. Turns out you're having an issue with it. You want to diagnose it before you kill it off, but it doesn't have the right tools maybe that you'd want to have in that container. So rather you bring one with the right each tool and other, or whatever that tooling is that you need, map that new one into that existing environment, take a look, do some diagnosis. I think the other probably common use case is like when you're in, if you're in Kubernetes land, if you will, it's not just, if you're using Rocket as your runtime, it's default networking is such that, let me describe a Kubernetes pod really quick. Inside of a Kubernetes pod, you may have one or more containers. And part of how that pod works is they all share the same network namespace. So actually by default, whenever you deploy a Kubernetes pod, they're all of the containers in that pod are all doing container mapped networking. So that's just like the style of networking that happened. And so why is that beneficial and why would you use it? So why is a lot of times you've got, maybe you've got a logging capability. You've got two containers that won't have co-scheduled next to each other and they've just got high frequency of traffic between them. And so you might want to map that second one into the first containers network because you just want them to talk over loopback zero rather than having them go out, having that traffic go out somewhere and come back in. Good, excellent, yeah. Thank you guys. Okay, we now have a 15, actually 20 minute break and we will resume at 3.30. So, see you then. Test, test, test, one, two, test, test. Two, two, two, two, hey, hey, one, two. So what's it like that? Yeah? It wasn't even on. Okay, okay. Two, two, hey, hey, one, two. Two, two, two, hey, hey, one, two, two, two. Yeah, who knows? Yeah, I think we're good. And this is the same country, right? Same one? Yep, yep, all right, we'll try it. All right, cool, cool. Try it. You want me to hang out? Yeah, it's okay. Where's comes to where, so I'll just change it. Just get it forward. We're just going to ask about the monitors for the exhibitors, so I'm right here. So if you need me on the back end. Thank you. No problem. Test, test, test. I'll keep the volume down there until you're ready to start. Make sure I don't bring this to the bathroom. Please. And talk something in there, I should not be saying anything. Don't drop it in. That's even worse. Is it good enough? Or I need to point to my mouth. That should be good. Right, that's something, it's a problem. So it's actually pulling, there we go. Okay, here we go. So. I do see security to be an important aspect. And I think this is why my session got accepted too, because it's popularity. Yeah, well it's something, because I think it's like, I wanted a lot of basics in here, right? So I'm actually, you know, I'm running this thing. So I wanted a lot of basics in there, right?  So I'm going to start with security. I mean mind you, my basic attitude is that containers and security are largely orthogonal. Those containers don't really give you any more security than you would with different processes running on the same machine under different users, right? So, but people think that because stuff is not immediately visible from inside the container that somehow in security.  For sense of security. It's a picket fence. This has to be willing to jump over. We have to be right all the time and the hackers can only be right one time and it's already good enough. For 40, 45 minutes and then take questions, right? And then we take questions, I'm going to run around with this mic so that people can ask you questions. And also, yeah, 35 minutes, I'll hold up my hand until you get 10 minutes. And you will be somewhere back? I'm going to be right here. Oh, thank you. I'll hold up my hand until you get 10 minutes left and then at 40 minutes I'll hold up the time. But I never timed my presentation. I don't know how long it goes. It may be shorter.  Yeah, yeah. That's why you get the 10. Yeah, that's good, yes, yes. Yeah, thank you for giving me this opportunity. Oh, with some kind of thing, not security, so. Are you local? Are you out of the Bay Area? I'm in Los Angeles. Los Angeles. I was surprised people start walking out. Why? Don't be fooled on a previous session and so I asked one of the people why people are leaving and said, oh, the mic was terrible. And with revealed JS, you have to tell a year's screen resolution before you start, which you hadn't done. I run Ubuntu on the old laptop. It's on. Oh, it is on, OK. It has the volume to be viewed. Oh, it's the volume to be viewed. OK, hold on a minute.  We're going to get started again this time with container security. Our speaker is Anthony Chow, who programs networking devices for Alcatel Lucent, and he's going to give you a general overview of container security. So welcome. Thank you. Welcome to scale. I feel better now because anytime when I see people walk out of the room, I can just blame on the mic. It's not my content or something I see. Actually, with a little minor correction, I don't work for Alcatel Lucent anymore. Alcatel Lucent got bought by Nokia. But that's another story. My name is Anthony Chow. I am a software developer. I work on writing firmware for networking equipment. But I'm also interested in security. And that's why I always look into different aspect of security. One of the part of some of the features I work on is also on network access control of the switch. I also help on DXCB snooping, where we do some consider as a security add-on feature to the switch. Last time I worked on is to help our switch to pass the common criteria certification. So there's a lot of things that we need to look at. We need to sear out the password. All these things, those are. Security to me is a fun, very fun subject. But it's also a very important subject.  I also occasionally, I write some blogs. I'm also a contract blogger for a car company. I once, I heard there was a conversation saying, in recent days when there is a technical conversation, container will always come up as the subject. So I think it is very true because this slide I put up to just put together as an introduction. I'm sure you guys can think of a lot more advantages of using container in your production. I don't see that we need to go very deep on that. But again, when there is an advantage, everything there is also some kind of disadvantages. And again, this is also an introductory slide that if you can just, I'm sure some of you can always think of some more other disadvantages of using container, but this is not the concern. The concern today is that we want to have a general overview about container security because this is a popular subject. But it seems like container is very useful, but security is not something that people are looking. Although this is becoming a topic that people really look, trying to look into. We, the different types of threats to a container, and I listed some of it, I think the biggest thing I think is people are afraid of being escaped. Escape means when we're running a container, the hacker can escape from the container because we think of a container, it'd be nice if we can run it as a sandbox. But actually, it may not be dependent on your requirement of how we look at the container because there is an article back in 2016 about, I'm sure you guys can heard of it, is that container do not contain. Somehow it was circulate among Twitter or the social media for a very long time at the end of 2016, even though the blog posts, maybe we'll take a quick look at that. This is a very, it's a good, even though it's an old article, but then it is, this is very relevant. I think that the containers do not contain, it lists out some of the things that what container can do and cannot do. I encourage you when you go back, you can take a look at this, just look for this title is from Red Hat, the containers do not contain, and the exact title of this are Dockers Container Really Secure. Also, since I look at Docker, I also have to make a little disclaimer. This topic is more on container in general, it's not specific to Docker, although some of the examples I use is on Docker because it's more popular. It's similar to the Xerox machine. Actually, it is a photocopy machine and it's not a Xerox machine, but Xerox makes it so popular that people intermix the company name to the product, and it's the same thing as Docker container. And for Docker, I like the way because before Docker, containers are already being used, but it's not that popular because it is not that easy to spin up, although Google has been using it for a long time. They also have the project, not the project, the feature board, BOR-RG I think now becomes Kubernetes, which is a very useful container orchestration tool. And also, there is a BOR-MAR. I read this from the book, something to do with... SRG, I forgot the name of the book, but it's talk about how people do DevOps in Google. I think this is a very interesting book too, but back to the subject. Today, we're talking about container in general. There is LXC, there's also Rocket, that different flavor of a container. So this is talk about how things... Because most people, when they think of a container to be a sandbox, they think, oh, the immediate thing of, oh, there is namespaces, there is C-group, and these are the two major things. But in this article, it points out not everything in Linux world is namespace aware. And something like, as I just mentioned here, some of the, let's say, the PROC FS or the SysFS, they're not namespace aware. So these are the things that they may not be able to contain a container using just using as namespace. Let's get back to the slide. The XCIP is one of the things we want to look at and also for container, they're running on a network. In the previous session, we talked about networking. And networking is a dedicated, a very, I'm thinking of trying to think of a great word to explain. It's a dedicated matter to make the network right. Before all these CCID, how to configure a good Cisco network, just try to make things pumping through. Although there's one thing as well, we want things to be able to communicate easily. Of course, when these things happen, security is also a concern. So the cross-container at text is also something we need to look at. I always said security and user friendliness is too spectrum, too side of a spectrum. If you want security, it may not be that useful. If you want usefulness, security may not be there, so we need to strike a balance, tailor to your application. So this is something, one part of the threat is that the cross-container at text, if I be able to set, do, let's say for example, for cross-container at text for app spoofing. We spoof, and then we'll be able to look at other container, look at the database within the network. Well, although for easiness, the bridge network is always an easy way. Even for VxLan, it's also considered, from the container's point of view, it's also a bridge because underneath is, I layer three networking, but then it will just bridge over to the other side of the whole. So to the container, it's still a bridge. Whenever there's a bridge, there's a broadcast domain, and there's a broadcast domain, it's very easy for container to look at other containers. And of course, other types is not, it's very obvious, it's the application vulnerability of the container. Let's say we run a NGINX application. The natural vulnerability of that application exists in that container. We still have to deal with it. It doesn't, if any run on a host machine, physical machine or running container, those vulnerability is always there, so we cannot forget about it. Even if we sandbox it, it's still there. And also, there's also the denial service or attack on the host. We might use up all the Inox on the host, or we might just need some kind of denial service on the host. These are some of the threats. When we look at the threats, then we'll try today. In this session, we try to look at container security from this different perspective with the host base, container base. I can point out some of the third party security offering and there is something I don't know how to categorize, so I put them into miscellaneous, which is a clever way of saying they are miscellaneous. For host space, there is always, I guess people mention about namespace as C group, but also, we've also found out there are also the root capability and also the Linux security modules. We will look at it. There's a general thing about host-based container security. Something about this presentation is that this is a general overview, it's more on the broad side. Anyone is a CISSP certified or GSAC certified? You know, the question itself is not too difficult, but if you have that many aspects of it, you have 10 domains, you have to know everything of it before you pass the test, that's more difficult. And also, same thing with this presentation. There are so many things we can go on or we can go deep into that I don't have enough time to present everything, but this is just to let you know, this will be a survey of all the two things that we need to look at so that you have a general understanding of what security concern we have with containers. I borrowed this slide from the internet. Of course, you can see from the end it's a presentation by IBM. I always said a picture is worth a thousand work or in a developer's point of view, a picture is worth a 1K, 32 bits. Anyway, this is a lousy joke, but just to see if you guys are awake after so many sessions and after a heavy lunch. So this is the problem with the namespace is for isolation. Mainly, namespace is used for isolation and there are a few things. As we look and see, there is the mount point, namespace, network, IPC, UTS. These are the very common. And the last one is the user namespace. This is something new. And I think for Docker container, it's got introduced in release 1.10 for the user namespace. But then, even though it's out of experimental mode in the Docker container, it is, by default, it's turned off. So I believe when I look into it, this is a very nice feature because it helps to mitigate lots of the security concern about container. But somehow, Docker is still making it the default as turned off because there are certain things that still need to be worked on. This is the namespace. And to turn it on is that we use this user, user namespace remap. Actually, is this big enough? Do I need to turn on the fonts bigger? A color, what's the good color? Black on white, let's see. Great on black. This one better? And how about this font? Do I need to make it bigger? Oh, thank you. How's this? Okay. I need to get to some of the cheat sheet. Let's say I have my cheat sheet. I'm not a professional presenter, so I need this to help me out. This is a Docker command that we, it's a simple Docker command that we use to spin up a Ubuntu container. And then it is to map slash, the host slash sc to root sc in Ubuntu. That's what it means. This is when username space is turned off by default. So I have this in here. And then let's see, I'm running as root. Let's say there's something I need to show. Let's do this. Actually, if you are, you can read fast, you will see what are some of the things I will try to do later on. So on this file system, everything is named under root. And then let's say if I want to touch slash root sc, let's say test one, I create a file. And let's see in here. See, within the container, without username space, I'm able to create a file in this directory where the container is running as root and I'm also running as root. And that is not a good thing because if I'm hacker, actually, this is not a good example. If I'm hacker, I have physical access of your machine, you'll be dead. I don't need this kind of thing to get into your network or into your servers. But this is not a good example. Or in some case, in case this is something once someone break into this container and try to do something or maybe do something with the password or to create some account, then this is not a good thing. One thing is that one day I was trying to prepare this demo, I was trying to say, oh, yeah, it's right. We have to set the flag when we start Docker. But where is it? Is this a 16, Ubuntu 16.04? Where do I make the changes? I spend the whole morning and then eventually I find an easier way try to how to reset it with the flag. It's not changed something to do with changing system D changing system D but let's say, let's see, hold on, let me get out my cheat sheet. I forgot, let's see, it should be, it should be here. We have a file called JSON file. But if we can copy, I already have it prepared home. If I can type, let's get out of this. DAE, this is a simple JSON file. I already have a user created as Docker doc remap. Let's see, a user. So I try to remap to this. So now I need to restart Docker to make it to work. I have a slow machine, actually it's not too bad. So I restart, let's try to do this again. Try to do the similar thing as we did before. We start, we try to start a Ubuntu container and then we also remap the two directories. This is expected because anytime we restart the container and then we create a new, even though I ran this command before, we have to download Ubuntu again because in a short way is that, let's see, it's varlib thing Docker. I hope this is it. Because now we see we have a new directory and this is because we turn on username space. So now this, now I am in the container. Let's see if we can do the similar thing as before by, let's say we do this ls again. This is inside the container. Before the file is owned by root and root, if you remember. Let's see if I can go up and see this whole bunch of file. Before the file is under this. Now we have username space enabled. If we start the container, it's changed to a different, it's nobody and no group. And then let's see if we can do a similar thing. Let's touch, let's see, say test two. We create another file in this directory. Now with username space enabled. Now we cannot do this. So username space is in a way is a very effective way to do some of the things, but I have to do more research as to why there are some other problems that prevent this to be the default to enable username space. But you can see username space with this because what it does is the root inside the container is not root in the host. And this make a very big difference. We remaps to a different user. So it will not be running as root. And this helps mitigate lots of security concern. And now gone in security term, we can only mitigate as much as possible. We can, we cannot totally eliminate security concerns. We just make it as low as possible.  So this is just to show you with name space, this is a useful way of doing, to prevent some of the things from current slides. And let's move on. There's also the, yes, if I can answer, yes. I am not so sure, I don't know. I do not know the second, but also, but it depends on how you spin up the container. The file, you have different ways. One of the quick way I do the demonstration is just to map, but you don't really have to map that. You can attach a volume to the container. That's, there's also one way of doing that. I think people, if they are doing in production, these things is, they can maybe help answer this question. But for me, I do not know how that works. And we move on to C group. And this is something, name space is to limit the view of the system. And the C group is to limit the resources. To follow, again, I borrowed this from IBM to prioritize control and obtain metrics of a group. So this help, this is something. And then there is one more thing is called the root capability. For the first line, this root capability is a fine-grained control over root privileges. If you go to the user on a Ubuntu system, I think Fedora also may have this tool in the same directory. And user include, it will have more, a lot more, let's see, let's maybe take a look at, this is the other one is root, this is better. So before that, I think we can show you the capability of Aspin, okay, SH print. This print now all the capabilities that is divided, and let's see, forgot where it is. There's too many things I need to use to include Linux. This is to show different ways, this is good. For me, because I am a developer, always like this .h files, it helps to, usually it helps to clarify different things because you're supposed to have a good comments on this .h files. If you're interested, just take a quick look. This capability, there are a few things we need to take a look at. This is one of the web page that shows all the different capability, if you're interested, just go in. And also, when we run a container, we can also specify. If we want this, we can do a cap drop, do a, this flag will help us to drop one of the capability. You want, I'm not sure if Ubuntu has the same thing, this article that I read is that Red Hat has something in this way, let's see if we can go there. It has a script to look at the capability of a container, so you will know what kind of capability this container has. Do they have, because we always want to follow the principle of least privilege, we don't want the container to run more than it should or necessary. So, if we can find what are the capability that a specific container that you are spun up, then maybe you can decrease the capability of that container. And this one, for Docker, has a good way of using set-com to control. To control, I can go to the website and take a quick look. This one, this is a good way. Maybe I will have this included in the end of slide, because my slide will be attached to the presentation. This, no, I already passed. It's difficult to think and it's the wrong way. This is not the right page. Let's see, let me jump ahead a little bit and go here. Wrong one again, this one, it's different. It should show me the, let's see, let's see, let's see. Thing is this one. It shows you how to use set-com to control because it has the default profile that you can specify with a JSON file to create or to limit what kind of system call because for this capability, it is trying to exclude the CAPE system in. What I have found during the research is that this is like a catch or capability, some of the things you can do and with the container to break out, to escape from the container is using this capability. So it'd be nice if you can find out what root capability a specific container has and how we should be able to control and to follow the principle of at least privilege. Then so we get back to root capabilities. And this is, another thing is access control. In securities, there are always a saying called, I forgot exactly, there's defense in death. Just like when you have a castle, you have a thick and tall wall, but before that, I'm assuming the European castle type that in front, there is also some gap with water, maybe you have some hungry kogledows running so that people cannot swim over and, because this is called defense in step. With all these namespace group are also root capabilities. We can also use this type of access control to help to mitigate some of the things. Usually when we look at the Linux system, we are looking at the discretionary access control, the file privileges, user, the group, let's say if I'm just a regular user, I'm not root, I have to ask you to root so that they are able to go to slash ETC to do different things. This is a discretionary access control, but there is also something called a mandatory access control where the system specify the subject can access the specific type. Everything is a label in the mandatory control. I think if someone is involved with the military, they're more familiar with mandatory access control. At home, I have two machine, one is running Ubuntu, not this one, this is just an old laptop, I have an NUC running Ubuntu and I have another system running Fedora, just for the purpose so that I can learn SC Linux, which is one of the flavor of mandatory access control. This is something to do with two of the things because Linux security model, there are two flavors. Actually more than that, there is also the GR security which are mentioned under, I think, miscellaneous. GR security is a specific hardened Linux system that you can use before it was free, but now I think you have to pay to get the operating system but it's more fine-tuned and it's more supposed to be hardened, more secure than the general Linux distribution. For SC Linux, there's three most enforcing permissive and disabled. Some people I know, just some of the developer I know, they will use Red Hat or CentOS, but they will, the first thing they do when they spin up an operating system, CentOS or Red Hat, they will disable SC Linux because there's too much trouble, they don't want to deal with it. But in production, it's a different story, we do want to know, but I think it's something nice that because we have three most, one is the enforcing permissive and disabled. Permissive is that it will flag you with the concern or the violation so that you'll be able to tune instead of really enforcing. Because this, again, tuning security in a container environment is dedicated as a fetch out issue. It's not a silver bullet that, oh, follow this rule and you can do it. But then this is before you have to move SC Linux to be the default, we have to put it, we can put it in permissive mode so that we know all these other things that's not so we can tune it. And then there is another way as the app or more, if I can say this correctly, the two mode is enforce and complain. I think it's also a default in, you've been to, this is through PPA more status. These are the things, it's a little bit different than SC Linux, this is a file base of a path based security. It's the way the implementation is a little bit different but it's a default way for if you look at what is that called Docker info. You will see in, if I can find it in, believe me, it's there. It's spotted. Yeah, you can see it there, it's good.  Oh, it's, anyway, if you see it, it just be this good, it's there. I also have a machine that I can show you more about SC Linux, but I don't have anything, we have time to go through that. But I do have to want to point out a one of this article. Again, this is from the same author that says, the talks about the container cannot contain. It talks about more things about the file system and it also talks about capability. These are the things that, I think these are the default. See, dockers remove these things by default. So for sure you will not, but if you need it, because sometimes depending on the application that your container is running, these capability might be necessary for you then you will use the cap add to add it on to bring that. But what I want to point out is that there is a good way, this talks about cap add and cap drop. It talks about namespace and them, but the very big thing is about the description of SC Linux. I think this is, SC Linux is a labeling system. Every process has a label and then it's again, but what I like it is that with the picture, again, the picture says it all. This is supposed to be, I think they make it as a colouring book. If you look at it, it says, this is the kernel.  This is something to call type enforcement in SC Linux, but that's not enough. There is also something you have to look at is the multi-category security enforcement. If you look at this, the food itself is for the doc, but specifically for the doc name spot and this doc is called Fido. So the kernel doesn't allow it to eat the food. This is a very simplified way of how SC Linux works. This captures the essence of how SC Linux is implemented to control user access to different resources in the host. So I think this is worth something going to because, again, this is an overview session so we're not going to get into more detail of that. So for any question, comments, because you might know this subject a lot better than me, but again, this is a general overview, so to prompt you to see if you can look into these specific areas. If no, we move on. We're done with the whole space security. Now we talk about container security. There are three different ways of looking at the container-based security. I look at it as two different categories. The first is the digital digest and then another one is the container scanning. Digital digest is an important aspect in security. Again, I get all this picture from the internet. At this time, I don't have the skill to draw my presentations yet, but this is a very good nice way of putting things, how digital signature works. Basically, you have a good container. It has a hashing algorithm. One thing you have to remind is hashing is only one way. You can only hash one way, but for other encryption, there are also the symmetric encryption where both sides have the same key. You encrypt and decrypt with the same key and also there's an asymmetric encryption which I think is necessary for this type of discussion. Asymmetric means that there are two key, a private key and a public key. You can use a generally available PKI. You can buy the certificate from some of the vendors or you can create it with the open SSL, Kamel, or Linux to create your private PKI. But the most important concept about that, about asymmetric encryption is that there's a public key and there's a private key. You encrypt something with your private key and you decrypt with the public key. If you encrypt with a public key, you decrypt with the private key. It's always the other way, so you cannot encrypt with the private key and decrypt with the private key. Even you yourself cannot do it. You have to use the other way to make sure that, and how this kind of digital certificate works with the container is that they hash the cold container. It's not like a CRC in networking because they just, CRC the whole packet and then if the two bits are flipped, then your CRC is the same. But this is different from digital certificate. It won't happen this way. It will just hash it and then use to see whether it's the sender's private key to sign the message. They use it to say, oh, this is the hashing value and send it to the receiver. And then receiver use the public key of the sender to decrypt it and then they do a same hashing algorithm to see if the number matches. If it is matched, the container content has not been altered or changed. So you are safe to use that container. This is one way to, I think this is how, in a nutshell, how Docker container trust is how it suits them. I think there's more in depth to that, but then this is in general, this is the general understanding. If you use something, you can decrypt and you can extract 509 certificate. This is two sessions. The beginning part is the certificate, which has all the values, the versions and the different contents. And then, again, if you look down here, it's the private key to sign this certificate to ensure this content is from the sender. So this is one thing to look at. And what I point out is that CoreOS has an interesting implementation of how to verify or to ensure the integrity of a container image is that it is doing it during runtime. I don't have time to go into detail. If you look at just Google DM, Verify and CoreOS and during the keyword runtime, then you'll be able to find out this interesting article to see how they do it in the runtime. And then for container scanning, there are different vendors that is providing different ways of container scanning to make sure you're not vulnerable. Because again, it's one of the threats to container is that generic vulnerability of the application that is running on as the container. So these are some of the offerings that is available for us so that we can be able to scan the container to make sure their security patches are up to date. Then these are some of the third-party security offerings that we can look at. I did not sort it with alphabetically. Our core is one thing. Encore, I'm not sure if any one of you is going to the Los Angeles Docker Meetup. I think they had a presentation last time when they meet a presenting this. And then if you're interested, just go to... I did not attend, I cannot make it to that meetup, but I know last presentation was about this product. Before we talk about TwistLock, the previous presenter asked people if anyone knows about TwistLock, I think. If this session is before him, then every one of us can raise our hand and say we heard about that. And then these are the things that third-party offering that you can go into and see. This is what I call the miscellaneous. OCI, the Open Container Initiative. When I Google this, when I put together this presentation, I did a lot of Googling. Open Container, I cannot find any specific content to see how they are trying, using the initiative to improve. But one of the interesting things is that one of the objective, one of the goals within OCI is that they want to improve the security of container. I'm not sure how far they have gone, but that's why I want to mention that OCI, plus also I think with the ecosystem, people are moving toward the OCI spec. Even CoroS, Dockers, they're all moving toward, and I think Dockers donated RunC to OCI to make it open source or to make it available to everybody. So I think for container security, OCI is definitely a thing to look into. Another thing is interesting is that the hardware assists. Intel has the clear container. It has a very interesting concept. What we talk about from the beginning until now is about running a container as a container. I don't know if this sentence makes sense, but Intel does have an interesting concept of running a container as a VM, because what they said is that a container running on software cannot take advantage of the hardware-assisted isolation, in which case we can all agree is that the isolation of a virtual machine is much better than a container, and this is where Intel is trying to put itself, is that it will try to change the KVM, I think it's the KVM tool and some other things to make the container take advantage of the Intel-assisted isolation, so make it more secure. The last time I had was in a seminar, it's some kind of workshop, and Intel clear container make a presentation. Lots of questions talk about, so are you a container? Are you a VM? I think they are both. It depends how you look at it. If you look from the application point of view, it does work like a container, but if you look from beneath, it does have the hypervisor and it takes advantage of the hardware assist, so depending on which you're looking down or looking up, it all depends, but the general idea is that Intel is moving this direction is to use the Intel hardware to help the isolation. And again, under miscellaneous, thing Docker 1.13 is just being released, and I have just to mention the secret management. Secret management is a very important aspect of hiding the secrets of the password or these certificates. I think this is also worth looking into. Let me take a look at the time. I almost run out of time, let's see, because for security, this is the, if you look at my description on the talk, there are three parts. The last part is supposed to be a demo. I'm not sure if I have time for the demo, but let's go on to see. I choose to use the Ansible container. I'm not saying, it is supposed to replace, I try not to say the politically correct word, not to replace, but the augment Docker compose. It's supposed to be able to use the same, because they use the same JSON file, they'd be able to, what I like Ansible, before container, well, I already like Ansible, because I look at puppet, I look at chef, but all I like Ansible a lot, and I have done some work on Ansible, so I think it is very nice. Mix of things, not only can this Ansible container use your existing Docker compose file and spin up, it will also add on different roles to help your security. To, let's say, let me see if I have time to make a quick demo on this. Let's see, for the Ansible community, they have something called the Ansible Galaxy, where people write already existing roles to different things. When I search, I search this. This may not be a good way to do it in production, but this is a good thing to show examples. Let's say, I download this, I copy this from Ansible Galaxy. Let's see, there are a few things. One thing I want to look at is that, SSD, oh, one thing I want to show is that Ansible is very, container Ansible is this, let's say, make can be Ansible container. This command is very neat. It's already, we can provide you with a structure. See all this is, oh, I did it on the wrong. This is a general mistake I always make, is that I make it directly, but instead of going into the directory, I did not do and do the command in here. But anyway, let's do it in just because of time. Let's do it again, Ansible container. It's not Ansible, but it's Ansible container, init, initialize. So it gives you the directory, and it also gives you the different files. A sample file, based on YAML, main dot, this is a sample file, that's easier. Then container, this is supposed to be the replacement of Docker Composer, how you spin up a container. But what it does is I show you here is that same thing. For this download, there is Ansible file, and then there are this container. So this is a simple, able to see, yes. This is to spin up a Ubuntu container with SSH using port 22. If you don't want port 22, you can change it to something else. And this is some of the things specific for container. But the good thing about what I meant by security is that when we can do this, I mean, well, I'm not sure why this is the case, but people said it's not an advisable thing to turn on SSH on a container. I don't know, although I picked this example, because it has the ingredient that I want to show how Ansible can be used as a security tool for container. Because if you look at this, we have all this, this is a very bad way of thing, because we are coding the password to be password inside a file. Maybe this is what the secret management can, I think Ansible also has something called a vault to help to hide the password, but this is not that important. So these are the things that we try to do. One of the things that's there for Ansible, we can configure the container or the VM that we want or the target. To say in Ansible's turn is the target. We look at the target and then it will be able to, let's say if you attend servers, then there will be 10 targets. There will do be the doing the same thing on these two. You can pick and choose. If you do want to have some security hardening of a Linux machine, you can do it on all 10 or you can selectively do some of it. And this is something you can change. One of the replace, this is to replace the destination. This will the target with something and this is how you can, this is a good way to also, just to orchestrate a container, but also can add on to make it more secure in a way, so some way to harden your Linux system. Let's see if I have time for a little demo for 18. So I still have a little time. Let's see if Ansible or my machine can work it. Oh, I forgot to show you the SC status I get and force on the. It's very simple. Just if in that directory, let's say in this directory I do this command. It will build. Are you in it or are I should be doing this in the Ansible directory? Let's do it again. So I pulling in the files, building. While it's building, is there any question and comments about security? I think this is a hot topic. Everyone wants to talk about it. And if you have some, I may not be answered the question, but if you throw it out, we can answer it together. Yeah. Will the slides that you're posting and all the URLs you've visited be in the slides because they weren't. I will put it at the very end. What area within a container security do you think presents the most serious vulnerabilities? I think the answer to, everyone can anyone hear about the question? It's okay. I think it depends because security contacts within different organization is different. It depends on the application you're building in different environment. Let's say for me, I used to write software for embedded system. So my security concern is one thing. If you're writing a web application, then your security concern is a different thing. That's why I think for this presentation, we try to get a much border perspective of different ways of looking into container security so that we'll be able to look into it that more in depth and to make the proper adjustment tailored to the requirement or the needs of the organization. I think closer, Mike. So one of the things that I see in how-tos all over the place, everything runs as route. How much of what you're talking about right here is able to kind of lock that down so that we can follow the how-tos without compromising security and keeping things from being able to break out of those containers? Do you mean the running Docker statement as a route? A good example, like the Ruby on Rails, everything runs as route. And we've been pounded in our heads for our entire adult lifetime, never run anything as route. I myself do not have to answer. Is anyone can answer that question? Will someone from Docker can answer that too, maybe? I think you get an answer. So eventually, user namespaces are going to solve that because they will allow to map user route in the container to something non-route outside of the container. So for now, if you want to be secure, you have to put user something in the beginning of the Docker file to say this is going to run as non-route. But eventually, once all the little problems that you mentioned in the beginning, like why doesn't Docker engine enable user namespaces by default because there are mini-little things to figure out mini-little features that break with your namespaces. But once this is figured out, then user namespaces will let you run stuff as route, except it will not be route, it will be mapped to a non-route user, so that'd be fine. Under Project Atomic, there's also a little project called Bubblewrap that basically simulates that until user namespaces are available and standard. Could you please address the use of TLS certificates to lock down the daemons that are running? So whether it's in Kubernetes or it's the Docker daemon itself that are running, it's system daemons, what's the value of that? And clearly it's insufficient for the container itself, but does it help at all in locking down the container? What do you mean by to lock down the container in what sense? So that authentication access to say Docker-D, Docker-Engine, the Kube daemons, Kube-CTL, Kube-API server, all that stuff is authenticated only with TLS. And does that help or does that hurt? I'm not too familiar with the tools, but usually in the security world, using a certificate is always better than using a password-based authentication system because with the certificate, you get more elements being put. You have the PKI, you can authenticate the user or to creation, you can have more sense of security with the digital certificate and use a simple password. I think this is my answer to your question, although I do not know too much about the tools security. Anyone can address that too because this is a beauty of the community. I do not know everything and then if someone can answer the question, then the community will grow. We'll have to follow up on that because we're actually out of time at this point. So thank you very much. Thank you. And don't go anywhere because we're gonna have a couple of speakers up here in 10 minutes so we're going to show you building Minecraft servers on Docker. Oh, just in case, I'm not working with Akital Lucent, I'm looking for a full-time job so if you have some job offering, ping me. Rex opened in Potato Technologies in Red Hat. Oh. See you in a moment. What do you buy me somehow?  So if you're interested, absolutely. Let me give you, I know you know where to find me. Is it easy to see the screen? Of course. If you don't see the particular point on the slide. You're just gonna have to like stand here for the video and like look at that. See right on. Okay everybody, take your seats. You're gonna learn how to deploy Minecraft on Docker so that you can not get any work done for a month. This is Proud Heng and Akira Wong, both students at different UCs, I believe yes, who are going to show you how to do that. So welcome. Thank you so much. Good evening. Like you said, if you're here for Docker and Minecraft, you're definitely in the right place. Thank you so much for coming out tonight or today. We're gonna get started right away. He already introduced us, but my name is Akira Wong. I'm a computer science senior at the University of California, Irvine. And this is my co-presenter, Proud. Hi, my name is Proud and I'm a junior at UC San Diego studying math and computer science. So let's get some audience participation going to start off this presentation. How many of you have heard of or played Minecraft before? Please raise your right hand. Yeah, that's a lot of you. Keep your hands up. How many of us have used Docker in production or development before? Please raise your left hand. Now some of us look like we're going down a roller coaster. Some of us look like we're asking a teacher a question, but we all have something in common today. Please put your hands down. Thank you so much. We all have something to learn from this presentation and that's that Minecraft is awesome. It's an excellent entry point into Docker. We can show you an excellent demo to show at your local meetup group, to show to your boss, to get you into Docker and to show what's possible with Docker in containers. And for those of us who didn't raise their hands, you guys are in for a treat. This is going to be really cool. So without further ado, let's talk about Minecraft. Minecraft is gigantic. Some would say huge. With over 23 million PC downloads and 100 million players over all platforms, Minecraft is a gaming phenomenon. Part of the Minecraft experience is exploring these huge sandbox open worlds with your friends and building complex megastructures like castles, farms, townships, who knows, right? The only problem with the entire Minecraft experience is the Minecraft server setup. It's a little bit of an ordeal. You have to jump through way more hoops than you probably should to do the thing that you care about, which is ultimately play Minecraft with your friends.  So how do we solve all these problems, all the difficulties with setting up a multiplayer Minecraft server on the PC? The answer is Docker and containers. For those of us who weren't very familiar with Docker, we're going to give you a very brief overview. Docker is an isolated application environment for us to host our Minecraft server in. It conveniently encapsulates all the dependencies we need, but is more lightweight and scalable compared to a virtual machine. This is because the kernel is shared between container instances on the same host compared to a virtual machine. Therefore, it's more lightweight and scalable and you can fit more containers per capita. In addition, containers are also portable. This is similar to the idea of the Java virtual machine where you can take any computer capable of running Docker and run any container on that computer. This means that our Minecraft server solution is completely portable to any platform and it's very general. This combined with the next facet configuration as code in the form of Docker files allows us to encapsulate everything we need to set up this Minecraft server and have it be compiled into what are called Docker images which are then hosted on Docker Hub, which is kind of like GitHub for Docker. We can pull down these Docker images and then Docker stamps these out into individual Docker containers. This allows us to automate our Minecraft server setup and get all that nonsense out of the way so we can do what we actually care about, which is, again, Minecraft. It's all about Minecraft, guys. This all sounds great, right? But it comes at a price and that is the price of statelessness. Statelessness is a core tenant of Docker. It's the idea that if you spin up a container and you write some data inside of it, then later when you kill the container, bringing up the container again, there won't be any data retained. And, you know, if you're following along, you're thinking maybe this has some implications as to the long-term sustainability of our Minecraft server. You're going statelessness. What was that? That's double entendre. It's pretty good, I'm sorry. Okay, so continuing on. Yes, there may or may not be a problem with our Minecraft server. If any time a janitor trips over a cord, they'll knock out everything, all our progress we made. But to get everyone on the same page, I'm gonna tell you a story just to make it more real to you. So imagine you watch this presentation. You were super enthused about it. You were like, man, it's so easy to set up Minecraft in a container. That guy who was talking, you did it in 30 minutes. It was so fun. You've been playing with your friends and family for two weeks. You've set up this massive castle. You've explored and created a very nice farm with tons of watermelons. You've established friendly relations with the local villagers.  Stateless.   There's a combination, Hurricane Zombie Apocalypse Firestorm. Again, a janitor could trip over a court, who knows? Your server goes down and two days later, AWS comes back online. Your lips tremble as you start up your container and your blood runs cold as you log in to see the empty expanse of a plane's biome. All your progress is gone. Your castle disappeared. Your watermelons never to be eaten. That villager that you named villager McVillageface, you'll never see him again. This is a disaster. We wanna play Minecraft with our friends. We want our state to be retained. How can we keep all the benefits of Docker, the easy deployment, the universal portability of a Minecraft container, but still retain our state between container instances? Proud will tell us exactly how we're gonna do that. So the burning question we now probably have is, how do we make sure that our Minecraft se