All right. We should be good to go, friends. How's everyone doing today? I'm glad to be here. This is my first open source summit, so very stoked. My name is Lee. Also, I'll be calling Nigel up here today later. And what we'll be talking about today is a little bit about login. Now, I'm from Colorado. I like parkour. And if you have met me in the community, it might be because I contribute to Kubernetes and to Flux. I'm a developer advocate with the Tanzu team. And I've been a dev advocate for about three years. But before that, I used to write a lot of software. And I used to write software that did operation stuff. So if there's any platform engineers or infrastructure people here, I share your pain and all of your experiences. And so let's talk about solving some problems, right? So this talk is about how sharing Kube configs and sharing credentials is not a really mindful way to do operations and to work with each other. Identity is a really important concept when it comes to doing shared computing to solve business problems or whatever your organizational mission is. And so we don't want to be in a situation where we're adopting new technology. We're doing cool stuff. We're innovating on how we iterate on our software. We have totally new things going on. We're like traffic shift in, playing with all cool stuff on this fancy infrastructure that's self-healing and doesn't page us in the middle of the night when a server stops for some reason. But then we're adopting this new technology. And we repeat our old mistakes. So I want to just kind of be a beacon for the community. And I think that in the open-source space right now in Cloud Native, we are building some really great solutions with some very open interfaces that has different tradeoffs. And I want to show you guys some cool solutions. So yeah, my name's Lee. I'll call up Nigel later. So the first thing that I just want to ask before we talk about Kubernetes and all of this other stuff with OpenID Connect and all these things, can I just get a little bit of a survey? Who's a Kubernetes user here? Pretty much the whole room, yeah. Some people are like maybe not participatory or they're just like, they like Docker sworn better. So how many people have integrated with some sort of authentication system before? Not authorization, which is kind of a different thing. It's about access. It's about login. Identity. So when I rushed in here late and I opened up my computer, the first thing I got to do is what? I got to touch my fingertip to this fingerprint reader and then the computer authenticates me. It's because it's really interesting knowing my identity because I like to write in my diary on this computer. It's very private information. And while I trust people with certain things, there's some things where it's just like I'm still trying to figure out who I am. I got private entries in there. And so that's kind of just for me. So even just in a one computer, single person instance, authentication identity is super important. So then you can imagine we've all been in that situation where we're starting to collaborate with our coworkers and we're like, oh man, I need access to that thing. We're doing cloud native software. That means that we have a lot of computers. It's almost too many computers to even be thinking about. We can only remember in our little brains, I've got like maybe this pool of servers over here and there's this whole complex system, but I'm going to just think about that as one thing because I'm trying to work on this completely other thing. Distributed systems. Oh, and I should probably page like Michelle over there, see if I could get like 30 minutes on her calendar because Jonathan and like the CEO, like they were asking for something and I just wasn't sure if it was going to be realistic to implement. And oh snap, I forgot. I told Larry that I was going to review his pull request after lunch and I didn't do that. And so we're doing all this context shifting and we're working with people and at the core of it, our systems need to know who we are. We need to learn and collaborate with each other because the problems that we're solving together as we're digitizing the whole world, they're problems that are bigger than any one person is capable of bringing the whole context to solve. We are collaborators, not just engineers. We build systems for people. And so we got to log into things. So I mean, we just got one computer and then now we're talking about many computers. What about Kubernetes? What is Kubernetes? Kubernetes is a distributed computer, but it's not a computer with an identity system, right? It has the ability to authorize things. It's got a whole API about access, about API groups and about verbs and who can do what. But it actually doesn't know who you are. It shifts out of the box like that by design because there are a lot of different interpretations about identity. And identity is the one place where in the cloud native world we need to be thinking about integrating with all sorts of systems because we don't want to be repeating the previous problems and duplicating credentials and having to get into this really nasty key distribution problem. So I want to talk a little bit about Pinniped. I had a slide over here that I actually accidentally deleted. And it talks about what comes out of the box inside of Kubernetes. You've got service account tokens, which a lot of people abuse and do all sorts of stuff. And then it also has kind of deeper in the infrastructure the ability to use a certificate authority to authorize and authenticate people. And that certificate authority, you know, you can print out mutual TLS certificates. But both of these things, I want to point out, as somebody who knows deeply about Kubernetes, that these are not things that are meant to be this yellow box up in the top left, right? Kubernetes, from an identity perspective, has some very lightweight options that are specifically engineered for bootstrapping Kubernetes components and running workloads and giving workloads identities. And so those things are not for people and identities that are outside the cluster. They're very scoped to the cluster. There's no such thing as certificate revocation inside of Kubernetes. Service accounts are not just some anonymous ID. They're actually intended to be mounted into pods automatically.  They might be a good way to get started if you have very few clusters. But if you're using certificate off and you're using service account tokens and you're handing out kube configs with secrets inside of them to people, and as your cluster list grows and as people come and leave and you cannot revoke these tokens or these certificates, which have a lifetime expiry date, then you're going to be in a kind of suboptimal situation. You're going to have to bolt some stuff on tops so that you can build these identity provider features. So when talking about Pinniped, Pinniped is an open source project that has two primary components that we can think about. So anytime I spin up a whole multiple Kubernetes system, several distributed computers in order to solve my business problems across the world, I'm going to be thinking, how can I integrate those computers with my identity provider? A lot of clouds have ways to do this. A lot of clouds are very willing to be your identity provider because if all of your access is glued to the identities of the person that you purchase infrastructure from, then you will never leave that person. It's called lock-in. But when we want to work across multiple computers, we're going to need this concept of identity, and that's so that when you have your developers, if your manager is interested in knowing what time you pushed out an update and doesn't want to ping you for stuff, we want to expose the information, the very valuable information from all of these different APIs inside of Kubernetes, contextually based off of access to the people with specific identities and memberships to groups. So with Pinniped, we can do that by deploying two components to a management cluster and then a single component to our federated clusters. And these lines over here, they're not network connectivity, it's just referential. So in the management cluster, the Pinniped supervisor does need to talk to the identity provider, but from a networking perspective, but from the federated cluster perspective, the concierge does not need to talk to the supervisor. So why do we have all these components and why are we deploying them everywhere? We have this fleet of distributed computers, and we want to achieve a couple of things. We want to achieve integrity of our tokens so that they are not bleeding across different boundaries in the federation domain, and we want easy access that's tied to the identity of the thing in the top left, it's the yellow box identity provider. So I'm going to show you just where you can go and get this piece of software already. I'm going to need another Chrome window for that.  Oh, I already have that, so I need to do this. Cool. I just go to Pinniped.dev and I need to be on the Wi-Fi or else everything is going to break. Sorry, friends, I could have been a little bit more prepared. That should work now. Pinniped.dev. So this is an open source project, right? Everything that we're talking about at mostly at this conference is open source. Batteries included, Kubernetes authentication, simple, frictionless, seamless. Why are we talking about ease of installation, ease of integration? It's because in order to integrate with Kubernetes, you need to influence the way that the API server interprets identities. That often means having control of the infrastructure of the API server of your cluster. But that's great until you want to start deploying geo-distributed services. I want a footprint. I want to be able to serve my users in Bangalore. I want to be able to serve my users in Seattle and then also have a presence in Germany. There's no one place or one cloud necessarily where I can go and purchase Kubernetes clusters or maybe I don't have the personnel, the data centers in those places in order to serve those geographies. Then I maybe need to use a Kubernetes as a service provider and that company is going to host my control plane. They may not give me control. Even if they do give me control over the way that infrastructure of the API server is configured, it may be a completely different interface and with different rules, maybe different product features than how I set up a bare metal cluster, how I set up a GKE cluster, an AKS cluster, an EKS cluster on Amazon. It's this really ugly problem. Now we have all this friction if we're deploying these Kubernetes clusters everywhere. It's supposed to be open source software with open interfaces. What's really cool about Pinupede is that you are able to integrate with any Kubernetes cluster without modifying the API server. This is a very, very unique feature of a authentication provider for Kubernetes that does not use all of the raw stuff underneath Kubernetes exposed to the user. I'm going to demonstrate what's cool about this. We have the supervisor, we have the concierge deployed to multiple clusters. There doesn't need to be network connectivity between the federated clusters. Then we just install the Pinupede CLI on all of the endpoints and clients that actually want to talk to Kubernetes. That means that if you could ask IT to roll out Kube CTL and Pinupede to your manager's laptop, your manager can check on your work all day. Or maybe you could use a UI like Octint or Lens. This is a really cool architecture and it's secure and safe. That means that when I exec Kube CTL, I'll show you one of these Kube configs. I'm catting out a Kube config on a terminal. The cool thing is nothing in here is secret. This is like a certificate authority bundle. We've got a couple of bits of information about how to reach the cluster, what contexts and things to use. Then there's this args and command section of this Kube config. The command is using Pinupede. Then it's calling login OIDC using the concierge and then asking for a jot token, making sure to validate TLS. Then it has a cluster audience right here. Here's this unique identifier for the cluster that this Kube config is tied to. This is a completely safe credential. I can use this and it will go and talk to the concierge. The concierge will go talk to the supervisor centrally, make sure that it'll use an open ID connect login flow, and then I will get a token for my cluster. I'm just going to remove the... Well, I'll just do it. For instance, in the way that I'm hooked up, let's connect to the management cluster, if I say Pinupede, who am I? Right now, I am using the admin Kube config because I am the personal person who made this cluster from this machine, so I happen to have my hands on the administrator Kube config. This is the kind of credential that you would normally use to access a cluster. Just right out of the box, this underneath, this is Tanzu Community Edition cluster provision with cluster API in Kube. You get this certificate, it does MTLS off with the API server. Now, I am this particular username and I have these groups. This is a group that everybody gets and this is a group that gives you a role binding, a cluster role binding to the cluster admin cluster role. If I were to use that special Kube config that I just showed you and then I say Pinupede, who am I? You'll see something cool happen. Assuming my network connectivity is up, yeah, cool. It is going and it is authenticating with GitHub. Then it's telling me I'm logged in succeeded, I'm allowed to close this tab. When I go back to the terminal, it says log in by visiting this link, it opened my browser for me automatically, optionally pasting the auth code, it didn't have to do that because, again, it opened the browser automatically and then it tells me that it's accessing this cluster using the normal API server that's been unmodified and that I'm this particular username, which is my GitHub email and then I'm a member of this GitHub organization, this GitHub organizations Kube admin team, this GitHub organizations Kube viewers team and a bunch of other stuff. Now, what's really cool about this is this is a tons of community addition cluster API, Kube-DM provision cluster. It's pretty much an upstream Kubernetes cluster deployed to some random Amazon EC2 nodes. It's not integrated deeply with the Amazon cloud APIs. It's not modifying Kubernetes in any way. I have Pinupede supervisor and concierge deployed to this management cluster and I'm able to talk to it in a way where it knows exactly who I am. For instance, if I use the admin Kube config to say get the nodes, I can do that. But as soon as I use that Pinupede Kube config, it says that I'm not allowed to do things anymore. So now I have a new principle. I'm no longer administrator. I've dropped all privileges except for what's normally given to authenticated users and this is something that a cluster management team, a team of people who have access to the cluster, can use to start allowing access to individual people from different groups. And the cool thing is that if I create a new cluster and I just install only the concierge to that cluster, then I'm able to use this exact same login flow with the exact same session token that's on my laptop. But then I'm going to get unique credentials for this brand new cluster that I install software to. So very powerful and not using any technologies that are coupled to a particular cloud provider or a particular way of installing, managing Kubernetes components. It's completely decoupled. So Lee, how does it work then? Well, if I do a Pinupede get Kube config and then I just I'll just write this to DevNol. Because I'm only interested in the debug output. Well, right here, this is talking to the concierge inside of the AWS management cluster that I have. And the concierge is telling me, oh, you have this credential issuer. And the concierge is operating in token credential request API mode. And that means that the concierge did some digging inside of the Kubernetes cluster and did some digging with how much access it had. And what actually is happening here is the concierge is minting short term X 509 certificates on behalf of an authorized by my OIDC claim. So I have these OIDC providers. I've got the concierge and the supervisor linked up. They're all talking to each other. And once Pinupede figures out who I am, the concierge on my behalf goes to Kubernetes and mints some credentials for me that actually let me talk to the real API server. So there's no proxies or any impersonation involved. I'm getting a credential that gives me temporary token access to the API. Now, if that's not available, there's different circumstances, you know, all the kinds of surprises can happen when you're deploying different kinds of Kubernetes clusters. Pinupede has a fallback mode that you can also just enable explicitly, which is this impersonation proxy. So instead of going to the API server directly, you can also use the Pinupede concierge as a intermediary. And you can make sure that you deploy it in a way that fits your security model. And then all of your API requests will instead proxy through the concierge as an optional deployment mode. And then that allows the concierge to use impersonation instead of the token request API, which is sometimes an easier way to support things when you're deploying it. So I made a kind cluster. It doesn't get more boring than this. These days, it's really easy to get plain Kubernetes. There's no cloud provider integration or anything. And let's look into my control repo. So in this control repo, I have a workload folder. Make this just a little bit bigger. That's probably too big. And there's a build of a Carvel project that I'm just using to manage my config. This auth infra is just a YAML file that you can get off the internet. I didn't write this. It's just the install of the concierge with the custom resource definitions and deployments and all the things in Kube public and Kube system that you need to install a basic concierge installation. No changed options or anything. And then this is the configuration that I need to add to a federated cluster. There's nothing secret in here. I just tell the federated cluster. Honestly, this could be a config map. It's just nicer as a custom resource. The JWT authenticator just tells the pinniped command line tool. Where is the issuer for the supervisor or whatever OIDC server I need? And then what is the audience of this cluster that I should request tokens on behalf of? And that audience being unique per cluster is important because that is the kind of like those are the bits that ultimately produce cluster unique tokens. So right. So this very simple configuration. I'm just gonna do I'll use cap just to make this a little bit easier, but you could do Kube CTL apply just as well. I think I've done this before actually. Yeah. Cap deploy. Config worth loads. So I will just deploy the infrastructure. So this is what a concierge deployment looks like. There is some cluster scope resources. They have the CRDs, a couple of different basics for cluster roll binding so that pinniped is able to pivot and do some secure things for you in order to manage identities intermediarily on your behalf.  So I'm just gonna say, yep, this is all security reviewed and actually part of real products that you can pay for. But all of this stuff is open source. And then yeah, a cap right here is just a little bit different than Kube CTL. It's just waiting on all of the resources to actually tell me that things are deployed properly. So I'm again, I'm deploying the concierge to my kind cluster. And yeah, I mean, there's nothing, there's no special networking or any storage being used. It's just a very lightweight shim that we're putting in there and it's got a downloaded container. I haven't tried this on conference wifi. Actually, this might be a bad idea. Cool, there we go concierge is deployed. And then I will different command that I ran earlier. So now that I have the concierge deployed, I'm just going to add that JWT authenticator custom resource to the cluster. And so this is just at the cluster scope, adding a jot off. It's going to make this resource. There you go. So again, if I ask pinniped, who who I've been to this point, I've been operating as the cluster admin using a mtls backed X5 on insert talking to the kind cluster. And if I instead do a pinniped get kube config, and then I overwrite whatever was there before, this is could not find a healthy agent pod. Oh, it looks healthy enough to me. I've never seen this error before. There we go. I must have not waited long enough. Cool. So here again, we're seeing concierge operating in token request API mode. And then I am able to say kube config. And now the kind cluster that I just deployed is going to go do some OIDC claims, make sure that I have my session updated. And then it knows exactly who I am. So even from an individual laptop perspective, right, I logged into this laptop, I own the compute. And so like, why is it so important that I'm logging into kind? Well, what you see actually in a lot of developer environments is that there will be like some shared virtual machine. And then somebody just really wants a quick way to get kubernetes and they run kind and explodes a load balancer on like all interfaces on the host just using some port forwarding stuff. And this is like a really convenient way to add some real identity to a development environment that's super quick and fast. Right. So very when we look at that, the promise that's here. And you're talking about anything that's open ID connect, anything that's talking about authentication, about TLS and all of the other mess of things that you need. And then also to deploy the supervisor, like you have to have like secure ingress stack with like it's got to be on a network that everybody can reach, which is probably internet unless you have a VPN. And like if you have been down this road, this is not easy and frictionless. Right. This page, you're doubtful, you're looking at it, you're cynical. This is, this is, there's no way that this is true. But once you got to that management cluster setup, once you have the supervisor hooked up to your identity provider, and I've just been, I've been saying OIDC, but we support a bunch of things in Pinnipet. I mean, like there's an LDAP connector in there, you can hook it up to Active Directory. That when you got that setup, now you have the ability to federate an authentication domain across any kubernetes cluster, regardless of who is providing it, regardless of how you created it, how you manage the API server, how long it's going to be around. It's easy enough and frictionless enough to, where if you make that investment to going back to the picture, if you do this and you can get the Pinnipet CLI on all of the clients, then scaling out federated authentication to, in a secure way, that's really nice and can tie into something like a GitHub identity, or like your Gmail account, or Google Workspace, or something from Active Directory, which like every enterprise in like the Fortune 500 has an Active Directory. Now you have a frictionless way to tie those identities into Kubernetes without doing really gross things, like minting a bunch of certificates that duplicate identities from elsewhere and having to distribute those kubernetes in a secure way using vault or a password manager or whatever. That kube config, it's got no secrets in it and every developer can use it to bring their own identity, which means that you can just check it into a Git repo. Like it is beautiful. Really nice synergy with GitOps and super fun to use. So that's sort of the meat of the demo. Some fun facts about this is if we go and like actually log back into the management cluster and then I'm using Flux here. These are the components that were ultimately necessary for me to kind of productionalize using a regular AWS cluster API provisioned cluster. The ability to deploy all of the TLS stack and the secure ingress necessary to kind of have OIDC end to end. And inside of here as well, there is another component that you can find in the cloud native ecosystem called DeX that does some similar things, has some similar overlapping goals to pinuped. And the only reason that DeX is in there is because it knows how to talk to GitHub's esoteric OAuth 2 implementation. So it just shimmed it in as an OIDC server that could talk to GitHub because GitHub doesn't implement OIDC. So in the final picture where we have the identity provider there, it's just in between GitHub and the pinuped supervisor. There is DeX in this particular demo. Pretty fun to hack on. And before we get into any other details and questions and stuff, I want to invite Nigel to come up and just talk a little bit about how this is really cool software that's not just something that we built for our product, VMWare necessarily, but that we intended to build in an open community way. So Nigel's Austin local here. If you're Austin local, go hit them up. And yeah, you can find them on Twitter. You also work with Contour. I do, yeah, yeah, that's correct. So hey everyone, my name's Nigel. I am a community manager at VMWare. I manage the communities for the open source projects, Contour and Pinuped. We're here talking about Pinuped today. So I wanted to invite you all to get involved with us. Pinuped's community is growing and we actively want you to be a part of it. This is an open source conference and open source projects don't work without communities. Community is important to us because, well, for me personally, I think that community is a necessity of the human condition and like our technology is just an excuse for us to find a place to belong and feel like we're contributing and feel like we're affecting the outcome of these projects. And regardless of where you are in your Pinuped journey, we have our governance document in the repo that describes the different roles from users to contributors to maintainers. Even if you're not actively using Pinuped, you're using some other solution, it's still helpful for us to hear your voice, letting us know why you choose Pinuped or why you don't or tell us about your use cases or come to us and get support, figure out how you can do all these cool things that Lee's been doing today with your own clusters. And we are actively soliciting your input for how we can help you be successful with the use of this project. My entire job at VMware is to create the conditions for a community to grow.  So I wanted to tell you a little bit about how you can find us. The Pinuped.dev website that Lee went to earlier has a community tab, has all of this information here, but I also wanted to put up here for you to see. We have our community meetings on Thursdays on the first and third Thursdays of the month. Community meetings are your opportunity to meet synchronously with our maintainers and with our other community members to learn about what's going on. And we invite you to join us every first and third Thursday at noon Eastern 9am Pacific. We have a Google group for email communication. So all of our announcements are requests for comments. We have a few PRs open right now that we're actively looking for folks to comment and tell us what they think or your contributions. You can find all that stuff on the GitHub page, but also we'll send updates out on the Google group. You can find us in the Kubernetes Slack workspace, just hashtag Pinuped and then interact with us on Twitter at Project Pinuped. We're here for you. We want to hear you. So please come join us. Thank you. Thanks, Nigel. Yeah, again, hit Nigel up. He's here and this whole control repo that actually is responsible for setting up the infrastructure is fully reproducible. I have destroyed and recreated this cluster many times with older versions of the Tanzu CLI making different AWS clusters, updating packages. You can read the history of this control repo. It's actually in a GitHub organization and if I actually go to that org, Stealthytale.  So in GitHub, like if you want to set up an OAuth app, you just go to your organization or your personal settings. And then I think you'd like go here and no, sorry, it is it's under developer settings under this and then are under OAuth apps. Actually, there's so many buttons. Yeah. So right here, like this is the OAuth app that I made for this demo.  This is where I configured the call back to DeX, you know, it's a couple of fields. And if you like know what to put in here, it's not that big of a deal. But sometimes just when you're setting up this kind of stuff, it's just a lot of research, right? Like, does this need us trailing slash or not kind of thing. And if you're really proud of yourself, you can upload your organizational logo here. But once you make your app, you know, you just generate this clients key client secret has an ID and secret combo. And then in my case, the secrets management for my repository is just in my repository. It's encrypted with SOPS. And I just have a values file for the management cluster here, where we update that. So here's the encrypted client secret, right? SOPS edit, you can see the decrypted version. The, you know, my private key is not committed to the repo. So that's like the kind of thing that you would. Anyway, that's about get off secrets management. I'm a get off nerd. But yeah, the whole point is this control repo, you can find it there. Go take a look at the code. There's a bunch of good stuff in there. I think I'm using like three different package managers. There's like vendor, which is just like plain YAML, vended into my repo, using a Carval tool. There's a YTT project in there.  And it's kind of gnarly, but that's the kind of thing that you need to do when you actually want to get organized on Kubernetes without introducing a bunch of entropy. And this is the kind of control repository that could scale to managing many different pieces of infrastructure with multiple teams via pull request, using your identities for when things go wrong, that you could take those identities to Kubernetes and have access to the things you need. So that's kind of the vision. And I think I have like a couple of minutes for questions. Yeah. So we just have to repeat the questions for everyone online. Yeah, for sure.  Does anybody have any questions either about Lee's amazing demo or about the community? Yes, please. So why Jordan? Kublogin. So Kublogin would require the question. Yeah. The question is why would you use pinniped over something like Kublogin? And Kublogin is an example of actually, is this the Rancher project or the open source project? One second. Let me get a refresher in Kublogin, because I don't want to say the wrong information. There's a couple of different shapes of things. So this is an OIDC authenticator. And I believe you would need to modify the API server would be my guests here. Connect. Yeah. So the reason that you would use pinniped over this is because you don't have to modify the cluster. So I don't have to go in this example. They're using GKE. I don't have to go into the GKE control plane options and do specific GKE things to configure my identity provider. So that's probably the biggest advantage of using something like pinniped. Also, we support more than just OIDC. So I don't know if the Kublogin project supports things like LDAP, per se. I know it's a very popular option, but you could use pinniped in the same exact way, but then just configure an LDAP connector instead of OIDC. Great question. Do we have anyone else? Everyone's good. You understood everything perfectly. But Lee's got all of his information up there or had all those information up there. If you have any questions and we're going to be around as well. Yeah. Yeah. I'm always happy to chat again. Would love to connect with you. Here's some stories about what you're doing. That's my Twitter. That's my GitHub. And thanks for for coming. And I'm learning with us. Oh, sorry. One more question. In this case, so the question is this works with really well with a Tanzu style cluster API inspired architecture, where you have a management cluster and worker clusters. And it does mirror that, but none of this needs to be any Tanzu Kubernetes underneath. You don't need any cluster API underneath. The clusters don't need to know about each other. I could deploy five different kind clusters on a bunch of different virtual machines in different clouds. And you just kind of pick one. That one is your supervisor. You host an endpoint anywhere where all of your clients can be. And then you don't have to do all of the other messy authentication glue in order to hook up with your identity provider on the rest of them. But then you just install concierge point them all to the central place. And then you have off that works. This does beg the question, which is like, what about failure domains and that kind of stuff? And with authentication, we're always that's always a really important concept, right? So that central supervisor endpoint, the failure model of whatever that is, is quite important. And it might be one of the reasons why you choose a different solution. If like, you don't have the operational expertise or the services available to you to ensure the availability of that supervisor endpoint. There's no, sorry, I don't I understand. Oh, voting. Yeah, the all of the kind of like leader election and resiliency and like cross AZ type stuff. You could just delegate to Kubernetes for that. So pinniped doesn't have to do anything inside of the project. You could just host a really resilient management cluster that doesn't host workloads that doesn't have ingress traffic from the internet or anything like that. It's just for authentication services and whatever other central things. It's also a good place to probably put like a secret store like Vault, you know, because this is authentication, right? So it's like, it's secure infrastructure that you want nothing else touching. And then that way you could also then have a multi AZ cluster. The costs are not going to be very high. Fail over is going to be based off of the fact that that multi AZ cluster is very resilient and backed by at CD. That's a great question. Thanks for the question. Cool. Well, I'm super happy to talk and won't steal much more time for you. I think on the schedule right now is a coffee break. So see you out there. Thanks, everyone.


We demonstrate second-order interferometric autocorrelation of a pulse in the vacuum ultraviolet VUV spectral range using an optical arrangement equivalent to a Michelson interferometer, which allows for spatial selection of ions created by non-linear two-photon absorption and enables extraction of temporal phases in addition to pulse duration. This article was offered by T. Gebout, D. Rompotis, M. Wheland, and others.


three o'clock, nine o'clock. Our goal is ten to two, but we're gonna start, and it doesn't matter which hand is over at the other hand, but we're gonna start with a head of mace at six o'clock. From there you just tend to rock it, and you rock it not by using the straight of your arms and shoulders, but actually by moving with your hips and torso. Down the row, when you're able to do this with a significant amount of weight, it's your hips and torso that will move. Not your arms and shoulders. Learn that today, day one. Don't learn that here is when you move that elbow up injuries. Okay? This part has to move, where you going? See right there, you're doing perfect, you're doing perfect. Starting at six, five o'clock, seven o'clock, five o'clock, seven o'clock, seven o'clock, seven o'clock, seven o'clock. Okay, that's easy for you. Go four o'clock, eight o'clock. Four o'clock, good.


All right, we're about to get started. Just a couple of minutes while people are finding seats. You're finding a little bit of space. Please do come and enjoy some fantastic tacos. We have a taco truck just outside of the convention center, dropped by for breakfast tacos in the morning or lunch tacos. I'm a native to Austin, and I'm so excited that people get to experience the wonderful thing that is breakfast tacos.  That's how ecologically friendly we are here in Austin. You can also enter to win. We're raffling off one of these orange boxes. Actually, not either of these, any of these from my demo. But we have one, brand new and ready to go. If you go to this URL, you will be asked to submit to fill out a little survey, and you will be put into a drawing for an orange box. I had a bit of a hand in designing the original orange box. This is an incredible machine. It's like your data center shrunk down into a suitcase. There's 11 nodes in here, one's a master node. There's 10 more nodes, and it's great for development, for prototyping. You can take it anywhere you want, plug it into any outlet anywhere in the world, and run a little cloud. Now, on that note, we're going to talk here today about how we are creating the world's fastest open stack. What do all of these have in common? The underlying technology that makes all of these go fast is fundamentally containers. And that's exactly what we're bringing, have brought for quite some time now, to Ubuntu OpenStack is containers, our containers. And that's containers at multiple levels. Now, when we think about containers, and we really have to take a step back, and we think about virtualization, we think about that first generation of clouds, whether it's VMware's vCloud, or something like an Amazon AWS that's built on a type one hypervisor, Zen, where instances themselves are running inside of virtual machines or para-virtual machines, or using another hypervisor like a VMware. And then there's another generation of clouds and plenty of clouds that use underneath at the plumbing layer a second type of virtualization, a KVM, or maybe on your desktop you're using a virtual box or vagrant or another layer on top of that. Well, there's actually a third type of containers or virtualization that's driven by containers, and that's really an incredible thing. Was anyone at the OpenStack summit in Paris? Does anyone remember Paris? So I had the distinct pleasure with my colleague Taiko of actually unveiling Canonical's plans around LexD. It's something that we worked on for a long time in something that we called LXC, Linux Containers, and that was work that was originally led by IBM that Canonical took over and has continued shepherding the Linux Containers project, linuxcontainers.org, but we'd been working for a few months at that time and we're ready to unveil the concept that we wanted to take Linux Containers to the next level, and that is the concept of LexD, a hypervisor for containers, but not docker containers, not process containers. We're talking about machine containers, okay? And that's a big difference. It's great to put one process, just one process inside of a container and say, hey, you are my Apache container and I want a lot of Apache containers and you are my MySQL container and I want a couple of those in a cluster, but if you want to run Apache and MySQL in the same container, that's not a process container, that's not an application container, that's a machine container at that point. Machine containers boot an entire operating system, right? We launch Sbeninet, we run Syslog, we run an SSH theming, we run all of the things that a normal application, a traditional application that expects to run on a Linux machine to run. And that's very different than an application container, which certainly has its place in a microservices architecture where you're going to chop up every piece of your architecture to do its one thing and one thing very well, but machine containers, you can treat just like a virtual machine. You can allocate memory and disk and network and IO to that and you can put limits, upper and lower bounds on it. You can guarantee that it has so much CPU and memory and disk. You can limit it from taking over the rest of the system. Those are machine containers and that's what we build our Ubuntu OpenStack on top of and what we put into OpenStack as another hypervisor. So machine containers kind of fit this really nice area right in the middle of, they're like physical machines, right, you know, that you shouldn't really know the difference between running a process inside of a virtual machine or a physical machine or container, but they're like virtual machines. They spin up really fast. They share resources with the underlying hardware, the underlying architecture and at its core, at its core it's using the exact same primitives that we use everywhere else in the world that we do containers, Linux containers. It's C groups under the hood. We're using Setcomp and AppArmor for the security and the protection profiles around it, discretionary access, mandatory access controls. We relaunch these containers as non-root users by default and that's extremely important. You're gonna see me do a few demos here in a few minutes and you're not gonna see me use sudo anywhere, why? Because those containers are running as unprivileged containers. Root inside of that container is not root outside of that container. Dustin inside of that container is not Dustin in another container. It really is a secured environment. Now very important to making all of this work is the fact that LexD is a REST API. It's REST API driven. So this sort of architecture shows you how we deploy OpenStack across these machines. So in each host, and those hosts in these cases are physical hosts, they're physical machines. There's ABC and we could go on and on. On each of those hosts we run one Linux kernel. Now that's really the beauty of containers and how we can achieve 10 to 20 times the density using machine containers as compared to KVM virtual machines. Because we have one kernel that's shared across all of the instances, all of the container instances on that system. So if I wanna run 60 LexD containers on one node, there's one Linux kernel for the host, there are 60 guests that are all sharing that same kernel and then they're each allocated their own disk space and CPU and memory. If I wanted to run 60 KVMs on that system, I'm gonna run 61 Linux kernels, one for the host and then we're gonna boot 60 more Linux kernels. Now for Linux on Linux workloads, that just doesn't make sense, right? Now if you wanna run Windows on Linux or Linux on Windows, of course you need a virtualization layer. Actually that's not true. Anyone heard about that Ubuntu sub system running in Windows? I need to update this deck now. That's a really crazy strange example that we won't go into here. But for the Linux on Linux workload, containers and container machines fit that model extremely well. Now each of those LexD processes, the LexD process is a single process. It's a demon, that's what the D in LXD stands for. It's part of Linux containers, LXC, but the LexD is the demon that watches all of the containers on the system. It's the single entry point for an administrator to talk to and say, list my containers, start a container, stop a container, back up a container, snapshot, restore, clone, live migrate, limit the resources, reconfigure. That's the one entry point that we have for all of those. And for the eight plus years previous, prior to LexD, of the LXC project, we never really had that. Those of us who knew what we were doing could type a really long command or we'd write a script or sort of wrap that in an interface, but LexD gives us that. And it's not just a command line interface, there's actually a REST API for that, which really fits well into the open-stack model of being API driven, so that we can have external utilities authenticate. Everything that LexD does is TLS authentication, SSL. We authenticate against that daemon and then we run whatever we need to run against it. We query it, we execute actions. And with that REST API, we then can write all sorts of applications that talk to a pool of LexD machines. We've done that ourselves, of course, with the Nova LexD project. So Nova LexD is an extension to Nova. It's a hypervisor plug-in for Nova that allows LexD to act as another of the hypervisors that Nova can manage. So Nova, of course, has always had support for KVM, which isn't going away. We still love KVM for full virtualization workloads. You've got other drivers for VMware and even Hyper-V. And LexD sits alongside those. The beauty of LexD is it can run anywhere you're running Linux. It doesn't require any hardware acceleration. It doesn't require VT. It runs in virtual machines. You can run Linux containers inside of a virtual machine. And the promise of recursive virtual machines inside of a virtual machine has never really come to. But we can run extremely efficient Linux containers inside of VMs. So we can deploy all of OpenStack into a public cloud for testing purposes or even some fairly useful use cases. The LXC CLI uses the REST API to talk to it. It'll talk to local hosts. It'll bind a socket to local hosts if you're talking to your local LexD. But you can also query and communicate with any other network-addressable LexD daemon on the network. Now what makes LexD really go fast in Ubuntu is ZFS on Linux. It accelerates the entire experience. Primarily, primarily because of the copy-on-write snapshotting and cloning. So underneath LexD in Ubuntu, you can configure, it's one command, LexD init. So you do run that as sudo because that is a decision that's shared across all the users using LexD on the system. So sudo, LexD init will prompt you for a couple of questions. If you answer the question, yes, I want ZFS. Yes, please, I want my containers to go fast. We use ZFS. ZFS has extremely efficient deduplication and snapshotting. So you pull down an image, that first image you pull down from the LexD repository. You can run your own repository. We put up public images for a number of OSs.  You download that once. You've got one copy of that inside of the image store that LexD manages and it's stored in ZFS. And then every time you start a container, you make a copy on write snapshot of that. If you don't use ZFS or another copy on write based file system, then you end up having to extract that entire copy and store copy after copy after copy and your deduplication is not present. You don't have a dedupt file system at that point. ZFS also does continuous integrity checking. You can know immediately if there is a problem and it actually can repair and recover a number of common problems or common file system problems and you won't even notice it. And efficient compression. You can actually store this compressed. There's a CPU cost to that, but if CPU is cheaper than disk, that's a personal decision. But ZFS is really what's gonna make these demos here in a second go fast. So LexD is fast, secure, efficient. It's extremely fast due to the fact that we're sharing a kernel, the fact that we're using copy and write snapshot so both storage and compute are very quick. It performs just as fast as running natively on the underlying hardware. It's efficient and allows you to use your underlying architecture very efficiently, get the most out of your machines. And then it's secure. We run as a non-root user by default. We have discretionary access controls that extend that non-root user into the host file system so that if a user does escape, they can only read and write with that non-root user. Usually a nobody user is able to read and write. We use mandatory access controls. We have app armor profiles that enforce mandatory access controls, ensuring that again a user that escapes that, that process has no more privileges than the unprivileged process that it was before. And we apply all that by default. So why is this the world's fastest open stack? And I'm gonna go into a couple of these. First of all, our hyper-converged architecture deploys in minutes. The conjure up tool that Mark demoed earlier, I'm gonna run that again, I'm gonna kick that off. The guys have been working on these boxes for a couple of days now. How many dozens of clouds have we deployed? We've deployed open stack hundreds of times on these machines in the last three days. Why? Because we can do it in minutes. We do it 10, 20 minutes and it's consistent. It's the same every single time. We can launch lexity instances in a matter of seconds. I'm gonna do that for you here. In a second Mark launched a few through Horizon as well. We can snapshot the entire environment in under a second. Actually it takes me about two seconds. I should update that. In two seconds we can snapshot the entire open stack environment. I'm gonna do a really nasty thing into one of my open stack nodes. The node that's running Glance or Keystone and whichever one we choose here in a second. We're gonna destroy it and then restore it from backup. The instances themselves that we run in open stack perform at bare metal speed. Equivalent to bare metal. We're gonna run a benchmark that demonstrates that. And then we can migrate services in real time. We can take an instance running on one node and move it to another node as fast as we can move that data across the network. And at gigabit speeds on these little machines, that's pretty quick. In your data center where hopefully you're running 10 gigabit or 40 gigabit or more, that's essentially real time. That's basically instant. That's important because it's fun to stand up those dozens of clouds that we've installed here. Install it, destroy it, install it, destroy it. But as our friends at Deutsche Telecom know, it's not about just standing up a cloud and then throwing it away at the end of the day. It's about standing up that cloud and keeping it up for months, years, decades maybe. And to do that, you've gotta be able to manage it operationally over time. And an essential tenant to managing it is upgrade. The open stack that we've got deployed here is OpenStack Mataka. Who's running Mataka already? Awesome. We're running Mataka already. It released last week, right? We're running Mataka, but at some point we're gonna upgrade from M to N to O to P. And we need to do that over time. And the way we upgrade OpenStack is we upgrade Ubuntu, of course, underneath, and then we need to upgrade OpenStack. And the way we can do that is by migrating services off of a system because they're contained, doing an upgrade, and then migrating it back. That's really important to the operational efficiency of a real OpenStack in production. To be able to evacuate an entire rack of all the instances running, all of the OpenStack services running, perhaps perform some physical maintenance on it, replace the RAM, add some CPU. And then bring it back into commission and put those services back on it. That's really why LexD is super important at that layer underneath the cloud. So at this point, I think we're gonna do a demo. And because I can't work on a command line with a stuffy sports code on, I'm gonna take that off. All right, here we go. So first of all, I'm gonna need to probably refresh a couple of web pages and see my authentication keys get timed out here. So let's log back into this one. We're gonna see we have an OpenStack already up and running. And in this OpenStack, if I go to the administrative tab and I look at the hypervisors, OpenStack Mataka, by the way, I look at these hypervisors. I've got three machines that are dedicated hypervisors. And the type of hypervisor is a LexD hypervisor. And that's really cool because these are little machines. These are Intel Nooks. And each one has 16 gigs of memory and merely 160 gigs of SSD, nothing like a real server. And we'll very quickly grant you that, but we can't really bring real servers or it doesn't work out well when we try to bring real servers into a room like this. But we can run on these very small machines and we can run a lot of containers on those machines. I've run as many as 636 LexD containers on an Intel Nook, on a 16 gig of memory Intel Nook. And that's because for those 636 containers, there was one Linux kernel running. It took about 20 minutes or so to boot all 636 of those containers. And about, sorry, that same amount of time for every one of those containers to draw its own IP address, its own addressable IP address. I'll admit that I had quite a bit of networking work to do to rearrange things a bit to get enough of an IP range to handle 636 unique IP addresses. And then once I ran it a few times, you can crank through several thousand IP addresses in a matter of a few minutes, as I said. So we needed a really big network, but doing that, you can do this at home on your laptop running Ubuntu. The second thing to look at is what kind of open stack we're talking about. For this demo, we're talking about a relatively basic open stack. This is the core of open stack. Jonathan, in his opening keynote this morning, mentioned compute network and storage. We've got Nova with an Lex-D driver. We've got Ceph for a storage backend providing both object and block storage. Neutron with a basic open V switch. MySQL and Rabbit, Horizon and Keystone. And that's really a core open stack. It's not a lot of bells and whistles here, but for our customers and partners, we can create some really interesting open stacks. We can model it first and deploy it using all of the same tools that I'm talking about here. This is what we're running and how they interact with one another. Each of those lines is an interaction. It's a Rabbit MQ providing the message queue for everything. It's the Percona database, the MySQL database providing a storage backend and so on. Okay, so the first thing I wanna do is I'm gonna kick off a conjure up. So conjure up is essentially a chaperoned experience for deploying open stack. It's a little like dev stack from one perspective in that it's one command that you can run on one Ubuntu server and deploy all of open stack on that one machine. But interestingly, it can also drive deploying that same open stack to many machines. If you have a Maz, it can deploy them straight to bare metal, as much bare metal as you want. And you can also switch from this slick command line interface to the much prettier autopilot, the graphical interface, the GUI through the web that we saw a little bit earlier. I'm going to deploy open stack with Novolex D and I'm gonna do it all on this machine. You'll see here that it's telling me that I'm running this on the Mataka release. And this is gonna take a few minutes as it bootstraps. It's gonna start up one node where we're gonna instantiate a juju controller and then that juju controller is then gonna spin up, I think about 16 or 15 other Linux containers, LexD containers, on the same system and deploy an open stack service one per in each of those containers. I happen to have that already running here on this system. I'll make this just a little bit bigger and I've got a checklist of all the things that I wanna show and there's so many that I had to make a list for myself so that I don't forget any of the goodness. So the first thing is to look at what's actually running here. So I'm gonna run juju status and pipe that to less and we'll see, this is a command line view of that same graphical web interface that we just looked at. So this is a graphical interpretation. This is the sort of text-based interpretation. You'll see three pieces of Seth, Glantz, Keystone, MySQL, Neutron, Nova, a dashboard, Horizon and Rabbit. You'll see how all of these are related and that's the, again this is the sort of text interpretation of all these extremely important lines and what their status is at any given time. And so all of these are related to one another. Now, how does that translate to LXC? I'm gonna run LXC list. I think I'm gonna need to pipe that to less as well. These are all machines, LexD machines. Every single one of these is a machine that has booted running Ubuntu Xenial, Ubuntu 16.04 which just released last week. You'll see every one of them has its own IP address and that's an IP address that's accessible to all of the machines on the network, at least at this level. Ignore the Snapshots column for now. Now that I've pointed your attention to it, you're not gonna be able to look at anything other than the Snapshots column, but we're gonna look at Snapshots and restore one in a second. Note to self, don't point out what you don't want people to look at. So all of these machines are running and I can pick any one of these at random. I don't even know what this one does but we're about to find out. So I can do an LXC exec, that machine and we're gonna go into bash in that machine and if I run a PS, we can make a pretty good guess as to what this one's running. This looks like one of our Nova compute nodes. Wow, how does that work? It works like this because LexD inside of LexD just works. We can run containers inside of containers inside of containers inside of containers and so on turtles all the way down and every single container performs at the same level as if it were running on bare metal. Why? Because it's actually running on the bare metal. These are just processes, process groups running on the host kernel. I was hoping not to look at a Nova compute node so we're gonna do that again and pick another one at random and we have a three and 16 chance of getting another Nova compute. So let's take a look at this and see what's this one running. Ah, that's a little bit smaller. This is one of our Ceph machines. We see Ceph down here at the bottom. So this is what it looks like inside of a LexD container. PS shows what looks like a system, that's booted. You see the first process, PID 1 is S been a knit. You'll see that we're running things like logging and simulating debus and R-Sys log which is important if you wanna get any information off of it. SSH, it's kinda nice if you want this machine to feel like a machine to be able to SSH into it like a machine. I'm sort of taking the fast track into the machine by using LXC to exact bash in the system but we could just as easily SSH in if we had our keys appropriately installed. So let's look at the storage underneath this machine. I said ZFS earlier. So if I do a Z pool, points for typing password right on the first try, we see we have our ZFS pool. The pool's name is LexD. It's online. I love seeing this line at the very end. That's always extremely important. No errors, right? Sudo's Z pool list. We'll show that this is a 200 gig Z pool of which 30 gigs is used. That's pretty amazing considering how many machines and how many snapshots of those machines I have here. I've got 16 LexD machines running. Everyone running its own copy, its own unique copy of Ubuntu 16.04 and I had five or six snapshots of every one of those containers. So we're talking about copies of over 100 copies of an operating system there and that's part of what that Ddupe, that copy on right, is really doing for me here. And then I can get a little bit of data out of this. If I do a C pool IO stat dash V. I thought I had, thought there was more information there. But what we can see is a bit of what's going on. If we were to watch, we were to watch that over time. Now launching an instance. It's just as simple as this. So if I do an LXC list, you can see currently all the, all the LexD machines that we have running were generated by Juju. Juju took names that were these, these UUIDs and sort of enumerated them. If I just wanted my own container, I could LXC launch. I'm gonna launch an instance, an image called Ubuntu Zennial. I'm gonna throw time on the front of that. So we'll see how long it takes to start these up. And this is about a three second process. That's three seconds to boot, to allocate this machine, boot it, run Sbeninit and draw an IP address. That's remarkable. We can do this over and over and over again. This machine is pretty heavily loaded at this point. Some of the stats you can see at the bottom in yellow, we can see that the system load on this little machine is seven, that's on a four core system that on 16 gigs of memory, I've got about 78% use. So this is a pretty heavily loaded machine but it's still quite responsive and I'm still able to do, it's running all of OpenStack. It's doing quite a bit here. And now at this point, if I LXC list, we should see at the bottom of our list those three new containers that we ran. All three of those containers have their own IP addresses and we can get into any of them.  There's no workload deployed on it but it is certainly there. So let's play with the limits. Let's take one of these systems. Actually, I'll delete these three. So let's delete dash F, that one and that one and that one. Reclaim some of those resources. And let's limit the CPU and memory for one of these systems. So we're gonna take this one at random. Clear the screen, LXC, exec, that. That's cat proc CPU info. And grep processor. And I'll see I have four processors. That means that this instance is able to see all four processors on the system. And let's run, let's see how much memory this system has. So this machine has, this is one of the ones I've already poked, this machine has access to 256 megs of memory. So it's not pegged on the CPU side but is pegged on the memory side of things. So I'm actually gonna set the limit of how much memory and CPU this machine can use. So I'm gonna hop over to my cheat sheet here as I've forgotten the exact command. And Tyco's gonna help me with this one. What's the limit? Limits.cpu equals one on this machine. It's on that machine. Limits.cpu equals one limit.cpu. One of those is gonna get me there. Thought I had a screenshot of that. Helped me out with that one Tyco in the meantime. We're gonna get that one. I do that one all the time and I've just forgotten the order. In the meantime, what we'll do is benchmark. We'll benchmark one of these. So on the host system here, I'm gonna run sysbench, testcpu, run. This is gonna execute a system test against the physical machine that's doing this host. It's gonna run 10,000 operations. And this typically takes, in this case, it took 10.7 seconds at this point. If I exec into any one of those machines, let's pick one here. And I may need to apt install sysbench because I doubt that that's gonna be installed already. I'm gonna benchmark the CPU inside of this container and see how it compares to our 10.7 seconds just a minute ago here. So now we're gonna run the same sysbench, test equals CPU, run. And we wait our 10 seconds. Now what's important here is that from a user perspective, you are a user of an open stack. You want an open stack that took a little longer. Let's try it again. What's important here is that you as a user of an open stack that gets allocated some instances, you want those instances themselves to perform as fast as possible as if they were on bare metal. There is a little jitter. We saw 12 seconds here. We see 10.6. What we're seeing is close to equivalent performance as running on the physical machine. In fact, what we see is, we see instances where customers or users want to use LexD as an alternative to allocating whole systems to a user for some period of time. There are certainly other alternatives to doing something like that. Ironic comes to mind where with Ironic you've got a driver that plugs into Nova. Nova can allocate a physical machine that gets installed and then that user receives access to that system. There's a lot of problems with that. That user with physical access to that system can do really nasty things like overriding BIOS or injecting Trojans. It really gets ugly and it takes a long time sometimes to reboot a physical machine into a... Reboot a physical machine into a... I can see everyone now. Reboot a physical machine into a brand new newly installed system. With LexD what you can do is say I want to run one container and only one container, LexD container, on that physical system. The startup happens in seconds as opposed to minutes to reboot and reinstall a physical system. You can unconfine the amount of resources that the user who's been allocated that LexD instance so that they have access to all 64 CPUs and all of the terabyte of RAM and all of the multiple terabytes of local disk so that it feels like that user has access to a big physical system and their processes are running on the host kernel on the physical hardware. But with LexD we can still prevent things like writing to certain devices, overriding the firmware, overriding the BIOS. And it's really a great model for the I want a whole physical system but I want the API and the manageability of an open stack. All right, so what we're gonna do now, Tycho check the syntax for me. It was the equal sign? Limits, there we go. And I could have just read the man page, right? And then let's change the memory. Limits.mem, mori to 512 with an m, mb. Geez, that's why we need machine learning to figure out how users actually interface with a system. All right, so we've now set the memory to 512 and the CPU to one. So now let's go back into that machine and let's check our processors. We now see exactly one processor, which is all the CPU that this machine's gonna receive, right? Let's check how much memory is available. We see we have a total of 512 megs of memory. So we've shrunken how big that system can be. We've limited how much, I don't even know what this system's doing, let's check. This is one of our Ceph nodes. So the Ceph OSD, it doesn't need access to that much memory or CPU to do what it needs to do. We've now confined it a bit. Let's snapshot and restore. How much time do I have left? How am I doing on time? Yep, all right, so I've got five minutes or so. Let's snapshot and restore. This is pretty cool. So let's take our list of LXC machines. I'm going to get just a list of machines themselves. I'm gonna loop over those XRGs. I'm gonna loop over those. Ready?  And in the time that it took me to say that sentence, we now have a snapshot of my entire OpenStack infrastructure. If I do an LXC list, we're gonna see that column that I told you not to look at earlier. You can look at it now. It went from five to six. I have a snapshot of every one of those containers. So now let's do something really nasty. Let's go to this machine. We're gonna go into that machine, which is running, oh, I don't know, this one's running, it's one of my units of staff. Let's do something bad. That's not advised, right? We're gonna do it anyway. And now we're gonna try to go back in and holy cow, I can't find Ben Bash. What happened to it? Well, it's a good thing I can restore from backup. So we're gonna take this system and we're gonna LXC restore. I need a backup name. So if I do an LXC info on that container, we're gonna see a whole bunch of snapshots taken. And I wanna restore snap three that was taken just a few seconds ago. And then I couldn't even start the sentence and it's already restored. So now we're gonna go back in to this machine and wow, look, Bash is up and running and we've got our OpenStack back. How cool is that? I think we're out of time. I've got maybe time for one or two questions. There's so much more to say here. This is really exciting. Of course, we can launch instances in OpenStack. Let me just launch some instances while I'm taking this question because that's always fun, right?  Absolutely, I can, so the question is, can we limit the root disk size of each container? We can currently limit the number of CPUs, the amount of memory, the amount of disk that any container consumes, the network IO, that's another fun demo to throttle the IO, do a big W get and limit the IO and watch the W get slowed down and then unlimited and watch it speed back up, and disk IO, IOPS essentially. Anything else, Tycho? That's it, those five. Yes, sir? What's the, the question is, what's the relationship, if any, to the Open Containers Initiative? So the Open Containers Initiative was sort of a, it was, it's a project led by the Linux Foundation. It was really meant to help bring Docker and Rocket, Docker and CoreOS together to, I guess, kind of stop a container war that was about to take place in the West Coast. We're supportive of that. I think the Open Containers Initiative has two goals. One is to establish a container runtime, application container runtime, and a second one to define an application container format. The containers that we're talking about here are machine containers, and so it's a little, it's a little different. The run time for us is boot. Run has been a knit, right? The format is a root tar ball. We didn't need a foundation to help us come to terms on that, frankly. If something comes out of the Open Container Initiative that we should support in LXD, we're absolutely supportive of it. Good question. Yes, sir? Great question. Everything that I showed so far here was Ubuntu on Ubuntu. The question is, can we do other OSes on Ubuntu? And the answer is yes, unequivocably yes, we can. The only limit being, if you're running user space code, that depends on a kernel feature. That's a, you know, a rel or CentOS or Fedora feature. No, but for doing user spacey stuff? Absolutely. We've got customers running scientific Linux, CentOS, Debian, inside of LXD containers. No problem. It does, but you know, there's a lot of code out there that doesn't depend on the kernel. You know, I've been around Linux for a long time, and there was a time when the kernel was the most, the numbers behind the kernel was the only thing you cared about, right? Thankfully, the Linux kernel has stabilized so much that it just kinda works usually. You might wanna tweak on low latency or tweak here or support for a video card or a network card or something like that, and that is important.  Yes, sir? Why ZFS instead of ButterFS? Does anyone else have that question? You're welcome to run ButterFS under LXD. We've got great support for it. I ran ButterFS for 18 months before I started using ZFS. ButterFS is native in the Linux kernel itself. It's been around for going on 10 years.  It's not a pretty experience. It's better now. It's better now, indeed, indeed. That's why, you know, you're welcome to use ButterFS. We've got support and guides on how to use just a basic directory underneath, ButterFS underneath, ZFS underneath. In our testing, we found ZFS to be bulletproof, just rock solid. And it's been a great experience for us. Thank you, though. Please do try ButterFS and let us know. File bugs you can get hub.com slash LXC is where you can find, I should put this up as well. So where you can find more information about this work, please file issues. If ButterFS is not working well enough for you, we'll look at it from an upstream perspective. Over here. Yes, sir? Wow. I have no answer for that question in this context right now. Grab me afterward and we'll talk about it. The question's about LexD and networking and specifically VLAN networking. The beauty of LexD is you just tell it what bridge to use and it just uses it. So as opposed to a whole bunch of code inside of LexD trying to recreate SDNs or provide network functionality, LexD just says, give me a bridge and I'm going to allocate, I'm going to point instances that boot to DHCP off of that interface. And that bridge can be a local bridge, which is what I have here. It could be, you know, it could be a Melanox or a Juniper or a Contrail or, you know, anything, anything like that. Excuse me? It does not have to be. Absolutely not. It just has to be able to communicate through that bridge. By default, it's going to DHCP and request an address. But if you've got a static route and you can get out through that bridge, go for it. Yes, sir? In the back. Yes, sir? Each vendor has its own distribution of. Each container has its own distribution. Each container boots a root file system that is an OS, Ubuntu 16.04 or CentOS 6 or Fedora 20, whatever. Yes, it's just a root file system that makes up that container. Yes, sir? Is it possible to run the container inside of LexD? Is it a Docker container inside of LexD? Yes, yes, yes. You can run Docker containers inside of LexD. If you check out my blog, I demonstrated that last week running on a Z mainframe.  Inside of that KVM, we had Ubuntu. Inside of Ubuntu, we had LexD. And inside of LexD, we had Docker. And it all just somehow works and works well. You can use constraints. How are we doing on time? Yes, sir? Everything that is supported by LexD is supported over the REST API. The REST API exposes everything. It's not like the REST API we bolt on after the fact and we add a feature and then for a while it's not supported in the REST API and then one day someone opens the, no. The REST API is where the interface into, that's where the command line tool talks to get directly to LexD. So REST API, first class citizen from day one. Last question, then I'll step out and answer some more. Yes, sir? That's absolutely right. So the question is, is the snapshot and restore utilizing ZFS? And that, yes, absolutely. So LexD does have a little bit of knowledge about the underlying storage. It knows about ButterFS. It knows about ZFS. And when it needs to perform a snapshot, it asks that underlying storage system to perform a snapshot for me on its behalf. Yes, so there's another slide here where in just talking about the process containers versus the machine containers, I'll leave you with this, the LexD source code is about 28,000 lines of go. 28K lines of go to create essentially a hypervisor that turns containers into virtual machines or machine containers. By comparison, the Docker source code is over 400,000 lines of go. They're both written in go, so from an apples to apples perspective. And part of the reason why is because LexD's defined mission is to boot a machine. That's it. It just wants to boot a system. Everything else is left to the OS running inside, to the administrator on the outside. It's really trying to stay out of your hair from that perspective. So from a storage, we don't have a whole lot of logic built into LexD itself. We're using the underlying system and to the network point, the same thing goes. We didn't bake a whole bunch of network knowledge into Docker, excuse me, into LexD, where we could just leverage good networking outside of that container. Thank you so much for your time. I'll step outside here in a second.


Banks have begun talks with the Nigerian National Petroleum Corporation, NNPC, on how to lend the corporation loans. The loan is to acquire 20 percent equity in the danglety $19 billion petroleum refinery. NNPC Group Managing Director, Millicuri, revealed that NNPC will not spend the federal government funds to acquire the equity but will borrow for the cash flow because they believe the business is viable and will return dividends as a refinery business. Careers of the NNPC and the banks have completed all the international processes for evaluation of the refinery and they are comfortable with the value of the business.


My module of the month this month is config bar sir, all right. Oh, uh, well the pros in the room It's in lowercase in python 3 and python 2 is it was configured capital C and p of capital p right, so Who here knows what an any file is? right Kind of let me just type one out Let me print you right so any file basically is Uh, you have what's called sections every section header is between these square brackets here And then in each section you'll have variables with a value. Okay, so separated by an equal or a colon So in this case, I have two sections one called db one called app I don't know. Maybe i'm trying to configure a web app in my db. I'll have a variable called host variable called port App add this little comment. So this comment is just for me It's not read in the configuration file It's ignored by most part by the good parsers and it's just for my edification right and Then a variable right so Any files is Basically go figure a way to store configuration as opposed to a lot of people would say use jason Some people others might use xml The glory and the the the benefit of any files is that they're very very simple to write out To and to use and arguably compared to say Jason you can put comments in it. Yeah, and the parsing is a bit friendlier A bit less error pro But uh, let's see how we can use any files in python. So I import config parser Little cheat sheet So I don't I don't type these from memory. I have to read them from a sheet So I create what's called a config Create my config parser object by calling it So I'm just going to call it config and config is going to read a configuration. So here it's going to read string This is the any file as a string that I typed out earlier and suddenly I have a config with all the sections I had in my any file And I can list the keys In that section so my db. I wound up my host and my port and finally I can get to that So in this case it has a key host and port Um, I want username username is not there So I'm going to give it a default value and like a regular python dictionary using the get method It will turn me my default value right, uh, do note that Uh, the very uh, everything's a string right. So this is a dictionary strings and config files So exactly my port is now a string if I want to use it in a configure in my in my database library And indeed it is an int. I'll have to cast it myself right you only get strings and also gloriously, it is also not Case sensitive, you know keep things easy Uh, so for example my admin email on lowercase as opposed to my actual variable Which is admin with a capital a capital a still pick up The thing so make sure you don't have two variables that are the same if you lowercase them and all that But other than that, you know, keep it easy keep it simple keep in 1980s and Also also works as the uppercase both will match to the same variable um What's good about config parsers? Let's say I have another configuration, right? So here I have Let me print this out for you So I have a section called dv like I had above and I have another variable port now this time 6 4 3 2 not the 5 4 3 2 well in my config I just read it again and it'll update the proper section. So if I try to get my port You'll get my new value again the string, but I still have access to all the old values I had in my previous config file, right? Um, why would this be useful? Well, let me write out these these these configuration files to file to through disk, right? So this is my main and let's say I call the other one Local right Well, one really really cool method is config. I call in you. So I created an empty one And config dot read right and to read I pass a list Of files, right? So in this case, I have main which I just wrote local Which I just also wrote and say testing testing doesn't exist But it might be there might be there on certain servers that I have it might be on my ci server, right? I call that it'll tell me. Oh, these are the any files. I found Oh, you passed one. I didn't find I didn't raise an error. I just ignored it No, uh, no pressure and I'll give you The the I'll give you the variables from uh, the sorry I'll first of all load all the variables from this one and then overwrite all the ones that I find in this one So it found this db host from the first one And it finds the port from the second one and you're none the wiser So you can layer on configuration files. So maybe you have like a main one and then you'll have Uh, a local configuration for local development or your testing or integration server might have a different Configuration you can layer those with these and don't have to worry. Oh, do I have to copy on everyone? Am I missing this variable and keep it simple like that and that's basically it