All right, welcome everyone to OSS Summit. This is the Exotic Runtimes Talk. I have a list of keywords up here, and I'm curious which keywords have brought the most people here today. I'm going to guess that it is probably this one, more than anything. I may be wrong. Can we get a show of hands? Who's using Kubernetes? Okay, that's good. Who's using GitOps already? Most of you, cool. And who is actually using Ruby? A couple, all right. How about Ruby and Wasm together? Nope, not yet. What about who's building a Kubernetes operator? Got one, two, cool. Kubernetes operator with Ruby? No, didn't think so. Okay, cool. Someone tell me that these were original thoughts that I put together and I thought that was cool. So I wanted to know a couple things from the audience. All right, so I already established nobody was here yesterday, but yesterday I presented this as a lightning talk. Today I get to slow down a little bit and we can talk about some things that there wasn't really space for in a 15 minute session. So that's cool. I'm not going to assume that anyone has already seen the session from yesterday. So I'll repeat most of that stuff today. So this is me. I'm Kingden Barrett. I am a Flux maintainer. You can find me on these social media platforms. And I'm a YouTuber. I am a open source support engineer at Weaveworks in the developer experience department. And I work on Flux, mostly in a community capacity, but I've also been, had my hands in the code as a Flux V1 maintainer and I'm trying to contribute where I can to the code of Flux V2. I have a weekly event that's called the Flux Bug Scrub where you can find me and other Flux maintainers if you want an audience with a Flux maintainer. If you have issues that you'd like to have addressed that they've sat in the queue, you can reach us directly and talk to us face to face on a Zoom meeting. And it's the forum for it. Like we have a regular dev meeting also, but okay. So as an open source support engineer, I am focused almost entirely on open source. Big surprise. And I lean into open source solutions as much as I can. We have other presentations also here at GetOpsCon related to Flux. If you scan this QR code, you can find out about all of them. And just a little bit more about me. If you go to my YouTube channel, these are the types of things that you'll find there. One of the things that you will find there is me live coding a Kubernetes operator in Ruby this morning to show off my Wasm bits. So, okay. So what are we here for today? Wasm and Ruby, I think primarily. And what is untrusted code and why do we want to run it? Well, I will tell you a secret. I do not trust any code. That's probably false. If you look at what I do from day to day, I'm probably trusting lots of code. But in a sandbox, that's what Wasm is built around is a sandbox that is for running untrusted code, which is very helpful if you are on a web browser or if you're on a server. So we're primarily gonna focus on a server today. And I had a question when I started preparing for this talk was can you run Ruby in WebAssembly? Because that was what it seemed was on the tin. And it turns out, yes, you can, but I'm not sure that there's really a good reason to do this, at least in a server context. So we'll get back around to that. But why WebAssembly? So we wanna build on a secure foundation. If you haven't heard of the bytecode alliance, they have a great page where you can read all of the details about why WebAssembly. For me, portable artifacts, that is what I'm after. And I would like to have language independence. I cannot sell you on Wasm. If you do not know why you need it, it's gonna be a tough talk. If you take it with you, I will get nothing. There's no commission. I don't know how Wasm makes their money. But let's find out together. Okay, so why Kubernetes? For me, the answer is, I'm a flux maintainer and you have to have Kubernetes to run flux on it. So if you chose Kubernetes, you probably already know why you did. But if you haven't or you're thinking about it, this is a really brief summary of Kubernetes and GitOps together. You have declarative versioned immutable artifacts that describe your desired state and your infrastructure is self-healing. It's constantly reconciling on a path towards putting those things towards your desired state, the way you described it. So these are some compiled languages that you can put into WebAssembly modules. And if you are not interested in Ruby at all, I promise there will be some value for you here today as well. So here's a little bit about my history as a Rubyist. I have been using Ruby for a very long time thanks to a friend on the internet who helped introduce me a long time ago. And I also, my first permanent job was primarily focused on building Ruby things. We used a lot of other languages, VBScript, various things, probably more shell script even than anything else. My second permanent job, I was really focused on Ruby and Ruby on Rails and I was an application developer there with a lot less responsibility. Better pay though. So for me, why Ruby, I know it better than any other language. And for you, whatever language you're choosing, that's probably the language you want to use because you know it well. For me, the benefits of Ruby, comfort and familiarity, debugging experience, can't be beat, bundler, handles gems, dependencies wonderfully. I haven't found an experience like that in practically any other language. And fibers, which are relatively new in Ruby, they are great for concurrency. So I'm using fibers. Minimum viable product is the goal. When you're writing Ruby, you want to write fast. You want to pull in as many dependencies as you can, probably, and don't write anything you don't have to write. If it's your core business dependencies, things that you really need to know inside and out, you probably want to write it yourself, at least once. But once you know it, you'll probably want to lean on whatever's the best in class library. So what are we using Ruby for most of the time? To run a website, I guess, or connect to a database or scrape some content from an internet or maybe build an IRC bot or something like that. Lots of different things you can use Ruby for. There's no compiler. It is duck typed and object oriented. Those things are all great. Okay, so why not build a Kubernetes operator in Ruby? I think that's a great idea. And I'm not the first person. Someone put out a nice library for that. We'll get to that. So WebAssembly and Ruby, here are your entry points. You can use, these are the things I tried, at least, Wesmer and WasmTime. I liked Wesmer a little bit better. I thought their examples were more approachable and more comprehensive. But I did start with WasmTime because that was the runtime that Spin was promoting more heavily first. I'll get Spin also. So what we're gonna do with these is run WebAssemblies in Ruby, not run Ruby in WebAssemblies. And what is a WebAssembly is a runtime format, is binary format. And it is also, besides being able to run applications as WebAssembly, you can build libraries and include them in other programs, like I said. So if you wanted to bundle Ruby interpreter inside of Ruby, I would suggest that you consider not doing that because I couldn't think of any benefit, even though that was the first thing I tried. Like I said, I only wanted to use Ruby. I'm not sure why I would use a Ruby in a WebAssembly for Ruby. There's actually, we have a question? I did. Well, with WasmR I did. Yeah. So there was really no benefit that I could tell other than I won't have to use another language. But for various reasons, the top of them probably being the WebAssembly size. So the Ruby interpreter itself as a compiled WebAssembly is like 30 megabytes. So if you actually intended on distributing these things to people running them in a web browser, you may have made a terrific mistake. So we're gonna try the other thing. And so this is the type of thing that I think WebAssembly users are doing based on going through all the tutorials. And if these are not the things that you thought you wanted to do, keep watching, hopefully they will make sense as we go on. So you can export functions from your WebAssembly and call them from Ruby, or you can, well, from a language that you run your WebAssembly in. You can also export functions in the opposite direction or import functions into your WebAssembly so that you can call them from inside of your code. That's a nice way out from the sandbox if you need to provide a way out, say for fetching or something like that. And the examples were really focused on the use of a compiler, which is kind of the point where I began to infer that maybe this was trying to trick me into using compiler. So, and then there's the system interface. That's the penultimate example that is good for all of the things that you can't easily do in a sandbox. So what is the system interface good for? Input and output, file system, if you wanna import some files so that you can just access them from inside of the WebAssembly, read them from the disk. Those are system calls. And if you wanna make connections to the outside world in a protected way, there's really no network inside of the WebAssembly itself, but you can still do it with something like Wezzy. This is not the only way in, by the way, I should mention, but it's the one we're gonna focus on most today. And there's an extension called Waggy. If you're not familiar with all these buzzwords, maybe you've heard of CGI. Waggy is like CGI, so you take your standard input output and you put a little header at the top that says this is the content I'm accepting or expecting and it's your connection. You don't have to manage connections now, so that's pretty cool. You just get a request and it comes in standard input. So these are some features that you won't find in WebAssembly directly. There's no string type. That's kind of tough, especially for Ruby. I found that if I wanted to get a string into WebAssembly, I had to allocate some memory and then keep the length of the string around and I was getting flashbacks to see and I don't think I want to do that. So I was able to avoid doing that thanks to Wezzy because we can just pass the string in that way and then parse it and pass the result back out through standard output. But as far as actually passing a string to a function, that was the first thing I tried after I tried running Ruby and Ruby and that also did not work for me. So I had to find a different way. What I like to emphasize right now is that all of these constraints, as I found I was running into roadblocks, there are other roadblocks I haven't mentioned yet, a C extensions was probably the primary one. If you use Ruby gems, most of them don't have a C extension but probably somewhere in the realm of 10% of them do. So if you wanted to use NoCogiri say to parse some HTML, you might need to find pure Ruby. If you were still thinking about bundling your Ruby in the Wezm, which I suggest you do if you're interested, but like I said, we're gonna focus on the other stuff. So I was trying to aim towards solving a real problem with Wezm. I needed some string prams and return values. I would assume I would need those things. Okay, so I started with spin, which actually has great docs. And if you're trying to learn Wezm for the first time, I suggest that you start there as well. They will tell you about your language, what level of support there is for that language, what types of things you can do. And there are great examples or links to good examples from there. I did find a nice Ruby example that showed me how to pull in gems. And I think that was in the spin kitchen sink. We'll look at some of those examples later, but so what is spin exactly? It is kind of a serverless framework. And if you know what serverless is, it's kind of not really a thing. It just means we're gonna run some code and then we're gonna shut it down and reclaim all the memory and wait for another request before we use any resources again. So spin is useful for testing locally also. Spin itself is running as a server that will, like I said, help you shuttle connections to your process. It kind of behaves like a router. We're not gonna run on Fermion Cloud because that is not open source. But we are not gonna run on Hippo Factory either because I don't think that is a serious idea. And I've already been sold on Kubernetes and if you're telling me now we need in addition to Kubernetes we also need a Hippo Factory. I know you're pulling my leg. So Hippo Factory is the open source version of Fermion Cloud. So we're gonna use Kubernetes not only because we already have it but because we wanna use Flux. And these are the things that we hope to gain by adopting WebAssembly and whatever shape that finally turns out to look like. We wanna get testability, reusability of our code modules, type safety between languages and I'm also gonna add in here now smaller payload size. We want our code to be smaller when it goes to the users so that it can launch faster because that is one of the primary goals of WebAssembly if you read all the marketing copy. And also we want our polyglot teams to be able to work together. If we've got teams that have specializations that depend on the libraries that exist in particular language we want them to be able to interact with each other with as little friction as possible. So I did build some things for this talk and I followed all the examples that I mentioned so far and let's see what we did. I found a problem that was big enough to require some code to solve. My boss goes to the webpage for each of our GitHub packages on GHCR every day and she doesn't really do it every day but it takes a screenshot and then feeds it through an OCR and figures out over time how many downloads have we had? I mentioned I work in developer experience so how many downloads, how many users are important questions for us? And it's one of the few ways that we have to track our users because we don't like tracking users. Many of our users are in an air gap. We know that there are limitations no matter what we do so this is one of the things we like to know is how many downloads there are and plus it's a really big number so we like to show off big numbers. We like graphs that go up and to the right. So what we do is we visit this page and we see that there is a number there and there doesn't seem to be an API endpoint for that number but it's updated in real time and if you hover the mouse over you can see somewhere deep in HTML there is an actual real number, real whole number of downloads you can fetch. So I built EKS cluster to run this stuff on and if you go to this GitHub repo you can find all about the EKS cluster definition. It's a multi-arch cluster since we wanted to demonstrate the platform independence and it bootstraps flux onto the cluster as part of EKS if you've never used EKS cuddle it's a really nice tool. I actually used it for the first time well recently I'm sure I used it once before but this is the first time I've used it in anger and I deployed some stuff to it. I tried to figure out how to deploy so all right forget Ruby how do we deploy any WebAssembly at all? Let's say we're happy to lean on spin so I came up with a solution for how to deploy a spin application that existed already called Bartholomew onto Kubernetes via GitOps and for me I thought the easiest way would be probably not Helm but I did decide eventually that Helm would be easy enough because I found this great library to help me build a Helm chart, this library called Helmet. It's basically you write a one liner in your templates and then you expand it out with values you just say what is unique about your deployments and whatever else it needs to depend on inside of values.yml How many people are Kubernetes users by the way? All right so these might be familiar topics to everyone. So at this point I began to understand a few things and it's gonna be tough to cover them so I won't be able to go into detail about all of those and show the code I'd rather show the code but Fermion if you know the history of Fermion who's leading SPIN and a lot of the bytecode alliance stuff leading the pack there in the least in my view. They're the inventors of Helm. Matt Butcher is the CEO of Fermion and he invented Helm and they helped develop Azure Kubernetes service after Deus was acquired by Microsoft. So I was pretty miffed when I pulled up all the documentation I found there was no Kubernetes example they didn't show you how to deploy on Kubernetes at all but I did become a little more understanding about that when I tried it myself it was painful for a number of reasons. So let's step away from that for a minute I also built this series of examples moving towards a Kubernetes operator and the Kubernetes operator has a design to solve this problem where we have a project CRD that has the name Flux CD in it that project CRD goes to fan out and see how many packages are there in Flux and then it creates a sub resource called leaf for each one of those packages that sub resource goes and fetches that page that I was talking about that has the number on it saves it in a file passes it to our WASM module which we'll also talk about it's inside of this repo and then the WASM module returns the number parsed from the HTML as a string since we said string return types was a difficult thing and WASI helps us solve it, this is the proof. So once we've got the number back in Ruby we parse it put it back in a number we store that number in the CRD status there's a field defined for the status that says how many downloads count and we'll come back later we'll build a Prometheus exporter so we can scrape this, feed it in a Prometheus we don't wanna store this data long-term but we'd like to have it for a couple of weeks so we can export it regularly and then put it in a spreadsheet. How did I build a Kubernetes operator in Ruby? Well actually it was really easy this project on GitLab is inactive if you look at it but it's based on a nice library called kube-client that is actively developed so with this example you can read a single page of code it's really a very minimal code all it does is it registers whatever CRD that you should be listening to and then it registers two functions upsert and delete it manages your finalizers for you if you don't know what those things are I saw, I asked when we started but how many people are developing Kubernetes operators? We have more people now than before more than a few, all right there's still very few so these might be really unfamiliar concepts but the finalizer is the thing in your custom resource that tells Kubernetes there's something that has to happen before this gets garbage collected if you get a delete signal wait for me to come around and remove the finalizer then clean it up so I wasn't sure what I was gonna use that for but it's common in the operator pattern and if you've tried to learn the operator pattern you probably reached for the controller runtime book which is a bit longer than a page this was very approachable I thought this was a much more easy way flux is based on the controller runtime probably there are a few other controller runtime like things out there I'm not gonna try to name them but here's Kube Client the Ruby library I was talking about has some nice features like it uses server-side apply if you don't know about server-side apply that's another relatively new feature in Kubernetes that flux uses gives you some very nice capabilities like the ability to wait for the resource that you just created to become ready before you return from your function call that said create this thing and if it fails to become ready or times out or anything else goes wrong you can get a signal back instead of just assuming asynchronously that it went okay so even if there's an admission controller or something on the Kubernetes API itself that would prevent that perfectly valid resource from going in you can get that back as a signal and that's great so if this was the lightning talk we would be at the end but we're not and these are some other talks you'll be able to see I just wanna mention them before we forget and run out of time because we're gonna leave the slides and start looking at code so yesterday when I presented the lightning talk the operator wasn't finished yet there is a link if you download the slides you'll get a QR code later if you wanna download the slides or you can just go to my YouTube channel and find where I did finish the functional parts of the operator this morning it actually does work hopefully we'll see it I wrote it in about three hours total if you think that sounds fantastic Ruby man okay so this is also I didn't mention this but I had to write a Rust app because I said well putting Ruby inside of a Wasm probably doesn't seem like a good idea I think it's trying to trick me into a compiler maybe I should let it so I learned a little bit of Rust I learned how to parse HTML and Rust this is my first Rust app we're gonna start with that and then we'll work our way back out to see how do you deploy Wasm and Kubernetes which may be what most of you came here for today and we'll come back to this if we don't forget oh, crashed if we can, all right okay so here is that stats tracker repo that I mentioned I think I want a different here, this is the better one set of tabs each of these repos that I linked by the way in the slides has a tremendously well-developed readme to explain what I was trying to do a lot of the things that I have explained here today are gone over in more depth here things like performance questions does this actually perform at all are started to be addressed in here there are some gaps still in those numbers because I like I mentioned I just finished the operator this morning so it's tough to collect performance metrics right now but what I found most of all was the fact that I kept being forced to make my problem smaller originally I said I wanna do this all in web assembly I wanna do it all in Ruby and that would have been easy except for a web assembly so each time I found a constraint I said okay we can work around this my C extension, no Kogiri there's probably a Gammo pure Ruby library that I can use to parse HTML then when that didn't actually solve my problem I said well how can we solve it let's find another HTML parsing library in Rust and so I did eventually get that to be quite small here's a nice graph going up into the right we'll hopefully see some performance numbers in the future but in each of these example directories there's another readme that shows some of the thought process and some of these things that I found difficult how to walk through them step by step we're not gonna have time for all of that today but let's actually look at that Rust program that I wrote can everybody see this a little bigger it's good, okay so in this directory you can find the Kubernetes operator as well and the lib folder is most of the code and if you go into the lib folder and I'm gonna run make clean here is a file that shows that we are trying to run a WebAssembly in Ruby let's look at that okay so like I said I developed this and I switched it over from Anoco-Giri to Gammo so there are different implementations of current download count in here they all do basically the same thing they start parsing some HTML and then they use a big honkin selector here's the selector way up here I was a little bit nervous that the HTML parsers were gonna have a hard time as I went to more and more obscure HTML parsers but none of them had a problem with that big CSS selector that was fine and this is the entry point for whoever is gonna call us they don't care that we're parsing HTML they really just wanna know the current stat and the time together so we wrap an HTML client cause I thought I'd need that we have a project and a repo and an image which we'll get from the operator structure that I explained so we can get all those numbers and collect them and we'll call this like this so let's make sure that Rust library actually works oops, cargo not found, oh no wait, I know what to do okay so it's doing a few things first it is compiling with cargo it already has the dependencies so that's pretty quick it then strips the wasm module that spits out so we're using this target wasm32wazzy so the compiler knows about wazzy and the strip makes the web assembly much smaller then wasmopt I found that in a guide I'm honestly not exactly sure what that does but I think it is optimizing I think that's what that stands for and then running Ruby we see we actually do get the output we were expecting from the Ruby app with our number that came from Rust and it's been parsed and there it is if we run it again we would see it as in fact a different number that number changes every time so there are people, well not every time but there are people downloading Flux all the time okay so that's cool let's look a little bit more of the code so how do we get our operator to run and let's point at a cluster so this actually works so there's a CLI uses Thor this is really simple let's look at lib CLI so we see how that works Thor lets you define commands like I want CLI controller to do something and let's say we wanted to run with a different organization than Flux CD we could pass it in here I wasn't gonna do that but alright so this is also a very simple code we can delete some stuff that we didn't use I thought I would need a loop or something but that's not how the Kubernetes operator gem works this is it so fiber scheduler basically this just makes sure that these two parts can run in parallel so they're not really running in parallel Ruby is a single threaded language but they do have capability to understand when the system calls weight and send the execution to another fiber so that we can run these things together so that's cool so let's see if that works um okay use local cube config what we see here I'm gonna scroll back up because it's moving really fast we've got our two fibers running trigger action for Flux CD it found the Flux CD resource that had already been created on the cluster let's see that resource looks like it's doing some stuff looks like it's working so that top level resource is called project see we've got one here and we see that it has an empty status don't know why we'll see why in a minute um I wasn't really interested in the project status I wanted to see the status on the leaves so and I like I said I had limited time so let's look at those leaves so it did fan out and find all those packages uh there's a piece of code we didn't see in lib called project reconciler that does that all in ruby I don't need those things to happen in parallel it's it's just one fetch to find all that stuff so I thought that would be fine to leave it this way um we can delete all of these resources as long as we make sure that operator is running it needs to clean up the finalizers like I said let me put that comparable size so you can see so what we should see is it's receiving delete events and it's removing finalizers one after another this is great this was super slow in my hotel wi-fi it's working great here so they all got deleted successfully uh project okay delete project flux cd that one can also get deleted I just want to show you all that we can do it again and it should work and it's actually going to run the wasm module and we'll see the current values so this is not a finished operator there's a list of check checkmarks on that front page read me all the things I had intended to do and it's really more important that I get up here and show some working code I think so alright so we create the controller and here's what happens when there are no projects and no leaves it starts the operator and it registers uh those functions that we set and there's no log output so then we can go to kubernetes operator so here's the code for that kubernetes operator by the way this comes from your gitlab and there's no license in this code I wish that I was able to get in touch with this fellow I I sent him a couple emails to just let him know that I was using it and I really appreciated they made it uh but just to be forewarned there is actually no license in this code um I assumed it was on gitlab so it was okay to fork it but you may you're mileage may vary I'm not sure I just asked nicely if you asked me to take it down I will um okay so in the examples directory here's the example operator this is a lot more approachable than the one I tried to build because it only reconciles one resource but you can see this is a page it's not complicated you register your crd and then you uh create you know you create a kubernetes operator from the object and you tell it what crd it's supposed to listen to you attach it to a logger and then uh there's some event helper I'm not sure what that's for um and uh you register the absurd and delete method and then you run it and that's it and that's what we did in our two fibers each one gets run if you pass uh follow the execute execution path you'll find that run method getting called um are we uh close to time I think I have arm not over time got five minutes four okay great all right so in the example directory there's a manifest directory this also if you're using controller runtime it will help you generate your crd's this does not help you generate your crd's so if you go into there's two example crd's this is a really basic crd definition and it's just slightly adapted and and here's what I wanted to show you uh the properties on the status so this is why our status message if you find the status message in the code didn't actually get reflected because I changed the crd definition there's no message field in our status properties so there's count and last update that's what I thought we should keep track of on both crd's so on the next crd this this one at the top is our projects that's our flux cd and then at the bottom is leaves those are customized controller source controller all of the various microservice components of flux and anything else that we've published as a package like the flux cli um some required fields so there are the three parameters that I mentioned uh get passed into the um scaffold for the uh wasm module and then here we have count and last update again let's make sure we run it before we're out of time okay so here we see triggering some actions all right it's doing some stuff we are going to see some errors in here I haven't tested every single resource but we are going to see also um it's adding events for each one it looks like it's processing most of them without errors I got a couple 409s I don't know what that's for okay all right so it's created all these leaves that's great how many downloads does the flux cli package have 57 million thanks wasm two minutes okay uh let's see what else was I going to show we have two minutes I want to make sure you know about this directory this is where flux is bootstrapped into by eks cuddle and in this here we have examples some of these things I depended on I'm not going to name each of them um I had hoped to show you scaling to zero with kata but the operator itself is not going to scale to zero uh and then taking bartholomew I think we're not going to get to actually run this deployment but I do want to make sure that you know uh that this also has a dedication here for my wife Catherine I'm going to help her build a blog we've tried this before didn't always work out I think this one is going to stick um do we have any questions you have a question uh for me I was not able to get there for me what I need to do is actually package the spin runtime in a multi-arch container image and then deploy that with helm and from there pull the wasm module uh using one of the tools so there are two tools they're available for that flux has its own oci artifact feature and um there's flux build artifact and flux push push artifact which I'm familiar with because I work on flux and then I found a similar feature was added in the spin uh I'm not sure if it's spin push artifact but it works exactly the same way and it's for packing up your wasm modules and then pulling them down at runtime so that's that's pretty much how this works um and if you go in and see the docker file let's look at the docker file this is really simple here we're just downloading spin 1.0 and there's a platform script to detect which spin platform this is a multi-arch container uh build so if you look at the google github workflows here's our consolidated build and it builds using buildx so we get multi-arch image out that we can run our wasm module wherever we need to go so um no I don't see it's going to replace right away but maybe long term there is a tool called kwasm operator if you go look at that that is going to use a field called runtime class that you probably didn't know was in deployment and pod and that runtime class allows you to actually replace your image with some other thing than a docker container it can be a wasm module and then it'll get run by whatever binary you pass in so kwasm operator is not for production um right now if you look at the readme it says right on the front not for production because it uses a privileged um access to put that spin binary in place so that container d can use it but I think we're out of time are we having more questions okay thanks


Good evening, my name is Kobe Olzer really excited to be here tonight. I Lead the DevOps group in a live person You're going to hear today about our great experience with the open stack How we change the way live person consider and decides on its technology how it took us to the next level in terms in terms of Infrastructure and cost efficiency To the tech guys here. I'm going to talk about some of the tools that helped us Maybe something of that you can find they're useful for yourself as well So taking you through the agenda for this talk I'm going to introduce you to live person a short introduction just to get you in the context of what live person does and how it applies on the open stack Going to go through the story how it was in the old days and how it is today Going to talk about some interesting stuff. What's cooking in our lab and Finally some tips and tools that you might find a useful Want to start with a short introduction about myself? Started my career in 99 being with a live person since 2012 have married with three kids and Love everything to do with virtualization automation big data and beer So live person live person been around since 1998 Sus company even before this word was invented. I think it was called the ASP before We have a 8,500 customers eight out of ten a fortune 500 and our mission statement is to create Meaningful customer connections through our a rich a multimedia platform, which I'm going to show you an example in the next slide You can see the real Verizon site. You can see a customer lingering in the checkout page Not sure if we want to go through this is the buy our smart system detects that they haven't maybe some doubts so a Windows pops with a invitation for a live chat with a real Verizon representative and if the visitor accepts then our rich multimedia SAS platform with a voice video and Chat a taking for the sale Increment in sale reducing bounds and helping the customer through the through the process Okay, so before we start a quick Show about how a live person infrastructure looks in a nutshell So we have three tiers the first tier is the web tier like any other a normal SAS company in the web to we have the Apache and the HHA proxy and we have some IIS servers for the legacy systems In the uplayer we have the mainly the Tomcat and the JBoss. It's all about the Java and the Scala for us And in the data store, we have a bunch of them We have a Cassandra Vertica couch base my sequel Oracle and a dupe especially proud of the addupe. We just crossed the one petabyte So as you can see the open stock is used for us For the first two layers, which are our non-persistent layers. I'm going to talk about what is non-persistent for us in a second Okay, our use case when we started this junior and a half ago We knew that we wanted to go as much as virtualized as possible so We had the minimal requirements The first requirement was to find a solution for our non-persistent servers such as the web tier and the up tier and The what by saying non-persistent I mean that those instances don't need to have any data that we need on them.  So There is no value to the data there The other use case the other minimum requirement is to be able to scale to thousands of VMs with commodity servers We had a lot of physical servers and we wanted to reuse those servers with the open stack And the last one we wanted to transform our infrastructure to platform as a service style so Before before we started with the open stack If we wanted to deploy a new service, we needed our operations To work with a VMware or physical servers It took a lot of time and money and then we got a new service Up and running It's not that the VMware is the best solution on the contrary But when we started when we wanted the when we knew that we wanted to go to As much as virtualized as possible We had a very small footprint of VMware and they're looking at our minimal requirement and The fact that we had a lot of physical servers Already in place the open stack was the best cost-efficient option for us So today we were managed to bring our infrastructure To become a building block very easy with the rest API Scripts using puppets and open stack. We are able to build as a cloud of Services in a matter of minutes and even maybe seconds. We have new services running almost new services every day So in the old days as you can see here We used to have a most of our servers physical and some a footprint of a VMware with a none, of course open stack and today most of our servers are a Open stack some VMware the same didn't change and the physical servers which are mostly for the data store And of course for the open stack hosts Okay jumping in into the Components and architecture as you can see here. We mainly use Our KVM is the hypervisor. We use my SQL as our database our distribution is a Ubuntu distribution and we started a long time ago with a Diablo distribution and Moved from there started went to a production with the Essex and today we are running a Folsom and Grizzly on the staging environment We use local disks like I mentioned before we use Our services as a mainly as non-persistent. So local disks work for us We are looking at the solutions for a distributed storage such as a safe, but it's a Will come only later next year We are a managed we are managing everything with a puppet and to end and Starting from a from bringing use new servers up to new services up in production up and running and ready to serve We have a for production data centers two in North America two in Europe One staging and another one in a dev to two other small data centers This is a real snapshot from our production Open-stack as you can see here. We mainly use a the Sentos 6.4 and the the windows the 2003 images and in flavors we have a lot of flavors, but we use mainly the Two cores and four cores with the four gigabyte run It's also possible to of course to use a much bigger flavors like you can see here We also use a 16 cores and the 15 gigabyte of from but So we don't need to be afraid and they can use a much more if you need Okay, so this graph is very interesting You can see the whole life person story. In fact, you can see that we started in mint 2012 built our confidence slowly since then and You can see that the in a me 2013 We got a decision where we are confident enough with the open stack So we moved everything else to open stock with P2VD with V2VD Everything we had left to open stack and since then just continue growing with the open stack We have a 40 we have a 1400 instances with more than 6,000 cores Okay, another interesting stuff with the open stock that we do and we use it for our QA and dev environment so In our lab as you can see here We are planning to a provision with the knock Using foreman foreman is a one-stop shop application for us to provision your services to provision your instances To use it with a to provision new instances even in easy to and to deploy new services using puppet Another very interesting tool for us is them collective of course like we are puppet users So and collective come naturally to us with them collective. We orchestrate Our clusters we use the the plugins out of the box with them collective to manage the the data centers But writing our own plugins to each service We use it to orchestrate the the clusters Another very interesting thing that is going on in the lab is Auto scaling we are looking at the heat and cilometer To and also scale out which is not written here to a auto scale It looks very promising and of course it's it's part of the of the day of the Havana distribution Which is going to be much easier to implement So like I mentioned The open stack is also used in our R&D environment. It's a it's a It's very dear to our R&D because they They can get their own quota to provision VMs. They can set their own private load balancers They have the same keys and metadata deployed on all services and then don't need big budgets to for and settle for few VMs Like they used to do in the past Okay, some tips and tools as I promised Okay, the first one I get to us I get this question a lot. How did I get my management organization to support? so It's a we started very small we started with a couple of servers in a QA and dev environment and Then we took some servers into production for servers that they Were due to decommission we started a new cluster and we started very very small with Not important services like a maybe some alpha services for new For a new services so Starting small beating our confidence slowly but surely Got us until here and of course the the cost efficiency Was it was a big part of the picture when Talking to the management and getting them to support us another tip is like I mentioned before the foreman Combined with puppet and then collective we have a good experience with it in the lab It's a I'm not going to go through of course each technology right now, but Using the foreman To for a one-stop shop to provision new services to deploy to for a puppet dashboard it's a it's a proven to be a Very good asset and last when if you are thinking of starting a new deployment with the open stack Or even you have an existing one. I would suggest looking at the filter scheduler for the soft provisioning It's something that they proven for us Very useful to to look deep into the soft provisioning and figure out exactly what fit for us How to a soft provision because it's not a straightforward you can you can of course out of the box use a memory This space and the CPU soft provision, but you can do it with a Much more Details for your organization and your needs, okay to summarize Like you saw before one a open stack is is a running It's a core business on 100% of the services at least one server and up to 100% And it's also being used in our staging dev and QA environment So if life person could do it, I think that many more companies can do it as well If there was one thing I want you to remember from this session is is this and Hopefully Our use case life person use case can convince most of you to try open stock in production and enjoy the cost effective solution Thank you very much


Everyone that I've sold some to, they say they wear them all the time because they're so comfortable.  You can either get them in small medium or medium large. Honestly, there's not too much of a difference. I can wear both, small medium fits me better, it's just more tighter on my head, but medium large fits great as well, you can pretty much fit either size. So if you guys are looking to order yours, you can get them at www.kickonthevastv.com slash gear. I'll link below. Now, let's go see what the boat looks like. What do you think? That's weird when you come up right here on the boat, does this stuff rub off really easy if it was right here, but it's really carried up. Yeah, it's like, it's got, it gives you a little bit of air. So I'm pulling it up on the trailer, it won't scrape all that because we put it in the rubber. What's going on people? Just got done off the boat here from Nick from ZD Cows. It looks phenomenal. I'm as you guys can tell. I'm sure you guys have seen it on social media, Instagram and everything. It's really great. The only thing we're going to do is we're going to put hashtag team KTVTV right here and then we're going to hit I run ridges to you on the back.  Yeah, absolutely. If you Google bass boat raps on the internet right now, we should come up number one. We've got an awesome bass boat program that we've put together. We've done tons. I mean, literally hundreds of these bass boats. So we really know how to put these together nicely. As a matter of fact, Noah's boat, as good as it looks, he wants to be able to maintain this so that it looks good for years. So if you can get this in the camera over here, you can hardly see it, but here's a tiny edge. We're using this paint protection film. The same thing they use on the front of Ferraris to protect from rock chips and road debris. So this way, when you get one of the co-anglers who want to put their foot up here, they're not going to damage the wrap. But other than that, you can see all the colors and stuff that we have here. We're color experts. They're very vibrant. Exactly what your sponsors standards would be. We can get that all in the wrap and we always use the high quality 3M materials. So just guarantees that your wrap is going to be something that lasts for a long time. Be something that your sponsors are going to be proud of. Definitely check them out, guys. Really great people. Really great family business, as you can tell. I mean, like I said, I've said it five times because it just looks so phenomenal. They did a really great job. We had somebody that was going to do it before, but we came to them and they took care of it. I mean, they had it done. I mean, in a day they had the whole thing done, but we came to them what, three days, four days before this, and they already got everything from it all. So if you guys ever, I mean, I don't care if you're not in Georgia, which I'm in Georgia, you're in North Carolina, South Carolina, Florida, Alabama. Definitely come to them. It's worth a couple hour drive out and get it done right. So definitely check them out. Thank you guys for watching. See you guys next time.


It's such a pleasure to get to be here. My name is Natalie Cartwright, and I run a startup out of Canada that's called FINN. What we do is we build virtual financial assistance for banks. So they take our product, and they put their label on it, and they push it out to their customers. And it's been really exciting to get to be here and to hear some growing consensus around the area that I'm really most interested in, which is tech governance and AI governance, since we use a lot of AI in our product. And at the risk of being indulgent, I'd like to talk a little bit about my company and how we grew it. Because I think it's a really good example of what's happening with companies like mine all around the world, and has some really big policy implications as well, so let me tell you about how we actually started the company. I actually used to work for UN organizations and an organization called the Global Fund. So my background is in a very different world than technology. And I realized that if technology or if international development was gonna improve, it needed a stronger business perspective. So I went to do an MBA, and afterwards one of my friends who was a management consultant in the program called me and said, Natalie, do you wanna do something in fintech? So while we were still on the phone, I googled fintech, which I can now with confidence tell you stands for financial technology. And I said to him, sure, if you wanna move to Canada, I'll create a business with you. So when we're talking about the next generation and doing technology and doing jobs that didn't formerly exist, I am that next generation. I don't really, when I first started at least did not know nearly enough about technology and most of the people that I hire on my team have not been formally trained in technology necessarily. We've got PhDs in data science and other pieces, but lots of masters in geography, other types of pieces. So we've heard that this has really big policy implications for education and I just wanted to share with you three of the things that I see as really fundamental to incorporating in our education as we go forward. So the first thing that we look for is we look for people who have an ability to code or to understand coding language. And it doesn't necessarily mean that everyone needs to be a developer in writing code, but it is the basis of our technology language and should be something that every child understands. The second thing that I look for in my team is people who know how to learn. And it's not just that they know how to learn, but they have to be able to learn independently and teach themselves. So whether that's going to the internet or asking a friend or reading a book, I don't really care how they learn, but they're always trying to improve themselves and take on new challenges. And the third thing that's really important for people on my team with their skillset is that they have to be failure resistant. When we're building technology, we don't build projects from zero to 100. We build them in these very small incremental steps. But often, when you reach one step, you don't solve it the first time. And often it's not the second time. And often it's not the third time. It could be the 50th or the 100th time that you approach a problem before you actually solve it. But that means that I need people on my team to be able to fail repeatedly, pick themselves back up and continue to keep going. So for people who are involved in educational policy, those are the three things that I think we should be looking at going forward. It was three years ago that we started our company. Today, we're about 40 people. We're hiring five to ten people every month. We've been able to attract quite a few banks, which has been fantastic. We work across four continents. We work with one of the ten largest banks in the world. And I just want to give you an idea of the scale at which my company, which is really just 40 people, relatively small, is working at. So just through one of our customers, which is our largest customer, we'll be ruled out to 20 million people over the next three to four years. And I don't say that because it's exceptional. I actually say that because I think it's entirely unexceptional. We know that there's some big players who are creating AI. Everyone talks about them. But I think that companies like mine, of which there are thousands around the world, are also rolling things out at scale. And the implication from a policy perspective is that we don't have the luxury of time. We have to figure out how we move quickly because there's companies who are building and scaling and really don't have a lot of frameworks to work with them. And I think one of the solutions for that problem can be borrowing one of the playbooks from the startup world, which is something called the minimum viable product. And it's a basic principle that Eric Rice wrote a book on called The Lean Startup. Basically, it's just you always start and you build the least amount that you can as quick as you can for the least amount of money that you can, and you constantly iterate on that. And that's one of the reasons that we can move so quickly when we're building technology. So while I don't know entirely what it would look like, I challenge you to think about what a minimum viable policy approach would look like to try and keep pace with this very, very quickly evolving world. And the last point that I wanna make today is one about how we actually went about building our company. So we started using a cloud service from Amazon. Formally, you probably would have had to invest $20,000 to $30,000 to get a server up and running just to get started. We went and bought a third party product called API.ai that we could put our data into to try and test the data models. We've since rebuilt that. I think in total, we spent a few hundred dollars to get a very small product up, but that was enough to get us some interest from banks and to realize that we should continue going. And the implication for me from a policy perspective is the most exciting because these types of companies should exist everywhere in the world. Technology and cost is not the barrier for entry anymore. And when Patrick and I were speaking about this earlier today, the question now is how do we create and facilitate ecosystems around the world so that companies like mine and all the other panelists can exist everywhere and really be the drivers of economies? So I'm extremely excited about tech. I love getting to be in it. I think AI is going to bring really, really great changes. And our challenge is how do we build government structures, policies, and systems that are nimble, innovative, and equitable? Thank you very much.