Hello, and welcome to Introduction to Kubernetes Operators. Before we begin, I would like to reiterate, for those of you watching virtually, there is a thread in the Slack channel. Well, actually, let's jet over there just for a second. So here in the Cloud Native Development Track, I've started the thread. If you'd like to chime in, make comments, ask questions, that's where you should be doing it. Please feel free to type in any questions you may have at any point. We'll be coming back to those near the end when there's time for questions. So giving you a couple seconds to look at that. And then we're going to move on. OK, so Introduction to Kubernetes Operators. My name is Jonathan Bercon. I'm an open source contributor who works for IBM. Who am I exactly, and why should you listen to me? So I'm an open source contributor for IBM. What does that mean? That means I work full-time, more or less, on open source. I'm a software engineer who writes code that is pushed to public GitHub repositories. Currently, I can probably take this off. Currently, I'm a contributor to Operator SDK, which is part of the Operator Framework, which is a CNCF project. It's not actually part of Kubernetes. It's a separate CNCF project. I'm a contributor there. I contribute code. I'm also a community co-chair, which means I help run the community meetings, plan releases, cut releases, all of the grunt work that makes things work. I'm also a Kubernetes contributor, although not so much anymore, since I shifted to working on an Operator Framework. But previously, I contributed a lot to service catalog and also some bits of Kube CTO. And then before Kubernetes was even a thing, I was also a contributor to a separate similar project called Cloud Foundry, which is an open source platform as a service that was spun out of VMware and Pivotal Labs, which I guess is now VMware. Again, it's all the same in the end. So I've been around the block. I've been working in open source in Cloud Platform stuff for about eight years now. And I like to at least pretend I kind of know what I'm talking about. So let's move on to what I'm talking about today. What is this talk in specific about? This talk is about the very important topic of how do we put the lime into the coconut and then drink them both up? Wait, no, sorry. That's what I'm doing after this talk. This talk is about how do we put software into Kubernetes? There's lots of different ways to do this, you might say. Jonathan, I've got like 1,500 ways I already put software into Kubernetes. I've got raw YAML I crud up and down. I've got Helm charts. I've got Terraform scripts. I've got CloudFormation scripts. I've got more ways than you can shake a stick at. I can shove as much software as I want into Kubernetes. And for the most part, you'd be right. Kubernetes does a lot of stuff for you such that if you can decompose whatever your application is into sort of the cloud native style, 12-factor apps.  And Kubernetes will do most of the heavy lifting for you, such that when a piece of your application, your web app, whatever it is, falls over, Kubernetes is going to kick it. It's going to restart. Everything's going to be hunky dory. But I would be willing to posit that your entire application is not easily decomposable into this 12-factor ephemeral app state. You probably have a dependency somewhere in your stack on stateful applications. The usual running example we like to use is that of a relational database.  But somewhere in the back end, they plug into a database where they store all of your stuff.  Because Kubernetes, ever-diligent, will in fact restart those pods or deployments. And they'll get sutured back into whatever the networking stuff you have set up with services. But the pod when it comes back up is not going to be happy. Because the previous pod had a whole bunch of state that is not going to be present in the new pod.   So operators, then, are a way to try and fix this. A way that if we could define the behavior that Kubernetes uses to bring that stateful resource back up and we could write our own custom logic in that, that would be a great help. Because we could input all of the knowledge, the domain specific behavior that's needed for your specific stateful application, and insert it into Kubernetes so it knows exactly how to bring your thing specifically back up. And that is what the point of an operator is. So what is an operator? An operator is a pattern. For deploying software to Kubernetes cluster in a way that's sort of different from the previous creding yaml pushing helm charts sort of thing. So that deploys software infrastructure statically to the cluster.  You push it that MySQL cluster is running containers and pods on the Kubernetes cluster, and it exists statically through time. What if, rather than that method, we extended the cluster API itself so that it understood MySQL as a resource type within Kubernetes itself. So rather than building a MySQL cluster out of pods and deployments and services, we teach Kubernetes to understand what MySQL, what a MySQL cluster is. And that extends the API of the cluster such that there will be a new resource type, MySQL cluster, that it knows. It speaks. It knows how to make them. It knows how to kick them. It knows how to delete them. It knows how to bring them up, break them down, back up, whatever you have. The name operator comes from the human operator, that friendly sys admin we had in the previous slide, who would otherwise be required to manage this state. We're sort of taking the knowledge that they would otherwise embody and inserting that into our operator, which will then manage this for us. And the way this is literally accomplished is by implementing in user space the same processes that core Kubernetes uses to manage itself, reconciliation, and resources. Now, you're probably already familiar with these processes from a user perspective. If you've ever touched a Kubernetes cluster, written something to run on top of one, used one at all. But we'll go into those in a little bit more in depth. So the first one, reconciliation. Reconciliation is how stuff actually happens in a Kubernetes cluster. It follows a declarative model, which is a little bit weird, because it's most people in the cloud computing space are probably more familiar with an imperative model where you create a thing, go make me this thing. In Kubernetes, you interact with the cluster by requesting state. You just say, I would like three containers running this image. And then that state is written to an LCD store in the cluster. And then at some later date, presumably, eventually, that state will become true. So the user requests the state by writing Kubernetes specs. There are processes in the background called controllers, which we'll go into a little bit more in a moment, that then goes and creates computing resources based on that requested state. And once that has happened, it comes back and marks the records with the status. What is the actual running state of the system? And eventually, after many iterations, those two will match, until they don't. And then we start all over again. So those controllers. So this is something, like I said, if you've ever used a Kubernetes cluster, you're probably familiar with the behavior of these, even if you aren't familiar with the specific implementation of how they run in the back end. A controller is a process that's responsible for the reconciliation of a resource type. For instance, in core Kubernetes, there is a deployment controller. There's a pod controller. There's a service controller. There's at least one controller for every resource type that you've interacted with. And when you go and request a pod, a deployment of service, what have you, it's going to attempt to rectify that requested state with what's actually running on the cluster, one very small step at a time. And these many controllers for these many resources running in sync, each iterating one step at a time, will eventually reconcile the desired state that you have written in all of your specs with the actual state of the resources running on the cluster.  And then the pod controller is going to fire and say, okay, there needs to be these many pods. I'm gonna go and start some pods. So any resource, any single resource you create is going to have a cascading effect that propagates across the system and causes many things to happen. So that's reconciliation. That's the first principle we need to understand. And the other is resources. So you're presumably familiar with core Kubernetes resources like deployments, pods, services, et cetera. The backing structure of that for operators is a custom resource definition or CRD. If you're not familiar with these, what a CRD is, is it's a type that when you create an instance of a CRD, it creates a novel resource type on the cluster. And you, the writer of the CRD, define the spec and status of that new type. So you can say, I want a spec field that's size, the number of nodes in my MySQL cluster. I want a status field that when the cluster has gone and then created, I want the names of the pods that the cluster nodes are running on to come back and feed the filter back up into my object. And the naming convention can be kind of confusing. So the three sort of separate steps. You create a custom resource definition. You create a CRD. You write that. And you create a CRD to the cluster. That causes a CR, a custom resource to exist on the cluster. So I create a MySQL CRD. That causes a MySQL CR to exist on the cluster. And then a user will go and create an instance of that custom resource, which we will call MySQLs. So you have the MySQL CRD, which creates a MySQL CR. And then individual users instantiate MySQLs. Now note that no matter how fancy you make your CRD, once you write it and upload it to the cluster, it doesn't actually do anything just yet. That's because there is no, even if I made this nice fancy MySQL CRD, there is no controller that is reconciling that CRD type with actual computing resources on the back end. So we then need to write our CRD and the accompanying controller. And those two components are what comprises an operator for the most part. For the most part, an operator is a CRD and an accompanying controller that implement and define a new resource type that exists on the Kubernetes API. Once we've created these two pieces, end users will interact with that new type the same way they interact with any other Kubernetes resource. They'll be able to kubectl create it. They'll be able to write Helm charts that interact with it. They'll be able to write Terraform scripts that make it.  Now for the purposes of this talk, we're gonna stick with this relatively simple architecture, one CRD and one controller that controls the behavior of that CRD. More complicated architectures are possible. For instance, you could have a very complicated operator that has many CRDs and many controllers that control the behavior of them. Maybe those CRDs are dependent on one another. So when you create a CRD, a MySQL CRD, I don't know, it's like sharded geographically and you have another CRD underneath that the MySQL CRD is composed of. Maybe you want to change the fundamental behavior of some of the core Kubernetes types. So maybe I want to have, whenever a pod is created, it automatically comes with a second pod containing some kind of sidecar deployment. So I could write an operator that consists of no CRDs and a controller that changes the behavior of pods. So there are more complicated kinds, but for the purpose of this talk, we're just gonna keep one CRD, one controller. Okay, and let's go ahead and dive in to the demo here. Okay, so what are we gonna do today? Today, we are going to implement a simple operator. We're gonna make memcached, which is on disk key value store. We're gonna instantiate a very basic operator and implement the CRD and the controller for that operator and stand it up and look at it go. For this talk, I'm gonna be using a tool called operator SDK, which is one of the things I work on and contribute to, which is a CLI for quickly scaffolding operators. It does a whole bunch of other fancy stuff. We're gonna be using sort of the base level stuff today. So I'm gonna run operator SDK init. I'm gonna feed it a domain and a repository. That's just to keep track of the going library that I'm gonna be generating that this is gonna be written in. And let's go ahead and take a look at what that wrote. So we have a Docker ignore and a get ignore, some basic stuff to keep track of all the junk we're about to create in this repository. We have a Docker file. So remember how I said that an operator for this part is going to comprised of a CRD and a controller? That controller is a process that has to run somewhere. Where should we run it? Well, we're running this on top of a Kubernetes cluster, so let's just stick it in a pod and that's what this Docker file defines. This is the Docker file for the process that will eventually be run in a pod that is our controller. And this is just scaffolded by the command I just ran, but this is just a regular Docker file. So you could modify this to make it do whatever Docker file stuff you want if you need crazy stuff in your controller. We have a make file that comes pre-generated with a whole bunch of tasks that we have for some fancier stuff. We're not gonna be going over that today, but I'll mention that at the very end. But basically some tasks for building and pushing the various images that we're gonna make to comprise our operator, generating some code generating and manifest generation that we're gonna use in a bit to generate the various parts of our operators, building and pushing stuff, fairly basic stuff. There is some more advanced stuff in here to like run some test suites and such. We're not gonna be using that too much today. We have a project file. So this is the one file that really like sort of isn't yours to keep. This is a operator SDK project file that's keeping track of like what version and what plugins this operator was generated with because you can do further modifications down the line by running more operator SDK commands to generate fancier stuff. And this is gonna keep track of what versions were used. So this is the one file you really sort of shouldn't touch. We've got a config directory. So this directory contains a whole bunch of YAML files and YAML stubs that will be used to deploy our operator to Kubernetes. Right now it should be pretty bare because we haven't actually created a CRD. We have a manager, which is sort of the shell web server that our controller will eventually be run with but for now it's empty. And that's actually what's in main.go. So let's go take a look at that. So for the moment, like I said, this is just sort of an empty web server that boots up and listens and doesn't do anything. This is written using a go library called controller manager and we're going to implement a controller which is really just a plug into this and then add it to this server. So it'll go and run the controller and it'll connect to the API server and talk to the Kubernetes and it'll do all that stuff for us so we can focus on the important part of implementing the reconciliation logic of our type. So that's pretty much it for the init. We generated all that scaffolding. We've got a bunch of stuff for managing the go project that we're about to embark on writing. So let's go ahead and write the next command. So I've got this stuff in a paste bin right here. So, oh, I didn't mean to enter that. So let's highlight that here for a second. Okay, so operator SDK create API. I've gotten a whole bunch of version flags that aren't really too important. The name of my kind is gonna be called nemcachedd and I want both the resource, which is to say the CRD specification and the controller, which is going to be the process that actually reconciles our type for us.  Most importantly, we now have an API directory, v1alpha, which is the version we passed in and we have nemcachedd types. So what is this? This is the go lang structure that will act as the back of the CRD we're implementing. This is defined using go lang structs for spec status and the overall nemcachedd struct. These should look very familiar because all these are is a go lang representation of the same structures you interact with when you kubect create YAML files. So they're going to be structured the exact same way, but this is the thing that's gonna cause our resource to exist. So this is where we define that structure. So let's go ahead and paste some stuff in here. So for us, we're going to implement a very simple spec and status for our type. We're gonna have one field in each. So for the spec, we're gonna have an integer representing the size, which is going to be the number of nodes in our nemcachedd cluster. And then in the status, we're going to have an array of strings called nodes, which is going to be an array that contains the names of the pods that our cluster nodes are running in. So that's gonna be what gets created when I actually write and instantiate this type in YAML files and then kubectl create those. And then the nodes, that's what's gonna flow back from our controller once we've written it. So let's go ahead and write that. I'm gonna run. So make generate, this is gonna generate some go lang scaffolding stuff that's gonna be used to generate the CRDs for this type. Basically make generate and make manifest. You're probably gonna wanna run these after you change any actual stuff just to make sure all of the scaffolding is up to date. And then make manifest, that'll take those go lang stuff that we just generated and generate the YAML that we'll use to create those in a second. So that's the CRD. Let's go take a look at our controller. Okay, so this is a huge mess of code. So rather than dive into that directly, I have some slides that I'd like to go over first. Okay, so the concept of our memcached object, remember we just defined the CRD which has a spec.size and a status.nodes. That's the information that's gonna be contained in our custom resource instances. What we want to have happen when a person creates those is we're going to say when a person creates a memcached object that memcached object is going to be backed by deployment. That deployment is gonna have a number of pods equal to our size. So when a person creates a memcached CR we want on the backend a deployment corresponding to it to come into existence. And that deployment we're gonna define the same way you would a normal deployment. We're just gonna say have these many replicas running this image and we're gonna let the deployment and the pod controllers go and make that happen for us. So that's gonna happen but then on the backend after that happens we need to go and collect the names of those pods and feed them back up into our status.nodes. So our controller is gonna be responsible for that part as well. Okay, so that's what we want. How are we gonna do it? We're gonna have to talk about reconciliation again. So specifically the reconciliation of a single controller is literally implemented as an idempotent loop. What does that mean? It means this loop, I'm gonna sit there and I'm gonna run it over and over and over again based on the current state, the current desired state in the SCD store of the cluster for the memcached D table. It needs to be idempotent which means that given a single specific requested state I should be able to run this loop an arbitrary number of times. An infinite number of times a hundred, a thousand, a million and it should always converge towards a single desired end state as defined by what the user requested by writing a whole bunch of specs. So I should be able to run this loop any number of times and the eventual end outcome should not change. Each single iteration of the loop has three possible outcomes. The simplest and easiest is just to exit and that marks the specific resource we're trying to reconcile as this thing has been completely reconciled, the actual state on the cluster is equivalent to the desired state in the SCD so stop trying to reconcile this object. The probably more common exit is requeue and the request. So that means for whatever reason this thing is not done being reconciled, you can continue reconciling this object so please continue to try and do that. And then the third possible exit condition is requeue with error. So this is sort of similar to the second one in that this object will continue to be reconciled. You can pass along an error that contains maybe some sort of helpful information to try and give the next iteration a clue about like maybe something is going wrong. Maybe the network is being slow. Maybe I'm like waiting for some other controller to kick and fire like I created the deployment and I'm waiting for some pods to come up so maybe you should wait a minute or two to that to happen because if you just fire again immediately you're probably gonna encounter the same error. So how do we implement our specific loop for our memcached type? Now remember on the upper right there I've got a little shorthand description of what we're hoping to have happen. We have a memcached object that is backed by a corresponding deployment. That deployment has one or more pods and those pod names we wanna flow back into our memcached object. So our loop is gonna fire and say I have this memcached object foo that needs reconciliation. What's the first thing our loop should do? Now this may seem sort of counterintuitive but the very first thing we wanna do on each iteration of our loop is actually go and re-fetch memcached object foo. Now you might say John this doesn't make any sense. I'm already reconciling foo. That's true but there isn't really any guarantees of when this reconciliation request was written and what the state of the cluster is now that it could be happening immediately. Maybe I don't know weird network stuff was happening and it happened like five minutes ago which is basically forever in terms of a distributed system. So we're just gonna go and fetch the object we're reconciling and make sure we have the freshest possible view of the system and the easiest simplest thing that could happen is foo no longer exists in which case our job is real easy because we could say foo doesn't exist which means there's nothing to reconcile. It's been deleted or who knows it's gone now. So we're just gonna exit and say like done our job here is done. We can exit, we can mark this foo object as reconciled. We don't have to do anything at all. In the event that that doesn't happen we have to actually do some work. So assuming memcached foo exists we move on to the next step of our loop which is okay given memcached foo there is supposed to be a corresponding deployment. Does that exist? Let's go look. So we're gonna attempt to fetch the corresponding deployment. Now there are two possible outcomes to this. Either A deployment foo doesn't exist. If it doesn't exist we say okay foo still exists so there is a requested state of this memcached object should be backed by some resources on the system. Let's go and create it. So I'm gonna go and create an entry in the deployment table that says okay I should have deployment foo make that happen. But again I'm the memcached decontroller I'm not actually responsible for making deployments happen. So there's nothing really more I can do at the moment. So I'm actually gonna say okay that's it I took my first step I did a thing I moved one step closer towards full reconciliation. I'm gonna recue I'm just gonna exit I'm gonna wait and eventually some point later down the line reconciliation for memcached foo is gonna fire again.  So we fetched deployment foo it didn't exist we created an entry in the deployment table we recued the request and whenever that happens which might be immediately might be five minutes from now we're gonna start at the top.  We're gonna check to make sure it exists if it doesn't okay we're gonna exit we're gonna go and fetch deployment foo again to make sure it came into existence. And if it's there hopefully we're gonna say okay deployment foo exists what's the next step in our reconciliation logic. So the number of requested size in our memcached object supposedly should correspond to the replicas in the deployment. So we're gonna check that and make sure that's true. And again we have two possible outcomes. The first is they aren't equal. Now you might say this doesn't make any sense. I will just create it how would it be different? Well you'll notice we didn't actually create when we I didn't mention this when we created foo we didn't actually change the replicas we just instantiated deployment which means its size was actually zero which means this step will actually fire every time we create a memcached deployment. But it's possible that someone for instance went in and manually added to the deployment maybe I don't know the deployment controller was having a bad day and randomly sticking numbers into things. But we just wanna make sure that this step regardless of like what we think happened or what might have happened or what did happen should eventually always move towards reconciliation of the desired in the actual state. So we're gonna go say okay check replicas equal size if not change them. And then again we're gonna go all the way back to the top we're gonna say requeue. Now in this instance there's a very easy thing that we know we're gonna have to wait for. If you change the replicas size of a deployment either pods are going to come into existence or pods are going to get terminated we know that's going to happen. So we wanna requeue the request wait a bit for that to happen. We're gonna come back into the top we're gonna execute each of these steps each of these checks all the way until we get to check replicas equal size again and assuming they're equal we're gonna say okay what's the next step? We need to check to make sure if nodes memcached.status.nodes is what it should look like. In this case it's supposed to contain the names of each of the pods that are backed by the deployment. So we're gonna go and verify that. If it's incorrect we're gonna set it. We're gonna say okay maybe some pods were created maybe some pods were deleted whatever we're gonna take those names and jam them into that array and that's it. That's fully reconciled. We took the information flowing in in size we made sure there's a backing deployment and that deployment's backed by the correct number of pods. We then took those names of those pods and flowed them back into the status and that's all there is to it. It's not a very complicated CRD but that's it. So we can go ahead and exit there and say this object has been fully reconciled. The other possibility is that nodes was already correct. We don't actually have to do anything at all in which case this object is also fully reconciled we can go ahead and exit. So in that case no matter what if we get all the way down to there we know we can at least that's the last step or the thing was already correct. So what does that look like in actual code? So I've already gotten implementation here I'm just gonna copy paste this in and then we can take a look at it. Okay so literally like I said this is written in Golang this is written using a Golang library called controller manager and controller runtime and this is going to take most of the like heavy lifting grant work away from us. The like spinning the web server up and connecting it to the API and making sure we have all the permissions that's gonna be taken care of four of us by all the scaffolding that was generated using those operator SDK commands. So really all we as the operator authors have to care about is implementing this reconciling method that's gonna implement that loop we just stepped through. So I'm gonna boot up I'm gonna start I'm gonna write down a logger so I can emit logs for the controller. I'm gonna instantiate a memcache dh object which this is a the Golang struct that we saw in those Golang type files that has it looks exactly like a Kubernetes YAML file that's got the object meta and then a spec and a status. This is just a Golang representation of that it has the same fields in the same places. We're gonna go and fetch that memcache d object and we're gonna say okay if when we tried to fetch it did we get an error? There's one specific error we care about in particular the is not found error if I tried to fetch foo and it's not found that means something deleted it in which case I don't have to care anymore. So we're gonna say we're gonna emit a log that says hey we tried to find memcache resource it wasn't found we're ignoring it because that means somebody deleted it and we don't have to care about it anymore. So we're gonna return a empty result and nil for no error. So that's that's how you exit the loop you return those two things specifically and that's gonna mark this resource as no longer needs to be reconciled. The alternative is if we got an error back and it wasn't an is not found error there's not really a whole lot of reasoning we can do about why that is. It might be we got a network error back for some reason the API server didn't respond. Maybe the network is just slow when our request timed out. We could sit here and guess all day about like what exactly these errors are trying to tell us but really all we can do is say okay I'm just gonna emit some logs say we failed to get the memcache d and then return an error and requeue and maybe hopefully at a later date event this will fire again and hopefully it'll succeed. If not it'll probably be pretty obvious to whatever user is interacting with the system trying to create this resource that's something weird and they'll come in and read these logs and debug it. Okay so we went and fetched foo. We made sure that it was there. We turned an error if it didn't. Okay so now we go and need to check the corresponding deployment. So we're gonna make a deployment object and just as how the memcache d object was really just a go lang struct representing the Kubernetes YAML that you're already familiar with. This is the same thing. We're gonna say go and try and fetch that deployment and this is gonna, you know I've defined this name such that there's a specific pattern the corresponding deployment will all be named in a certain fashion and have labels tagging it as own by the specific memcache d object that is its parent. We're gonna do the same thing, check for an error. If we get specifically an is not found error we know okay memcache d foo exists, it's still there. If the deployment doesn't exist we want a deployment because the memcache d object still exists. So we're gonna go ahead and create it. So we're gonna inflate a deployment. We're gonna issue a create request. If that create request fails we're gonna blow up and then we're gonna say okay we created the deployment. We're gonna go ahead and exit and re-queue. If we failed to get the deployment, if that means, or if we failed to get the deployment because of an error that wasn't is not found. Again we can't really sit here and reason about why that's happened, maybe the network's slow, maybe the API server isn't responding, who knows. We're just gonna return and say hey try again later. Okay so assuming we've gone through step one fetching memcache d object step two fetching the corresponding deployment we have to check that the memcache d size and the deployment replicas are the same. So we're gonna grab those off of these two objects. We're gonna compare them, if they're different we're gonna update the field on the replica because again we're sort of flowing a source of truth as our memcache d object that's flowing down. Everything else should be changed to correspond to that desired state. We're gonna again do some error checks if for some reason we couldn't talk to the API server we're gonna blow up and re-queue later. If the write goes through and succeeds we're gonna implement this fancy little flag re-queue after we're gonna wait a minute because again if we're changing the replica of a deployment we know pods are either going to come into existence they're gonna need to be terminated so wait a minute because we know that's gonna take time. And then finally assuming all of that worked we need to go and fetch the list of pods that corresponds to our deployment. So we're gonna make a little pod list, a little structure for containing a list of pods. We're gonna write a filter that we're gonna use to list only the pod that are corresponding to our linked deployment there. I'm gonna say API server list me all the pods that look like this. We're gonna grab the names of all of those pods in the list. We're gonna reflect them to make sure that they should be equal to the status.notes. If they aren't equal we're gonna write an update and say okay change these and we're gonna return assuming that doesn't blow up with a nil result and a nil error. So that means we're gonna be fully reconciled and we're gonna exit. So that's all of the implementation parts for our simple operator. Of course it could never work at the right time.  Okay, so hopefully that should generate some manifests. Okay, so let's go ahead and build the Docker images for our controller. Hopefully this shouldn't take too long. So we're gonna build this Docker image, push it up to a repository. That's gonna be the container that our controller is. That's what that's gonna run. And then we're gonna install the CRD which is just going to exist in our Kubernetes at CD statically. Let's go ahead and make sure our Kubernetes is up and running. Okay, that looks good. Still building that image. We can go ahead and so one of the things we have that was generated here is a sample ammo for our novel type, memcachede. It's gonna be called memcachede sample and let's go ahead and define a size of two. So that's gonna create a memcachede that should, if we wrote our controller correctly, cause a deployment of replicas two and then two pods should come up. Assuming my controller ever finishes building. Yeah, I can talk for a minute about some of the more advanced stuff we're not doing today. So this is sort of a basic, very simple example. Oh, we have a question? I'm sorry? No, so that was in the status of the object. So if you're not familiar with Kubernetes spec and status, a Kubernetes spec is what the user writes. It's what they request, say I want a memcachede object of this size. Status is what the controller writes back to the object. So the spec is the requested state. The status is the actual state of stuff running on the cluster. So I as a user, when I instantiate memcachede objects, I will not write, I'll never write anything in the status. And that's the same way any Kubernetes pod deployments services, you never write status manually. That's for the controller to write when it actually goes and reconciles so that you have an insight into what is actually running on the system. Does that make sense? Right, so again, this is sort of the concept behind our operator. We have the spec.size, the user's gonna write that.  And those names are gonna be automatically generated and they're gonna be named A and B and then our controller is gonna go fetch those and stick them into the status. Okay, so that ran, so we can go ahead and make deploy. And if all of my building blocks fit together, we should have a running operator. Oh no, I did the demo this morning and this is gonna take a while because the internet here is so slow. Oh no, there it goes. Okay, so there we have the controller coming up in our memcached operator system namespace. Once it's up and running, we should be able to kubectl make this. We should be able to get our new type, which it should understand. So we have a new memcached sample. We should be able to see some pods come up and the corresponds to those. And when we take a look at our object, the ammo, uh oh, extra space in there, we can see that the names of those nodes have been inserted, which should match. We got 6C, 765, 6C, 765. So now we know where the nodes that are actually running our resource type live. So that's sort of the whole life cycle of what you can do in an operator. This is obviously a very simple operator, but out of those moving parts that we demonstrated, you can build much more complex things. For instance, when changing the size of the deployment, if I could check if that size is going up, I know that new nodes are being added to the cluster. If this is some sort of database, they probably need to be synced so I could modify how those new nodes are getting started, say oh, you have to sync with the pre-existing nodes of the cluster to fetch the data. I could add a field to the spec that says trigger a backup and define what that means in my reconciliation behavior. Out of these simple building blocks, you could build much more complex behavior. And that is sort of an iterative process rather than a geometrically scaling one because all I need to do is I need to add little links in this relatively complicated chain. But each individual function, each individual link is relatively simple. You move one more step and you have three possible outcomes. So that's hopefully not too much complexity you need to juggle. Shall we do it on time? We've got seven minutes left. Okay, I'm just gonna quickly run stuff over this and then we can go to questions. So my slides are posted on the page. This is a whole bunch of resources. If you're interested in writing operators, Operator SDK is the tool I use today to see a live for scaffolding operators. There's a link to the docs. We support multiple architectures. The one I just did today is written in Go. We also support writing operators in Helm, Ansible, and recently Java. KubeBuilder is another tool. It's sort of also a standalone tool with CLI for scaffolding operators. It only does go and at this point, KubeBuilder and Operator SDK projects have sort of been merged. The everything I just did today, the Golang stuff under the covers is KubeBuilder. So KubeBuilder is now today the Golang plugin that Operator SDK uses and we have separate plugins for different architectures. So anything you make with KubeBuilder and anything you make with the Golang part of Operator SDK should look the same. They should be compatible. Operator Hub is a community repository for operators. There's I think a couple hundred up there by now. You can go there and take a look and see what other people are doing. And then finally I have written some Operator Tutorial courses on cognitiveclass.ai. You have to sign up for an account but other than that they're free and this will cover all of the stuff I talked about today and then way more stuff and way more detail if you wanna learn how to write operators, how to use the advanced features of Operator SDK like Operator Lifecycle Manager which lets you deploy and upgrade operators in place in a nice fashion for real life running on a GEA cluster as well as some of the other fancy stuff testing frameworks and such that you can scaffold out with Operator SDK. So yeah, that's it for my spiel. We've got five minutes left. Does anyone have any questions? Down in front. I saw you, we'll go to the next. So the use case is really just sort of a stateful application that we're hoping to sort of demonstrate like why you would wanna write an operator. MySQL is sort of just a good non-con experiment, a thought experiment that we like to use because it's very easy to grok. Like if you write a web app you probably have a database, MySQL was just an example. No, so like if the entire MySQL of every single cluster node on the MySQL cluster blows up, no, we would not be, I mean you could, you could have some field on your MySQL type that triggers a backup that would have externally, you could do that. You could even have that internal, maybe it backs up to some sort of persistent volume claim that's within the Kubernetes cluster itself. But you would have to define that behavior in your controller, which as I just showed you, it's just adding one more link to that chain. And that would allow you to do literally whatever you want. So you would, yourself as the operator author would have to implement that functionality. But then once you wrote it, once it's deployed in the controller, that's just gonna, that loop's gonna sit there and do whatever you told it to do forever. So that gives you a great deal of expressive power that you as the operator author can define. But then the end user doesn't, they just have to know how to interact with your Kubernetes type. They just have to say like, okay, backup request equals true or whatever.  And then over here. Okay, so the question is when I try and fetch the deployment, if it's not found, what should I as the controller go and do? Okay, so remember, I always start at the top of the loop. So every single time I go through all of those previous steps before I take one more further than I did the last time. So if I'm reconciling memcachedefoo, I fetch memcachedefoo, it exists, which means someone out there still wants a memcachedefoo because otherwise it would be gone because they deleted it. And then I fetch deploymentfoo and it doesn't exist. What does that mean? That means memcachedefoo exists, somebody wants it to exist, it should be backed by a deployment. I went and looked for the deployment, which I sort of glossed over this, but all the stuff is written up with tags and labels such that like if I have memcachedefoo, I will be able to guess the name and namespace of the exact deployment that corresponds to it because it also has an owner tag pointing back up to me. So if I go and look for that specific deployment that is shaped in this name, the very specific way it should be, and it doesn't exist, that's a problem because I know memcachedefoo still exists, so somebody wants a memcachedefoo cluster, but the deployment that actually runs that cluster doesn't exist. So if somebody wants a memcachedefoo object but the corresponding deployment doesn't exist, I need to go and create it. Yeah, really this is just a whole bunch of naming tricks to make sure that memcachedefoo has exactly one deployment that corresponds to it, but that's all managed using labels and tags and owner references. Okay, next question. Okay, so the question is how do you measure sort of the muncherness of an operator in terms of features? And you are correct in assuming that there are sort of two axes, axes, accesses to that question. One, which is like say I'm writing a memcachedee operator, like how much support do I have for all the buttons that memcachedee has? You know memcadic, I want a backup feature and a sharding feature and this and that, and that as the operator author, they're gonna have to write that support for those features themselves into the reconciliation loop, so that's sort of up to them. The other axis is like operator features. So I did mention OLM, operator lifecycle manager. That's sort of the current operator framework way for, so I went and wrote this operator, but it needs to be deployed to a cluster and I'm gonna write new versions of this operator and that thing's gonna need to be upgraded in place and all the stuff that's currently running has to not explode. So that's what OLM is sort of the solution to currently. It's basically a package manager.  And then when you go and write version 2.0, you'll define some more stuff in your bundle that says like okay maybe I updated my CRD, I added this new field. Here's how you change 1.0 to 2.0, here's what that means. And then you write a bundle for version 2.0, you deploy that to your cluster and your cluster will be like okay, I now know how to write 1.0 and 2.0 and when I go from 1 to 2, I know I need to do this list of things that you told me how to do. So that's really just sort of a path for managing that upgrade case. There's OLM, there's another feature called Scorecard which this isn't really as like integral to the operator SDK experience. Scorecard is a thing that operator SDK supports that lets you quickly scaffold like a test framework for integration unit testing and testing of a deployed operator. That's like, it's nice, we support it because we love testing but you could do that your own way if you had some testing framework you really like. So OLM integration, that's really the big one. We got a couple other things like Scorecard that are nice to haves but could be done other ways. Does that answer your question? Okay, give me just a sec. I did say I would look online if anybody asks some questions. Okay, so we have, does a custom resource need to be created in order to create an operator? So like I mentioned briefly, not, I mean the answer is no, literally. But I imagine and in my experience most operators are going to have a custom resource generally one or more. So like they could be like ours where we have one memcached type because I'm trying to implement some sort of database thing and then the corresponding controller that controls it. I could have multiple CRs and multiple corresponding controllers. Like I know the Istio operator has a whole bunch of types that all mesh together to make Istio happen and then corresponding controllers for those many custom resource types. Like I said also an example of an operator that doesn't have any custom resource types is let's say I want to like fundamentally alter the behavior of a core Kubernetes type. And the example I gave was like let's say every time anybody creates a pod anywhere I want to make a thing happen. Like maybe I want to inject a sidecar router into that pod or deploy it in the pod right next to it. I could write that as an operator but I would write a controller that like sits there and watches the pod table to like say like oh somebody created a pod I'm gonna have to go do a thing. But I would not have a corresponding custom resource type for that operator because I would only be changing the behavior of pods that is a pre-existing resource I wouldn't have to write a CRD for this. So that's an example of a operator that could be written without a CR. And then is there a mechanism to detect conflicting controllers? Short answer, no. So literally speaking how that controller reaches in with its newly appendages is like a whole bunch of RBAC was generated that gave that controller a service account with admin level permissions to the memcache table the deployment table and the pod table. Cause those are the three things that I need to read and manipulate. But from like Kubernetes perspective that's just another service account that's like issuing requests to the API server. There's no way for me to like I don't even know that they're coming from the same place they could be you know I'm just seeing requests coming in. So the short answer is no there isn't any way to figure that out. And yes it is kind of a problem.  I mean that's like not really a solvable problem.  And you know how would I detect that? The API server has no way to know it's issuing the requests for the same service account. So I guess the real answer to that question is is that's why you should only have one controller for each of your custom resource types. It's you know good practice to never have more than one because that sort of collision can happen. And that can get you very easily into a state that's very hard to get out of. Okay that's it for questions there we're technically over time. So I'd like to thank you all for coming. I'll be sticking around. So if you'd like you can come speak to me in person. Thank you.