Okay, so if you have seen the speaker video, I think I have said like no PPTs. So apparently they needed some PPTs to show it as part of the video. So just like put together some PPT and most of it is going to be just like link to code or some code samples, some code snippets in there. Okay. So I'm Sudar. I work as a research engineer in Yahoo Labs. Okay, so I've been dealing with web applications throughout my career and currently doing something on Node.js as part of my day job as well. Okay, so that's it and this is what the agenda is going to be. I'll just like be quickly running through the slides and then we'll get to the code and we'll see how we can use different modules in Node.js and things like that. Okay, so I kind of assume that most of you would be slightly new to Node.js but I think now I mean since most of you are already used to it I think I can just like pass in or quickly like go through all these slides. So as you know it's basically it's going to be an evented non-blocking I.O. built on V8 the usual stuff and it's a JavaScript programming environment support C++ add-ons and all that. So was Node.js the first evented system available? No. So there were a couple of other things which were available as an evented system even before Node.js. Some of them was like the twisted framework from Python or the evented event mission from Ruby and all that stuff. So the main difference between all these other frameworks and Node.js is that Node.js provides this event I.O. as a language construct rather than as a library or as a framework. So all these other things provided more as a framework and you might have to probably use it as a library but it's not available as part of the language construct itself and that's where Node.js is different from other evented I.O. things which are present. By the way I assume that all of you know about evented I.O. So anybody here have need to need me to explain about evented I.O. So I kind of assume that all of you know about it. So again Node.js is not Ruby on Rails or it's not Danjo or it's not code lignite.  And in Node.js you have a lot of these frameworks available. The famous among them is Express and Socket I.O. Connect and all that stuff. So this is just a small slide which I put in to explain what is evented I.O. I'm just going to quickly run through it assuming that all of you know about evented I.O. So the basic things right I mean if in any traditional programming language whatever statements you have that gets executed in sequentially. So if a particular statement is going to get blocked for some reason. Maybe it's waiting for some I.O. Maybe it's like waiting for a DB call and it's waiting for a network call and things like that. The next statement is not going to get executed immediately. It's going to wait till that particular statement finishes its execution and the next statement will execute only after that. So this is a typical code flow which you would have probably seen in most of the web applications or any application which involves network calls. So the problem with this is when your execution stack gets stopped at that point and it's waiting for the other part so basically it's going to wait for the input or the output which is going to happen. And generally when you in any computer architecture the input and output is way slower than the execution which is going to happen. So to cope up with this most of the network programs what they do is like they spawn new threads for each of these requests almost all of these threads would be blocked on the IO at some particular point in time. So what's the problem with that? The main problem with that is threads are not lightweight or the resource part. So if you have to create a new thread or a new process the OS has to get involved and it has to create the thread it has to allocate the memory for it and it has to maintain the descriptor for it and all that stuff which takes time and which also takes up a lot of resources. And also there should be a master thread which should maintain all these different threads and things like that. And that's where Node.js is slightly different or Node.js or any event touch system is slightly different there. So what you're going to do is you're going to give a call back to any function which is going to get delayed whether it's going to wait for an input or an output.  And whenever that particular input or output returns back the call back gets called. So this is a typical code flow for that. So here the call back function which is given as a second argument to that method call or whatever would get executed when that IO returns and when initially the code is getting executed the statement after this would get executed would get executed immediately. So what is the main advantage of this? The main advantage is that you're not going to get blocked, you're not going to create multiple threads. What you're going to do is you're just going to have a function call.  The OS doesn't need to create a new thread or doesn't have to spawn a new process and all that. So it's just going to have some heap allocation done and then the particular code can just keep executing. So that's one of the main advantages of an event at IO. The advantage gets really imminent if you're going to do a lot of concurrent client request models. So basically where if you have a network program which has a lot of concurrent clients connecting to it then this becomes a real advantage for you because you're not going to get blocked by a single client's request. The main program which is going to receive or which is going to consume all these requests can just listen for these requests get it and allocate a call back to it and then can end the execution and then it can go and then probably serve the other clients. So your client queue is not blocked because of input by another process. So any questions so far or am I going too fast or something? So hope everybody is clear about event IO and why we need event IO and all that stuff. So let's see how a call back works now. Sorry can you be a bit loud? Sorry can you be a bit loud I'm not able to hear it all. Yes Yes. So I didn't get the last part. How do you So the question is that since the event at model is going to be slightly different than the traditional model how are you going to do locking right? That's the question. So in Node.js there is no concept of locking. So you don't do any locking here. So the locking comes into picture when there are multiple threats which are going to try to access a single resource so here that scenario is not going to happen so what you are doing is like every request which comes in gets a call back assigned to it. So there aren't any concurrent things which are happening so you don't necessarily need a lock for it. So did I answer your question? Okay. Okay. So the question is if you have two operations which needs to be done so and these operations have to be done sequentially and more than sequentially let's assume that they are dependent on each other so the second operation is dependent on the first operation. So in that case what you have to do is the second operation should start only after the first. So you will have the callback for the first operation and inside the callback for the first operation you can start the second operation. You can assign the callback inside that. So that would solve the dependence issue but generally you don't do that but unless and until they are really dependent on each other and there is no point in proceeding with the second option if the first task is not. So in that case you can put it inside the callback of the first. So your question is like do you have to rearchitect your program. So basically you have to go get into event at IO if you want to do things in Node.js you cannot continue to do it the sequential way. So it also depends on what you actually want to do with your program. So Node.js is good but it's not the solution for all the problems. So there are certain types of problems for which event at IO could be an advantage for you but there could be other if it's just a batch job where you just want to execute a sequence step of jobs and it has nothing to do with so in those cases it may be better you don't do it in the evented way and then you continue to do it in the typical sequential way which you do. So it again depends on what problem you're trying to solve. So Node.js is good but it's not the solution for all problems. So there are set of problems for which Node.js is good there are set of other problems for which the traditional sequential programming model is good. So we might probably have to decide that before you take it out. So any questions before? Okay, so let's see an example of a callback. Okay, so is the text visible or do we want this is good? Okay. So this is just an example for callback and to just explain the concept that things don't happen sequentially. So what we do is we have a set timeout function. You can compare it to a timer or if you're familiar with Java then you can compare it with the threads you create in Java. And it's not technically a thread but I'm just like giving a vague comparison. And so what happens here is so you say that after 2 seconds so this is like 200 milliseconds 2000 milliseconds which is like 2 seconds. After 2 seconds call this function. So this is the function so this is the function which is going to get called after 2 seconds. And the callback is assigned to it and then the execution is going to continue with the next statement after that. So when we execute this you see that first the second statement is going to get executed and then after 2 seconds the callback gets kicked in and then that would get executed. So you can see that. So first the second statement gets executed and after that after 2 seconds the callback gets kicked in and then it gets executed. So, this is how things happen in Node.js, so everything is evented and it is not going to get executed in the order in which it is present, so let us move on, so that was just a small example for evented IO and so you have decided that you are going to use Node.js to, let me tell you how do you do that, so first you have to install Node.js and install NPM, NPM is nothing but a package manager for Node.js, from the Linux world you could probably compare it with an app get or aptitude or yum or something like that, it is a package manager for Node.js and you can use commands like NPM install and then give a module name to install any Node.js modules. And you can also use Node as an interactive console to quickly probably not run commands, so this is similar to what you do in Python, so in Python you have an interactive shell with which you can interact, so the same thing can be done with Node.js as well, so after you installed Node, so in my laptop I have already installed Node, so what you can do is you can just like give Node, just type Node alone and then it opens an interactive command for you, okay, so you can do things like normal assignments, so is it visible or do you mean okay, so you can do things like this or you can test Boolean expressions, something like true, not equal to false and you can also create functions and then call functions and all that, you can import modules, so you can probably try something like V is equal to 3 plus 2 and stuff like that, so it is more like an interactive console where you can, if you are just getting started with Node, you can just like use this, you can try some of these programs and all that, so it can also be used as a calculator or things like that, so basically you can get a feel of Node by using this interactive prompt. And so there are like couple of commands which you can use to get more information, so one is help, so if you say .help, it gives you what are the other commands which are available for you, so you get break, you get clear, so which is to clear the screen and all that, so there are like lot of these commands available, so you can like try it out if you are like very new to Node and then just want to get a feel of it, okay. So the other way to use Node is like creating servers, so Node is very good when you are creating network programs especially server components, okay, so let's see how we can create a server component in Node, so what I am going to do is I am going to just create a HTTP server, so let me, so this is a code for creating an HTTP server which can serve content and things like that, so this is a very simple server, so what I am going to do is every time a client connects to it, I am just going to print the word hello world to it, okay, so this statement here, what we are doing is we are trying to import a module, so this importing a module is similar to what you do in Java, you have import statements in Java and maybe in Python and things like that or hash include in C, so this is similar to that, so you, so this is the common JS format and so what you are going to do is right now I am going to include the HTTP module into this program and HTTP has a method called create server by which I am going to create a server, so that is going to take a callback, see this is the callback function here, so this function would get executed every time whenever a client request is made to the server, okay, so inside the callback what I am doing is just I am going to set the content type for the response as a plain text and then I am going to echo hello world to it and then end it, okay, and then this is going to listen on code number 1332 on the local host and this statement would get executed once the callback has been assigned, so when we execute this you could probably see that this gets executed first and whenever there is a request to the server the callback gets executed, okay, so let us try running this, by the way all these code samples are there in GitHub, so you do not actually have to write them if write them, I would share the link on my ppt where you can get all these code snippets, so I have started the HTTP server now, so it is running on my local host on port 1332, so let us open this from a browser and you can see that the callback gets executed and then you get hello world as a response back, so again the main concept here is that the statements are not executed in order, so the callback gets assigned and the callback gets invoked only when that particular event happens, so this is just a very simple HTTP server but you can probably do a lot of other stuff like if you want to serve static content you can just read it from the disk and then flash it to the response object so that it is served to the client who made the request, you can also set the content type, you can live stream it, do whatever you want, so there are like a lot of modules which help you to do that and so any questions, yes? Okay, so the question is how this particular small server helps when they are going to be concurrent request, so in a traditional web server for example if you take Apache, InginX is slightly different but let us assume it is Apache, so what is going to happen in this scenario, so let us assume there are 10 concurrent requests which come to the server, so the Apache server would start creating threats for each of these 10 requests, so what you will end up having is you have a main thread and then 10 different threads which are created to serve 10 different lines, so Apache is like slightly more intelligent in that way because it does not keep creating threats, it tries to reuse threats and all that, so let us, for the argument let us assume that it creates thread for every new request, so here what is going to happen is there is going to be only one thread, there is no threads which are going to get created at all, so what happens is when a request comes it gets this callback assigned to it on the same process, it is not that you are having multiple processes created for it, it is just that, it is like a function call, so when you do a function call it gets allocated in the heap, in your heap memory you get the return address and all that stuff and then you get the function call, so same thing is going to happen here, so it is not going to create multiple threads, it is just a single process which is executing this, sorry it does not maintain actually, it does not maintain different sessions, so what is actually it is for Node.js it is just a function call, just like a callback which gets called up for each of those requests, yes. Okay so it again depends on what application you want to do, so the problem with Apache is that when there are like lot of coherent requests it ends up creating lot of threads which eats up lot of resources and then the load on your server goes up and things like that, but if you want a way by which you want to talk to multiple processes at the same time and things like that, then you might probably have to go with Apache, right, so Node.js has advantage in certain scenarios, it is not as I said it is not the solution for all problems, so did I answer your question, maybe like you can take it off. So any questions, yes, yes, yes, okay go ahead with your question and I will answer that, okay so to answer your question the thread, so one process taking time or one particular request taking time is not going to affect the other processes, sorry not the other request, so because what is going to happen is every request which comes in is going to have its own callback, it may be IO it may not, okay so see blocking, so there are two things, two places where it can get blocked, right, one is you are doing some huge number crunching where at that particular point it is the CPU which gets blocked, you get blocked at CPU level, there are other places where you get blocked at the IO level, you get blocked because you have made a DB query or you have made a network request or you are reading from a local file, so in that case the CPU is not getting blocked, it is just the IO which is getting blocked and because of the IO, CPU is idle, right, in the first scenario where the CPU itself is getting blocked in that case Node.js is not going to help you, Node.js would help you if your CPU is idle but the process gets blocked because of some IO which is happening, so the second case where it is getting blocked by IO Node.js would be, okay, so again it depends on your application, if predominantly you are going to do only number crunching and you are going to do very, very small amount of IO and your bottleneck is not IO, it is just the number crunching part or the CPU part, then it is not recommended for you to go with Node.js, you can probably go with your, no, you can spawn child processes in Node.js, I have code for that a little later, so you can spawn child processes and then do it but that is not the general way or the more efficient way to use Node.js but you can still create child processes, yes, sorry, once you spawn child processes, yes, yeah, so that is why I said Node.js is not going to give you any advantage in those scenarios where Node.js would really help, let us assume a case where you have to serve a lot of static content or you have to serve big files, so what happens in Apaches, you get, Apaches is going to create a lot of threats for you and so that would probably consume more resources on your server and your server load would go up and your server's capacity to serve the number of clients would go down but if you are going to use Node.js for that scenario, with the same server, with the same resources, you would be able to serve more clients, yes, so the OS is not going to get kicked in and the OS is not going to create multiple threats which in turn is not going to consume more resources, yes, so for each function there has to be some memory which has to be reserved, that is what I said, the heap memory has to be allocated for it but when you compare it with the amount of resources that needed for creating new thread, this is slightly less, so you get an advantage there, so it is not that Node.js is like, so, yeah, that can also fill up, so relatively if Apaches is able to handle 1000 clients, Node.js might probably be able to handle 2000 clients or maybe 3000 but eventually Node.js is also going to make, you know, it is also going to run out of memory but the amount, the number of clients, relatively speaking, the number of clients you can access or number of clients you can serve with the same resource is slightly higher, there are benchmarks I can share it with you offline maybe, yes, in Node.js it is a non-blocking call, so you get a call back for that, okay, I have some code samples for that I can show you how it is getting done, yes. It is not actually going to stop it, right, so what happens is, so there is a queue, there is a client queue which is there, so there are like multiple requests from clients which are coming, so there is a queue and this queue is getting handled by the event loop, so every time a client request comes in from the queue, what this is going to do is, it is either, so in Apaches it is just going to create processes for each of those requests, Node.js is just going to do a call back for each of those processes. Yeah, so the only difference is that it is not, yes, okay, so that is why I was slightly asking whether people have an understanding about evented IO and all that, so the call back function, so the main difference is that if you take a threaded example and a call back, so if you are going to use threads, the OS gets kicked in, so the OS has to create a new process, has to spawn a new process, it has to allocate its own program memory and all that and then it also has to allocate its own heap and all that, so which is slightly more resource intensive operation which has to be done, but in case of a call back, it is not going to go to the OS level at all, it has been maintained by the process, I mean the process which is running Node.js itself, so from the OS perspective there are not going to be any more threads which are going to get created, so okay, okay, so let's probably do that then, the concurrent call, okay, yeah, I think that's okay, so let's take it offline, let's see the other examples which I have, okay, so this is moving on, so this is like how we will be using Node.js as a client, okay, so let me quickly show you the example here, so here what I am basically trying to do is I am including the same HTTP module and I am creating a request object, so for the request object I have to specify the host, port, path, method and all that stuff, the typical things which you have to do for creating an HTTP request and then I am assigning a call back again here, so what's going to happen is whenever there's a response which comes back, this call back gets called in and then it's going to, so from the response object I can get the response status code and again in order to even get the data from it or have to use another call back, so this probably explains the dependency scenario which we talked about, right, so first in order to get data from a request object you need the request object first, so the first call back is doing that, so you are getting a request object first, sorry, a response object first and from the response object you are going to assign another call back when the data is available, okay, sorry, I think this should fire it or maybe like I probably missed it, I'll check that out, I think the request on would fire it up, so when you assign a call back I think it also fires the request as part of it, I'm not sure if I'll be able to run this example because I don't have connectivity to the internet right now, so let's, sorry, okay, then let's, so the server is running here and okay, I think let's move on because like I have a lot of other examples to cover and we have very little time, so these are some of the code modules which are available as part of Node.js, you have the processes for handling client processes and things like that, you have the file system module, you have the networking module which has like a lot of classes of modules, sub modules, using which you can create a TCP connection or the HTTP connection and DNS, you can do DNS and all the stuff and then you have the utilities module which also has like a lot of functions which can make use of in your Node.js, so this link here you can follow this link to find out what are the entire list of modules which are available in Node.js, so I think they are close to some 30 modules which are available, these are part of the core and for, if you need more modules then you can use any of the other extensions which are available to you, so these are built-in modules, these are not third-party modules, okay, so the philosophy of Node is that the core part of the library would be, the core part of the framework is very light and if you want to have any extensions or if you want to have additional functionalities that is available as an extension to you and you can find those extensions and then install it using NPM, so let's see some examples for the process module, so somebody asked me about creating child processes, right, so let's see how we can do that, so this is a code sample which shows you how you can create child processes, Node.js is not going to be very efficient in doing this but suppose in case if you have to create child processes you can do this, so there is a child process module which you have to import and then there is a spawn module, sub-module inside that, so that is going to, using that you would be creating a child process, so here what I'm doing is I'm just like using grep, so the first argument is a command which needs to be executed and the second is a list of arguments which you have to send to it, so I'm just like creating a new process which is going to execute grep, okay and sorry, yes, so let's quickly see three minutes more, okay, so it has spawned a new thread and this is the child process ID, okay, so there are other modules like file system, let me see if I can just like show you the source code for the file system modules, so this basically shows you how you can read files, so this is like, so this particular function, the stat function is going to give you more information about a file, so you can find out what is the size of it or what is the date in which created last modified date and all these meta-information which are associated with the file and this is going to tell you how you can read file from the local system, local file system and as you can see all of these have callbacks, okay, so let's quickly, so you can see, so it was, it just like went ahead and then I'm just like dumping the entire object here, so you get all the information about that particular file, the date in which created, modified time and all that size, what is the UID, GID and all that stuff, okay, so you can also check the other files which are there in my GitHub, you have a lot of these examples to try out different modules, so this is a networking module, so you have the HTTP request, HTTP S modules, DNS modules and all that, you can also make HTTP S calls, the secure calls, I'm not going to run these examples now, again you have these utility module where you have, so I've been using console, so console is part of the utility module, that is basically used to output something to the standard output, the standard output which is available, you also have utils, so let me see if I can quickly show you the code, database access you might probably have to use third-party modules, it depends on which database you want to access, whether you want to access MongoDB or CouchDB or MySQL or whatever, so there are like third-party extensions available and then you just like, install them using npm and then you can start using that, sorry, I don't think there are any native things for database, so as I said the core part of node wants to be as simple as possible, so that you can extend it and then there are like third-party things which are available where you can use it, time's up, okay, so just one more minute, so as I said node.js is useful for certain applications, maybe like these are some examples of applications but again before you even you start you might probably have to figure out whether it's like really useful and things like that and these are some of the technologies, npm is the package manager and modules is like a plugin or add-on for it, then there are like a couple of other modules or frameworks which are available, we have Express, Jade, Socket IO I think, Aditya is going to talk about Socket IO a little later and then there's also a talk about Express where we show you how you can create MEZ applications and all that, so these are the links, thank you.


First of all, thank you for attending this talk. I expected a lot less attendance. So this talk is about writing OpenShift and Kubernetes controllers in Python. And my name is Subin. I am a senior software engineer in the AI COE team. In the last one, two years, I spent a lot of time writing controllers in different languages. And this talk is about sharing some of my experiences writing in Python. So what we're trying to talk about today, what I'm trying to talk about today is about controllers, which are the brains behind Kubernetes and OpenShift. Just to get an idea of what the audience does, anyone use OpenShift here or Kubernetes? How many of you are from, not from Red Hat and use OpenShift or Kubernetes? It's just one. That's good. So I didn't talk for today to understand more about OpenShift and Kubernetes operators. We need to understand what Kubernetes is and learn a little bit about controllers and CRDs and operators, and then talk about how to design a controller from scratch, like what are the considerations you need to take care of? And what are the options we have to write a controller? So we will pick two of those options. One is the Go Client and the Python Client. And then briefly talk about Operator SDK and do a couple of demos to showcase the Python controller. So what is Kubernetes? Many of you might have interacted with Kubernetes. We are the UI or the CLI.  There's a master node and there's a couple of slave nodes. And it does something with containers. But that basic idea may not be enough to start out writing controllers. You need to learn a bit more about the components of Kubernetes to basically understand which component to interact with when you write controllers. So the Kubernetes components are shown in the diagram here. There's a master, which contains its CD, which is like a store, which stores different states of objects in a Kubernetes cluster. And you have API server, which is the main management entity. And then you have different nodes on which these application pods are deployed by the Kubernetes. So again, coming back to the question, what is Kubernetes? We started with the basic idea of this master slave. And then we talked about deploying containers. Kubernetes is a container orchestration engine which is designed to deploy containerized applications on a set of nodes. And so there's nothing new about anything like a cluster manager which does deployment of some objects on nodes. There has been in the last many decades countless such cluster managers. So what's the difference about Kubernetes and the previous cluster managers is that many of them previously implemented. They were monolithic in architecture. What that means is that there's a central brain behind these clusters, which knows the entire state of all the objects, which the cluster manages. And when it tries to achieve any action, it tries to base that action on the entire set of state of the entire objects in the cluster. Whereas Kubernetes is slightly different where it's a decentralized approach. It has multiple brains. And each of these brains are independent of each other. And these brains take care of specific parts of the cluster. And they interact with each other to basically manage this cluster in an autonomous way. So that autonomous is basically the big thing in Kubernetes. And we will learn more about how this autonomous behavior is achieved by these different brains of Kubernetes. Another thing about Kubernetes is that it's a declarative orchestration engine. What you mean by that is the objects which are there in the cluster, they start out the journey with a certain declared desired state. For example, what you see on the slide is a deployment specification, which defines that I need a deployment with three replicas of this particular container. And this is defined by a user. And the Kubernetes cluster basically looks at this declarative specification and starts about the journey of realizing this deployment. And what Kubernetes does is that when it tries to take actions on objects which it manages, the most important thing it always does is that it knows that it needs to take an object from point A to point B. That is a state A to state B. It always starts out by looking what's the current state. It's not looking at the previous states of an object. It takes the current state into consideration and then decides based on the current state what's the action it needs to take. And what it wants to do is that based on the current state, it wants to move one step further towards the final user defined state of this particular object. So this movement from one state to the final state can be a single step or it could be multiple actions. And each action changes the state of the object. And so we start out with the current state and the Kubernetes eventually reaches the final state. And when the final state is reached, there's no longer actions which needs to be taken on the state because the final state is the desired state of the object given in the specification. So once the final state is reached, the Kubernetes cluster has reached a steady state for managing that particular object. And all these state changes for different objects to be stored and they are stored in the HCD node. And these states are stored by the API server, which is the central management entity. The API server is a management entity which directly talks with HCD and manages the state of different objects. It also has endpoints which we can use to create a controller. So when we interact with the API server, we need to be careful about what kind of objects we deal with. In Kubernetes, there's a concept of each object is a type. And that type is called kind. For example, you have pods, images, application controllers, blah, blah, all the others. So all of them form a kind. When you create an instance of any of this kind, it's called a resource. It's called a resource because just like the HTTP resource like the rest resource, this instance of the object has HTTP endpoint where you can do the cloud operations of creating or updating the object. And another thing, the API, each of these resources may or may not have is the sub-resources. The thing which we need to be careful with sub-resources is that they may not be HTTP endpoints. There could be some other protocols. So the first demo I have is just trying the baby steps of how exactly we can explore or understand controllers and how we can start out this journey of understanding what if I am right in the middle of Kubernetes cluster as a person, what would I see? So I have here a demo where I'm creating a pod. And this pod has the OC or the kubectl command line. I hope people are familiar with the OC tool. And so let me just show you this particular grid. So I have here a pod. I basically named it as my controller dev environment. It's just a pod. And that pod has this OC tool. And it's running an infinite loop. I just have a bash infinite loop. But if I go to the terminal, I can actually execute the commands of OC. And I can do stuff like OC, get pods. So what I'm trying to show you is the first baby step where I am in the Kubernetes cluster. And as a program, I am in a pod. And I'm using something, a basic tool like OC, which we use outside the cluster to manage clusters. And we are using that tool inside a pod to do the same operations, like get pods and look at the events. And the API server exposes something called events. I'll just come to that later. We can also explore different options, command line options of OC in that pod by looking at different resources available in the API server. So this slide talks about the different kinds which are available. The kinds are the type of the entities which Kubernetes manages. And on the right hand side, you can see the different kinds. So what I'm trying to do with the baby step next is get more understanding about how Kubernetes work. So so far, we have an understanding that it's a decentralized cluster. There are multiple brains. And one important thing these brains do is that they look at the current state. They're not looking at the historical state and all that. And they also look at the desired state. So we understand it's a decentralized and there's a state change of objects. And the current state is important. So with this kind of basic information, we still haven't really understood controllers or how these brains work. So the brains or the controllers are these control loop. It's basically an application running in a pod, in an infinite loop, which does something. We'll come to what that is later on. So it's a process which constantly runs in a pod. And it's doing something to manage the resources and entities managed by Kubernetes. So this is just a diagram just to show how a controller basically looks. Just a sum in this infinite loop within a controller pod. But we still haven't come to how exactly does it do it. And so the controller, what it does in this infinite loop is to constantly read the state of the resources it manages. And as soon as change of state is received, what it does is it tries to find out what's the next action it needs to take on that state. And it does the change. So the controller does actions. It listens for events. And it does an action. And then the state is updated. And then it goes back into this infinite loop, waiting for the next event change. So if you want to write a program programmatically, it's like this. The algorithm is you get the first user defined state, which is the final desired state based on the specification. And you get the current state. And then you invoke, let's say, make changes. Basically, it figures out what's the action it needs to take based on the current state. So these state changes in Kubernetes object are called events. So every time, let's say, a pod starts or stops, an event is generated because the state of this pod has changed. And these events can be captured by this watch interface, which is exposed by every resource. This watch interface is like a change notification feed. Every time there's a change in the state, you can basically hook on to this feed and get to know what's the change. The most important kind of events we are looking at from this watch interface is listed here, added, modified, and deleted. I will show you a demo soon, what that means. So here's a demo. What this demo is doing is that I have written a simple Python code. So this is a pod in which this Python controller code is running for a long time in an infinite loop. And it lists, basically, the events seen by this pod. All these events are specific to pod. So this is a controller pod, which just looks at pod. And you can see that it lists all these events, so it is captured for the different pods created here. Just to explain, let's pick up the second last line. The second last line says, a modified event for this particular pod. And it shows the source version. The source version is a very important, it's like a number which basically identifies the state change. Like if there are n state changes, there would be n different source versions. And the last line shows a deleted event where this particular pod, my controller dev pod, has been deleted. And you can see that there's a corresponding source version change for that particular event. So this example what it shows is it's a Python. It's a very basic Python controller which is having an infinite loop. And it's listening to different events, specifically for pods. And it's trying to capture the output of all those state changes. So coming back to this question of how does Kubernetes work, we have understood that it's a decentralized. We have multiple brains in it. And all these brains are basically the controllers. And the controllers are infinite loops running in a pod. And these controllers consume events via the watch interface. And this interface is between the API server and the controller. And the API server is part of the control plane. So this slide basically diagrammatically explains how a Kubernetes cluster looks with respect to a controller view. So a Kubernetes cluster has many controllers. And each controller does specific things. For example, the demo I showed just now has a Python controller which just looks at pod. So you might have a pod controller. You have a replication controller. You have image controllers. For every entity type in Kubernetes, there's a separate controller. So we have some basic understanding about how Kubernetes works. And we now understand that controllers are important. And we also understand some basic idea about how controllers and this infinite control loop exist. So now let's dig deeper about how exactly we write a fully functioning controller. So we have also looked at how does if I am a controller code and I'm right in the middle of the Kubernetes cluster, what do I have? What are the tools I have? And to access the cluster and look into the cluster and do stuff in the cluster. But there's a lot more to writing a controller than just looking at the events and using the watch interfaces. Assume that a controller has crashed and it comes up again, how does it know what to do next? So there are thousands and thousands of resources in a Kubernetes cluster. And how does it identify where I am? Because it has completely lost track of what it was doing previously before the crash. So the first thing a controller needs to do after the crash or when it actually comes up is that it needs to get the entire state of the object it manages. So what it does is get to do a list function which you need to get all the object states. And once you get the object state via the events of each of these resources, you need to do some calculation like which is the most recent event of this object. And that is done by looking at the resource version. Another thing we may be tempted to do when we write controllers is that we may use the top level event resource which should not be used. It is mainly used for logging purpose. And it has a kind of cleanup activity where after every one are all the events in the top level event resource are deleted. So if you're writing a controller, don't use the top level event. Just use the individual events available for each resource. And we learned that watch interface is the interface which we use to capture all the events. But there's certain problems with watch calls. They are actually very expensive. And we shouldn't be using too many of the watch calls in a controller. Especially when you're writing Python application, we have to be careful because of the concurrency concerns. Our Python application might crash if we are using too many watch calls. So controller has come up. And the first thing it did was do a list call where it captured all the events. And then it looked at what's the most recent resource version of the pod is. And once the list function completes, the controller has all the state information until that moment, t is equal to 0 moment. But from that moment onwards, it only needs to do a watch call on the most recent resource version, meaning once it identifies the latest pod, all it needs to do from that point onwards is to find what's happening to that pod from that state onwards. So when you're writing a controller, you need to start with the list call but then immediately move to the watch call based on the resource version. And controllers are these infinite control loops running infinitely for long periods of time. So it needs to store a lot of event information. So it's very recommended that you use a data structure or some cache when you're writing in Python. So there are certain options available in Python. One is, of course, you can use a simple hash map.  One is you can use it to check if it is something like an indexing purpose. You can check if have I seen this pod before. Because when you capture, when you do the list and you capture all the state information of the pods, they may not come sequentially. They may come haphazardly. And you need to actually write some logic to figure out which is the most recent event. And if you're using a cache, there's one problem is that Python writing stuff in Python, there's always a performance delay. It's not like writing in CC++ or even in Go. So the whole flow of this API is to get the events and store it in a cache. There will be certain delays. And so there will be always certain lag between the events in the cache and what might be there in the HCD. So we need to do something like a re-sync. Re-sync is something like a real list every 30 minutes or 15 minutes, depending on how many events we have. And controllers, especially the ones written in Python, when you do the watch calls, they tend to break after a certain time. So what we need to do is we need to look for those network breakages and immediately restart the watch stream call. Else, your controller would constantly be re-initializing all the time, and it would never get a chance to do the business logic. And one other thing with the Python controllers is that when you receive these events from the watch interface, you need to do some kind of validation also. You need to validate the resources. For example, if you're creating a controller for a custom resource, you need to validate the input fields of this custom resource events. Because we don't have enough tools to do the schema check of the resources. So if there is a spelling mistake or there's an incorrect ASCII key in the event captured, it might lead to some net failures. So also have proper default values if the event state change of the resource does not have values for certain fields. So now we come to writing controllers. So we learned about some of the design considerations we need to take care to keep this controller code running for a long time and not failing and always doing thing what it is supposed to do. So there are some options to write it. One is, of course, the Go Client and the Python Client. The Go Client, you can access the Go Client here. I'm not going to talk much about the Go Client. But the most important thing I want to talk is that Go Clients have a few data structures which the Go Client have some data structures which are very useful because one is the Informer data structure. What it has is that it has a cache implementation and it has a wash interface and event handler and Lister so that when a controller is using the Informers, you can basically use this data structure for writing the controller. It can do the listing. And then it can just get the latest resource version and then watch that resource version. And the Go Clients also implement a re-sync functionality where it tries to update the cache frequently. And another data structure the Go Clients have is Work Queue. So there are situations where, for example, you create a pod and you delete a pod and then you create a pod. Let's say that's a flow. And in between this creation and deletion, assume that your controller fails. And when the controller restarts, it's going to capture a whole bunch of events related to creation and deletion. But since the deletion event and the control failure happened simultaneously, the controller never actually deleted the pods, although the delete event has come. So if you have a Work Queue, what you can do is you can prioritize deleting those pods before scheduling the next creation of those pods. Else what you'll have is in your cluster you'll have multiple pods which were supposed to be deleted but they were never deleted. Unfortunately, in the Python client, we don't have these two data structures. So if you're writing controllers in Python, you would eventually have to implement some of these. So the next thing I want to talk about briefly is an operator. So far we looked at a controller. Operator is pretty much a controller which manages a custom resource. So I previously talked about resources, like the pod entity resources and deployment configs and the application controllers, et cetera. They are the default object entities of Kubernetes. But you can go ahead and create your own custom resource. For example, here's a custom resource definition of a foo resource. It basically does nothing. On the left-hand side, I have mentioned how you define user-defined custom resource. And on the right-hand side is instance specification of the foo custom resource. It basically does nothing. The specification, if you look, all it has is just a version number. So you can basically write operator, which is in Python, which just looks at this foo resources and tries to do something with it. Maybe create a foo resource and then create 10 pods from the specification. I don't know. So when it comes to Python, the first option you have is the OpenShift REST client, which is based on the Kubernetes Python client. This is used, if you look at the topmost right corner. It is very heavily used in many examples. Used by almost 321 projects so far. And I have a few examples, like the example which I showed here is actually using this particular project to capture the events for pods. And what I wanted to show as part of the demo was to show the ad modified events for a pod. And this uses the OpenShift REST client. And the next option is the Python OpenShift Client Python. The difference between this and the previous one is that this is based upon using OC. So if you are happy with using OC in a pod, then this particular client can be helpful. The one drawback, of course, is that this does not implement a watch call. But I have seen a few examples where they have used both the clients. They have used OC using this particular project to do a few stuff. And it uses the other project to do a few other stuff in the controller logic. And I have recently, in the last two or three months, came to know about this particular project. It stands for Kubernetes operator framework from Zalando Company. So this particular project implements, it doesn't implement informers, but it implements certain work queue and event handlers. But it's not, I don't know. Personally, I don't know if it's very popular. Like if you look at the right hand side corner, it is used by about 14 projects compared to the Python client. I work in the AICOE team. And in the AICOE team, I work in the Thought project. And we have developed quite a few operators and controllers. So if you go on to this GitHub Thought Station repo, you can basically search for a couple of Python operators and controllers, which we have written. And if you plan to do something, you can kind of use this as reference. The last example I want to talk about is operator SDK. When you write controllers in Python, this is a lot of work. One is you write to your Python code, and you might have to implement informers or work queue or some event handlers. But you also need to package them in a Docker file. You need to create this container image. And you need to, let's say, take care of security policies all back. And what operator SDK does is that it basically does all this work for you pretty much very easily. So all you need to do is just focus on writing the controller code. But the problem is that this is go-client-based, and it's not based on Python. And what I've heard so far is that the operator SDK folks with the Python client, they're working on an informer in the Python client. So eventually, maybe in some time future, who knows? Maybe we will have a Python operator SDK. So I have one more demo where I have written a simple pod controller which is using the operator SDK. And this particular example, again, does the same thing. It just tries to capture the events of pods. All these examples are available on this GitHub repo. And you can look up and experiment with the examples. So the first folder contains the controller dev environment where you have the OC running in a pod. It has all the templates. If you want to just do an OC create template and do an OC new app, you have that for you to experiment. And so there's a MyGo controller which is a simple controller written using operator SDK. And there's a Python controller which contains the template. And it also has an example where if you want to write an operator with some CRDs, there's a sample CRD which you can experiment with. And the source has a couple of code for writing controllers for CRD or writing controllers for existing entities in Kubernetes. So that's it from my side. In conclusion, I just wanted to say that to write a very good controller, you need to have a good understanding about Kubernetes and how to interact with the API server. And also be a little conscious about Python layer itself that there are certain concurrency and scalability issues with that. And so do not, I would say, don't write very extensive hardcore controllers if you have a very reasonable amount of logic which can be done in Python. I think Python controllers are a good option. But if you need production grade requirements of a controller, you need to write in a Golang as of today. So thank you so much for coming here. I can take some questions.  In the Python world, you're battling the GIL. And my correct understanding, the controllers are single threaded, right? It's a single control loop that's going to be running continuously. Yeah, I mean, regardless of, because the controller is. Hi. The controller has a single control loop that's just cycling through continuously, right? So if you need to go off and do a significant amount of work in Python, is it just in case we say, OK, as soon as we get to this state, rewrite and Go? Or is there a way that you typically handle those kind of like long running IO tasks? So if you're using Python 2.7, they're limited options. But if you move to Python 3.6, this is a sync library. You can explore that to do that. And also, Python concurrency, there are problems with it. So there are certain patterns where you can, I think, use process, multiple process, where you have one process, which is the control loop. And you can have other processes, which are basically communicating with each other via, I don't know, locking mechanism or some shared cache or something. But it's tough. If you go to the thought station, one of the controllers, which I wrote, has three or four sets working together. And there's a lock mechanism where the locking mechanism is used to basically stop one on multiplex actions. Hello. We're using AKS. In that case, are we still need to write a controller for Kubernetes? I'm sorry, could you please repeat the question? Microsoft Azure has AKS. It's like an Azure Kubernetes service. It's already probably including maybe controller inside of it. Do you think under a work kind of a situation, we still need to write our own controller? It depends on the business logic. Let's say I'm just thinking top of my mind. If I want to write a controller which tries to connect to Azure service, and let's say spawn off something, I can, earlier I could have written some simple service. Maybe today I will write it as a controller with the CRD. And I can declare a specification. I can say, in the specification, connect to this Azure and spin off so and so thing.  If you're designing a system with autonomous thing in mind and you want to isolate the business logic to a certain, like a brain, you would try to do this. And basically it's helpful because if your other stuff or other business logic is also written in controllers, this kind of scales pretty well. So that's an advantage. I would say write everything in controllers and operators. Thank you. No more questions. Thank you so much for attending my talk. Thank you very much.


Metal oxide nanoparticles are essential technological and biomedical materials with high biomedical performance and low toxicity, which can be used in various fields of nanomedicine such as tissue therapy, immunotherapy, diagnosis, dentistry, regenerative medicine, wound healing, and biosensing platforms. Their antimicrobial, antifungal, antiviral properties along with biotoxicology were debated in detail. The significant breakthroughs in the field of nanobiomedicine have emerged in areas predicting tremendous application potential and enormous market value for metal oxide nanoparticles. This article was authored by Maria P. Nikolova and Murthy S. Sowery.