We have Eshwardhan Kukreja with us. He'll be talking about writing your own Kubernetes operator in Python with COP to enhance and harden your Kubernetes clusters. So, Eshwardhan, the stage is on yours. Sure. Thanks a lot. So, hi everyone. I'm Eshwardhan and I hope you had a great time with all the talks till now. So, today I'll be talking about writing your own Kubernetes operators in Python with this framework called COP to enhance and harden our clusters. So, moving on, let me first of all introduce myself. So, I'm Eshwardhan and currently I'm working as a Site Reliability Engineer at Redact. And in my free time I'll have to do open source, they were on my favorite cloud native tools. Also, I love to read about distributed systems, astrophysics and just to break the programmatic technological monotony, I love to do long distance cycle writes as well. So, moving on, let's just super quickly talk about what Kubernetes is because it's the most fundamental piece of tech in our talk, right? So, Kubernetes is basically a container orchestrator and assuming that you have a very basic idea about Docker containers, so I'll just keep the lingo in terms of containers only. So, yeah, basically Kubernetes manages schedules and basically automates a lot of managerial manual stuff around dealing with containerized deployments. So, just spinning up your containers in a reliable manner, deploying your applications on containers, reliably managing them, even debugging them and architectureizing them in a very mature manner is done with the help of Kubernetes alongside other countless things. So, yeah, but from a system administrator standpoint it saves a lot of time and effort. So, moving on, so basically Kubernetes, it's safe to say that Kubernetes is one of the, it's the most majorly adopted container orchestration platform out there. This means that it serves a heck lot of functionalities and use cases out there. But for a business, like at times you might have some custom requirements from Kubernetes which are not natively present into it. So, the beauty of Kubernetes and actually one of the best things of Kubernetes is that it's very pluggable in nature, which means that you can actually develop your own add-ons and plugins which are programmed to perform some custom functionality and they can be easily, very easily integrated as a part of the Kubernetes stack itself. And another great part is that you can even use some third-party plugin or add-on to serve this functionality. So, these add-ons and plugins are called operators.  Like for example, some resource is getting created or deleted or updated and accordingly performing tasks as a response to that. So, and basically, if you look at a Kubernetes cluster and see that, okay, as a system administrator, I'm performing one manual repetitive task again and again and again, then that serves as an opportunity to write an operator. Now, if it seems slightly complex, don't worry, I'll exemplify this further for you. So, but before moving on, let me just first of all quickly talk about custom resource definitions. So, if you have even a very slight idea of Kubernetes, you know that Kubernetes has a lot of resources, right? Like pod deployment, stateful set, replication controller, etc. So, similarly, you can actually create our own custom resources as well, which means that if you have a custom resource with the name MyCustomResource, you can actually write the email with the kind MyCustomResource. You can actually QCTL get MyCustomResource. But to actually make your Kubernetes cluster aware of this custom resource, you define it first of all, right? And how do you define it? Well, you define it with this file called custom resource definition. So, basically, it resembles the schema of your custom resource, because of course, your custom resource, when you'll write a YAML for it, it will have a structure, right? So, the custom resource definition will encapsulate all the information around your custom resource. For example, what is its name? What is its, you know, basically structure, etc., etc. And then your operator behind the scenes will watch for these custom resources. And whenever these custom resources are created, updated or deleted, the logic of what's going to be performed, that's going to be written inside the operator. So, operator is kind of like the back-end side of custom resources, whereas these custom resources are the front-end side of, you know, the user. So, moving on. So, if I give you a very simple example, and actually this is what we're going to implement today in the talk.  And if you look carefully, it's a custom resource. This is not a native resource in Kubernetes, right? Like the API version is my name, the kind is PostgreSQL. And basically, what this custom resource is meant to do is that whenever it gets applied now, the table field inside it, and the other values like name edge content inside it are fetched by Kubernetes and then written to a remote PostgreSQL database in the provided table. And the logic to do so, the logic to actually parse this custom resource, and parse the table value, parse the name edge country values, and then create an insert query and write to the PostgreSQL, that logic is programmed in the operator. So, operator is kind of like back-end in that sense. So, how can we write our own operators? Well, in Golang, there are very sophisticated and great libraries such as ClientGo and even Controller and Time, which can be used to write operators. But they're pretty complicated. But even to help us further, there are really amazing projects like CubeBuilder, Operator SDK, which in just one command basically establishes the whole boilerplate code for you. But the problem is that although they set up your whole project for you, they bootstrap the project for you, they're still complicated and hectic to deal with because there are a lot of Go files, packages and whatnot. So, as usual, Python comes to the rescue. In Python, there is a very awesome framework called COPF, which is Kubernetes Operator Pythonic Framework. It is a framework with which you can write Kubernetes operators from scratch using Python with ataries. And I mean, there are no extra files like literally to write a basic operator, you need one Python file and one Docker file, that's it. That's the beauty. And all of you know how straightforward Python is as a language, so the convenience is top notch. So, I hope you folks are hyped enough. So, let's dive in and code our own operator. So, as I present, as I exemplified about Postgres writer, so we're going to be implementing this one only. So, let's move on. So, first of all, we have to, let me just again, present you how our custom resource is going to look into action. So, after everything is ready, as a user of this operator, as a user of this custom resource, I'll just simply write this custom yamble, apply it over my cluster and behind the scenes, some magic is going to happen. And in my Postgres database, this row will be created. And I will also program a logic to ensure that whenever this custom resource is deleted from my cluster, automatically the deletion is synchronized in the Postgres database by deleting this row. So, moving on. So, as a first step, first of all, we have to make Kubernetes aware that, okay, what is this kind Postgres writer? What is this demo.sh.com? So, for that, we'll define the CRD. Now, don't get freaked out. It's a very simple CRD. I'll just quickly go over it. So, first of all, everything has a name. So, even custom CRD has a name and the name has a format. So, the name has to be in the form of your resource name in plural form dot group. Now, what is a group? Well, let me just go back quickly and notice the API version. So, if you notice the API version, it is in a format. The format is group slash version. So, the group is demo.sh.com and version in my case is v1 and kind is Postgres writer. So, whenever I... So, these are group version kind and together a group version kind basically, you know, acts like a unique idea of a resource in our cluster. So, in my case, in this situation, the gvk, the group version kind is demo.sh.com v1 Postgres writer. So, that's what I'm doing here. The metadata, the name of the, you know, basically CRD is the plural form of my resource, which is Postgres writers dot group. Inside the spec, I've defined the group which is demo.sh.com, which I just explained. And inside the version, I've defined v1. So, your CRD can have... your custom resource can have multiple versions depending on the functionalities. Now, inside this version v1, I'm actually defining the structure of the YAML to expect. So, if you look carefully, the structure here is expecting a spec section. Inside it, there are four fields called table, name, age, country. And each of them have a data type, right? Table has a string data type. Age has an integer data type, etc. So, all of this stuff is defined carefully inside the schema. You can see that, okay, there is a required field spec. And there are required fields like table, name, age, country with their respective data types. Then at an operator, at a custom resource level, I've defined the scope as namespace. Basically, this means that I can apply this resource over different namespaces. You can also have some certain, you know, use cases where you want just your custom resource to operate at the cluster-wide level, not specifically like at the namespace level. And if you are aware of cluster roles and cluster-roll bindings, they are similar in that sense. But don't worry. So, in my case, the scope is namespaced. And finally, I've defined the name of my operator, of my custom resource, right? So, the plural is going to be Postgres writers. Singular is going to be Postgres writer. Kind is going to be the Postgres writer in a camel case. And short name is PSW. So, all of these names are basically, you know, for doing cube CTL get Postgres writer, cube CTL get PSW, etc. So, this, I promise this was the complicated part. Now begins the fun part. So, first of all, now comes the, now, once we define a CRD, we are done with, you know, telling Kubernetes that how our custom resource will look. But still, what's going to happen behind the scene? That will be coded in an operator. And this is where we begin. So, first of all, we'll import the relevant libraries. So, the first one, you know, the PsyCop G2, this is for, you know, talking to Postgres database and, you know, inserting and deleting stuff. Cops is basically a framework to write operators. And then Kubernetes library is being used to basically, you know, make our operator communicate with our cluster, right? And finding the OS library to get environment variables. Now, the first function basically I've written is that to initialize DB. So, basically, of course, first of all, you have to establish a connection with the database, right? So, I've already set up a database of Postgres on this specific IP.  But in this case, this function is returning me the connection to the database. The second function is initialize queue. And it is doing one thing. It is basically establishing the connection between my operator and the Kubernetes cluster so that my operator can talk to Kubernetes cluster and accordingly, you know, watch for the customer resource and take actions. Now, you might see that there is an if and else present here. Now, the thing is that you can run our operator in two ways. First of all, is obviously you can just package it as a Docker image and deploy it on your container or deploy it on your cluster, which is the normal productionized way. But while developing an operator as a developer, right, it's, you know, hectic to every time when a code changes, you package it as a Docker image and deploy it. So for the sake of convenience, you can literally run your operator locally through command nine and it will talk to your Kubernetes cluster via a cube config. So whenever you do kubectl get something, like that thing is routed to your cube config, right? So similarly, this operator will perform at that level as well. You know, a dev environment variable to tell my code that, okay, how to talk to my Kubernetes cluster, you know, when I'm developing locally or I've deployed it. So moving on. Now, this is the main helper function which will insert a row into the table. It is very straightforward. It is just performing an insert query into my database. It is taking my database connection to a variable and you know, feeding the table name, the name age country into it and just throwing that piece of row inside my table. And as I told you, we're going to be handling the deletes as well in a graceful manner. So I've defined a function to delete data as well. And again, it's a super easy delete query, right? So now moving on. Now, this is the bell of the wall. This is the piece of code. This is a function which will watch for the creation of my custom resource. Don't worry. I'll explain it. So if you look at the first line, I'm just like it's very self-explanatory, right? I'm just telling cough dot on create and then inside I'm defining the gvk. Again, remember what gvk was? It was group version kind. So for, so whenever cops, basically my operator, whenever it will notice the creation of a resource of this gvk, of this group version kind then it's going to execute the following piece of code. And the following piece of code is basically doing nothing except extracting the table name, the name, the age and country fields and throwing them inside the table of their postgres database, which my operator is aware of.  That resources namespace slash name just for the sake of establishing uniqueness. So go super quickly back. You see that this table, like this record which got created. You see the ID, how it looks. It is namespace slash the resource name, the one which is defined here. So that's just for the sake of, you know, keeping things unique and graceful as to what any conflicts. So yeah, again, this operator, whenever it captures the creative end of this gvk of my postgres writer, it will essentially, you know, table name, age, country fields and throw them into my database. Similarly, whenever it will catch the deletion of a resource of postgres writer, my custom resource, it's going to again capture the spec of the resource which got deleted and accordingly, you know, look at the namespace and name and accordingly delete the corresponding row which was present in the database just to synchronize things and ensure that the database contains only those rows, those records which are actually present in the Kubernetes cluster in the form of postgres writer resources. And running the operator locally again as a developer, you can run the operator locally with this command called cop front, then your Python file and it's going to run beautifully. So let's just quickly hop back to our terminal, right? So let me just actually increase the, yeah, because my display is too, okay, so I'm afraid it's not going to be visible to you folks. Let me just, I hope the display is not fine. Perfect. Okay, cool. So right now I'm inside my project folder and first of all, let me just show you, let me actually, you know, get client level access to my postgres database. So right now this postgres database has, you know, this table called students which are already created and basically, so yeah, it is just a second, just a second. Sure, then I think the font size is too small. Could you please increase? Oh sure, I'll increase it further. How is it now? A little better, but it can be. I mean, it should be readable now, I guess at least, or is it still not? Or I guess I've increased it on my end, but it hasn't synchronized on your end. Is it still not enough? It is better now, it is better now. Okay, okay, I guess it was delayed a little bit. Anyways, never mind. So if you look back to the talk, so if you look at the left terminal, you can see that I'm a table created and a student's table doesn't have any rows and stuff. Not do one thing, I'll run my operator locally and but before running an operator again, let's go step 5. Let's first of all create the CRD, right? So I'll just create the CRD again, again, the CRD file looks exactly like the one which I demonstrated in the PPT. So I'll just QTTL, apply CRD and done. Now if I click a little QTL, it will get CRD. So my CRD exists, right? Now if I, before applying my custom resource, let's first of all start the operator because QTL has to be aware that what's going to happen when the resource is created. So in the lower terminal, I've said the dev is equal to true environment variable and now quickly I'll run my operator. Perfect, so now this is a sign of the operator running successfully and if I define this sample.amble, basically the sample.amble is the exact custom postgres writer file which I'll apply now and carefully notice the values, the table is students, the namespace is, oh sorry, name is sample student and namespace is default. So if I, let me just keep it here and if I apply the sample.amble, yep, it got applied and you see some logs are captured here. And now if I, again, now if I go back to my left-toilet terminal and actually see that if my database got updated or not, yep, see, the data got reflected. So behind the scenes, exactly the way it was expected to happen, the way it worked. And similarly, if I do kubectl get postgres writers, you can see that in the default namespace, this resource got created. Now if I delete my, this postgres writer, right, and samples student, oops, yep, yep, it got deleted. And again, the operator was able to catch this deletion and hopefully it deleted the row in the database. And if I go in the database and actually do select star from students, perfect. So the creations were synchronized well with the database and the deletions were synchronized well with the database. Now this is, like this proves that the operator is working perfectly fine. Now this, but still this is like the way I executed the operator, it was very local developmental way, right? So we have to deploy it the right way, right? So your operator, in a productionized manner, it has to be first of all packaged into a Docker image and deployed over a cluster. And then in that cluster, it will run as a, you know, this thing a pod and serve the functionality which it is supposed to. So let me just create a namespace called postgres, where I'll be keeping all the resources associated with the operator. And if you notice, I have a deployment defined which basically contains, which basically encapsulates the logic to run the, you know, my operator and my operator's image is going to be postgres-write. So first of all, let's quickly build the image of my operator. So I'm building the image of my operator and yep, it's built. And ignore this last command which I'm executing. I'm just making use of this tool called kind for Kubernetes testing. So ignore this last command which I executed. But yeah, I'm packaging the Docker image with the name postgres-writer latest. And once I apply this deployment, my operator will be spun up as a replica, as a pod actually. And this pod will actually serve this whole core functionality of the operator. But before that, we have to also provide our operator enough permissions to watch over this custom resource and take those necessary actions. Because Kubernetes does not believe a new incoming piece of code, incoming piece of operator. So I'm doing one thing. I'm doing called a service account. And that service account is basically defining something like an account with the name postgres-writer. And the next thing I'll do is that with the help of a cluster role binding, I'm going to correlate admin level permissions with this service account. So never do this in production. I'm just doing it for the sake of convenience. But in reality, you should only correlate the minimum amount of permissions with your operator. But yeah, in my case, for the sake of convenience, I'm just correlating the cluster admin role with this service account with the help of a cluster role binding. So if you see the name of the cluster role binding is admin access. The service account is going to point to is postgres-writer. And the role which is going to point the service account to is going to be cluster admin. So once I define it, now my Kubernetes cluster is aware that whenever the service account, postgres-writer will be used, it's going to correlate to admin level permissions. And if you notice carefully in the deployment file of my operator, you'll notice that I've defined the service account name, which means that the pod, which will run my operator, it will have the accesses associated with this service account called postgres-writer, which indirectly has admin level accesses. So now I'll run a deployment and hopefully it runs. So it got created. So let me do pods, postgres, gorgias. So it's running fine. The running status is an awesome green signal. Now let me just screen the logs of this deployment and see the logs are here as well. So again, we're going to do the same thing. We have already defined the CID, so we won't reapply. So we will again apply, basically the sample YAML of the student, which I applied previously for the sake of example. And before that, just to confirm again, this student stable does not have any rows. Again, it does not have any rows. So once I apply this custom sample, the student got created and you can see that actually the logs were captured. And if I again hop back to my postgres terminal, the resource were created. Now this is a perfectly working operator, which is perfectly deployed in a productionized manner. And if I delete this resource, then again logs were captured by my pod, my operator's pod. And if I look back into my postgres terminal, the rows were deleted. So everything is working perfectly fine. Just a few things I would like to mention. See the piece of code which I wrote, actually it's not very mature piece of code because I just want to demonstrate how an operator works. But in reality, you would have to handle exceptions in a graceful way, ensuring conflicts and whatnot. So just a heads up regarding that. So the deployment file, which I wrote, I'm already inside the folder. So if you notice, okay, my terminal is missing up. If you notice I'm providing the password and the username and all this config directly as plain text, never do that. Please never do that. Please always inject your environment variables in the form of config maps and secrets accordingly. Although secrets don't exactly encrypt your data, they encode it, but it doesn't matter. Additionally, you should inject your environment variables in the form of config map references and secrets. So again, I just did not do that here just to keep things convenient and straightforward for you. But yeah, just to keep the things out there. So this was my presentation. And so the key takeaways from this presentation is that we saw what Kubernetes operators are and why are they so cool. And we also learned about CRDs. We depth deeper into coding our own Postgres writer operator and deploying it. And we also, you know, see like how we actually practically saw in the terminal how things worked out. And you can check out this slide deck, the code to the above operator on my Github profile with the name yashvardhan-cookreja, yashvardhan-cookreja. And that's pretty much it. And also one more thing, actually, I forgot to mention, you might have noticed that for deploying our operator, it was pretty complicated, like installing service account, cluster role binding deployment rate. So all of this stuff can be automated as well. And there are a lot of tools out there to automate this stuff. So one of the most commonly used tool is called Helm. It's spelled H-E-L-M.  And all the, as a user, what I have to do, I just have to provide my, you know, basic variables. Like for example, my Postgres host, username and password, and that's it. REST Helm will handle for me. It will automatically feed in my variables into this deployment, automatically apply them, set up the namespaces, et cetera. So do check out the Helm project as well. So I guess we still have around two to three minutes left, or three to four minutes left, for taking any questions. So yeah, my four questions. And also I'm free to reach out to, on my Twitter and on my GitHub and on my LinkedIn as well. Although I haven't put it up here, but yeah. So any questions, feel free to shoot them. Girish, I can, I guess I can take up some questions, if there are any. Yeah, there are questions coming in. The first question is, can I use Kubernetes operator instead of using job when I'm deploying a backend application? And I want to run the Postgres operations like migration sectors, et cetera. Sorry, can you repeat the question? Your voice broke a little bit, I guess, on my end. Okay. So the question was, can I use Kubernetes operators instead of using, instead of using job when I'm deploying a backend application? And I want to run the Postgres operations like migrations, et cetera. Yeah. So see, I guess this is a very simple use case. And I guess if I were in a position, I would have written a simple cron job only. See, because the thing is that in your use case, right, you do not need to necessitate upon establishing a custom resource and, you know, basically making use of the reconciliation logic of Kubernetes, right? So, okay, I'll give you like a certain use case or, you know, basically a perspective where you can see and actually notice that, okay, I can use operators. So operators, don't look at operators as something which your application developers would use. Rather look at them, what are cluster administrators you would use, right?  But the application level stuff, like, you know, basically performing some script operations or a basic script which happens periodically, the Kubernetes cron job will do most of the job very easily. Okay. The second question is, what are implications of providing all permissions in cluster role binding while application deploying production scenario? Oh my God, it's very messy actually. Because again, see, like most of the times you're going to be actually making use of third party operators, right? So in that case of scenario, if someone is malicious they can actually inject a piece of code in their operator code. Maybe inspect all the resources in a cluster, inspect the secrets in a cluster and dump them somewhere out there, right? So that's risky.   So by mistake, if you coded something messy in your operator which performs some deletion in your cluster, right? So there would be anything there like a gateway to stop it from doing that. So just to avoid the risk of, you know, your own human from mistakes or someone else being, you know, getting malicious and privileged access to your cluster. Please do ensure that the level of service account and the roles assigned to the operator are minimal. Right. There is one more question. So what are the performance obligations of using Python to write an operator as opposed to writing one, say, Golang? Honestly, I went like benchmark both of them and honestly in my previous organizations we heavily use Python based operators and Golang based operators as well. See, there are fundamental performance benefits in Golang, right? Because it's like directly compiled into machine code and it has great performance level benefits. But I guess at the framework level there is not much difference. And honestly, like in my previous company called Grofers, we actually heavily used our operators in which we written a code and there were, you know, alarming or critically slow issues with Python which, you know, caused us any issues. So there are very minimal differences. But from my experience, you know, dealing with Golang and the level of, you know, gigabytes, I would prefer Golang to code operators and the performance benefits are also more than Python for obvious reasons. Okay, I think there is one can you define some use cases for Python operators that you use personally? Yeah, so for example, this post writer one, right? So this is exactly like something very similar to this we used in a previous organization. But instead of that, so basically in a previous organization we, you know, use the software called console KV for, you know, basically this thing called configuration management and we wanted to establish a practice where, you know, people instead of directly writing the key value pairs over their console web portal, then instead applied them on Kubernetes and some operator fetched them and threw them into console. So this interfacing and for this we created our own operator and apart from there are countless operators, right? For example, aqua six, oh sorry, yeah, for example, there is an operator called starboard, which is provided by this company called aqua security with which, you know, basically which constantly watches for your incoming deployments and pods and actually inspects them to see that if they are vulnerable or not. For example, if they're having unprivileged is equal to true or host network mode is on something like that. So these kinds of use cases I think we are almost out of time. We can take one more question and that is, can I interact with other resources slash CRDs using a CRDI, right? Yes, you can do that. So see, again, operator is just a fancy term behind the scenes. This thing is a controller which we wrote, right? An operator is basically a controller which is watching over CRDs. So what you are saying is that you are basically writing a controller which watches over other custom resources. So of course you can do that. The way I defined in the PPT, right, if I go super quickly, so here I defined the feature of this my resource set, I could have defined the other custom resources as well and then my operator would be, you know, watching the other resource and taking actions accordingly upon the creation, updation or deletion. So yeah, of course you can do that. So yeah, we are out of time. So I think there are no questions. I think we'll have to take those offline.


So just before we start, were there a lot of changes in S7? I thought it was a strike all, right? Yes. So I can tell you off the top of my head. So it went to two committees in the House. It went first to health care. And the major changes there were date changes in terms of when the reports were due. And then there was language in sections one and two, which are both reporting sections that say stakeholders have to be consulted when coming up with the report. So those are the two changes from health care. And then the change from human services are two changes. In section four, which was the section on the report back on Dulce model and on the home visiting, that section was struck. And then in section one, which was the report that's required by CMS about ACOs and that the Human Services Edit Language that said take into consideration future plans about integration of long-term services. Okay, we'll look at those two. I'm just... The internet is not working. Here we are. We have a seven is passed by the Senate. We have S146 proposal of amendment. Let's look at that first. Can I have her copy? I'm sorry, I can't get mine. Is that working? Yeah, maybe we should each get a hard copy. Because this is so slow. We were considering a cloud tax and it's gone. I said it's her. We were considering a cloud tax and it's gone. Okay, it popped up. We got it. Okay, let's look at it. Okay, so what this amendment does is it strikes out the whole subsection on membership of the new council and adds and replaces it with your changes. So the changes that you requested are in yellow. So first, with regard to the executive committee, we have the commissioner of health serving as chair, not as a co-chair. Second, in subdivision B, we have a community leader in the field of substance misuse prevention, jointly appointed by the speaker of the House and President Pro Tem. And that person is no longer a chair. Next, with regard to the members of the council more generally, we have language that members of the council shall collectively offer expertise and start over. Members of the council shall collectively offer expertise and experience in the categories listed below with the understanding that a single member has the expertise and experience in multiple categories to capture that idea that one person could serve. Okay, let's go through it all and then we'll make a suggestion. Okay, good. And then there is just one last change on page three, line four, and that was replacing substance use disorder, substance misuse prevention with youth Vermont population. And now it reads community-based nonprofit services. But we still have substance use disorder or substance misuse prevention in the older population. And what did that replace? It replaced the one about youth in a substance... I think it contrasts pretty closely. Substance use disorder, substance misuse prevention within the youth population. Oh, okay. Okay, all right. All right. So let's go through each one of these. And I did have a conversation with the administration about the sum of this. And I think that there... I think that if we go back to the governor appointment but lead the community leader as a member, that would be probably better received instead of having a pro tem and a speaker. Well, okay. Yeah, I do think one of the most important things is that we don't have co-chairs because I just don't think that... Not having the co-chairs, I think, was the important piece. And the other thing is we can... We're going to look at this thing unfold and sort out what happens. So as a proposal of amendment KAH do this, but can we appointed by the governor? No. I'm sorry. I hated to do this. No, it's nice to have your mind made up, isn't it? That's why people born in September like this. Never make a decision. So, the other question I would have is, okay, right now we have a chair but no vice chair. Do we want to suggest that the community leader be the vice chair? That's no big key. Yeah, I don't know if I need to roll up. I don't either. Whatever you'd like. I don't. You have three votes. Yeah. How does Sheila Livingston care about that? That's the jailor from the health department. That's fine. Okay, let's put vice chair in. We still have some leadership identified. Now she gets four votes. Yeah. She promoted Sheila to senator history. I know what she's saying. Deputy commissioner. Okay. Is that it? All right. There are two. Ensuring that the categories might be represented by one multiple categories for one member. That was good. Do we want to say anything about ensuring the wilderness? The size of the council. The total size of the council. Is manageable. Something. Do we want to say that or should we just not say anything? Yeah. I feel like it's a little micromanaged. It seems like that. We'll just avoid that then. That's good. Just so you have it straight. And then the last one. The last one. Is there a problem with that? I think. Actually, that's what representative from Kelly. Kelly. That's right. Yeah. Suggested anyway. So let's do that. So this will be our proposal of amendment. A further proposal of amendment. Back to the house. When we get it. And it is on the calendars. So we could do it this morning or this afternoon depending on your schedule. Probably this afternoon it feels like. I could probably get it too by 10. Okay. And just so you know, I may want to run it by the house. Just to make sure that we're not running into a brick wall. Okay. All right. Thank you. So now on to S7. Okay. Okay, we need Linda's going to get the one past my house. I don't know if I've got that. Okay. So it's about today. In the house. So there is an aversion as past house. It's just a committee report. It's just a committee report. It's the house health care committee report and the house human services committee report. And we have this fiscal note from no one on 146 this December. I think it will be great to have S146. Yes. Is there a reading? I need a overview of what S7 is. Remind us where we are. Sure. So this is an act relating to social service integration within Vermont's health care system. If you remember, it started off as just a single section bill that was asking for about the integration of social services with the ACO. And so that section is still in there. Section one, you added asset or report that's required of the agency by CMS. Also come to this committee and to the committees upstairs. Then as I said, section two is that report about service integration. Section three added no, there is a section two A which dealt with the budget review of the accountable care organization and added a new criteria. And then section three, which was the director of trauma prevention and resilience development, the responsibilities of that individual and you added a duty for that individual. Section four was the report on the Dulce model and the home visiting programs. And that was it. So... Okay, and so we'll look at the changes. Human services make changes to section four. Yeah, so there are two committee reports. I almost wonder if it's easy. I pulled up the calendar, the house calendar for today. Both committee reports are one right after the other. So if you were to pull up the calendar for today and go to page 2726. Let's do that. Then... Because we don't have the bill yet, but obviously times of the essence. 27... 2726? Yeah, it should be an easier way to get to that. A word search. So is this on... I didn't chat. Is this on notice today? No, this is up for second reading. Okay. Okay, so we're not that... Okay. Do you have it, Rick? You'll find it. Okay, go ahead. Okay, so the first report is the report of the health care committee. And as I said, the main changes here have to do with date changes. So in section one, subdivision A1, the report is due on or before January 1, 2021. Had not aligned with when CMS was requiring the report. So now, this is more an alignment for the agency with when the report is due. And then there's a new subdivision A2. This is in preparing the report. The agency shall consult with individuals receiving services and family members of individuals receiving services. So that is the number two is a new addition? Yep, A2 is a new addition. Similarly, in section two, there is a date change for when this report is due. It's now due December 1, 2019. And I'm not a lost for what the original date was. But again, in this section, let's see, in the big first paragraph after the cross-reference to 18 BSA 9382, there's a new sentence. In preparing the report, the board shall consult with individuals receiving services and family members of individuals receiving services. So it's that same sentence being added in both places. Okay. And then the next section is 2A, Oversay of Accountable Care Organizations. And this is the budget review process. And I don't believe any changes were made to this particular section. Section three, the Health Care Committee did not make any changes to the Director of Information and Resilience. So that is the same. Similarly, section four, the Health Care Committee did not make any changes to the report back on Dulce and Home Visiting. So that's the Health Care Report. And then next is the Human Services Report. And as you'll see, there are two instances of amending the Health Care Committee's report. So the first change is in section one, striking the whole first paragraph and inserting a new paragraph. But the only change is the last clause at the very end of that paragraph is in addition, after All-Pair Financial Target Services Common, including future plans for the integration of long-term care services with the Accountable Organization. Okay. And as it reads currently with the Health Care or with ours, I would suppose that how do we, what do we say about that? Anything? Okay, we'll have to look at this. It's on our website now. I'm going to go to that because it's easier from here. I have to go back and go back. I lost where I was. Okay, so we can look at the Human Services Separate from the House. Thank you. I haven't voted on yet. So the Human Services Report amends the Health Care Committee's report. Yeah, okay. So, but essentially the Human Services Committee the Health Care Committee didn't do a whole lot to the bill as compared with what Human Services... So Health Care made the date changes and they added that language so... Yeah, stakeholders. Human Services again, they struck the whole first paragraph but they replaced the paragraph of nearly identical language except the last clause and that first paragraph is new. The clause including future plans for the integration of long-term care services with the Accountable Care Organization. But then they strike out a section four in its entirety. Yes, so that's the second instance of amendment that has gone. So that's the language that don't even know visiting. Why did they do that? Do you have any sense? I mean, once we have the bill we'll have someone come in and talk with us about their rationale for that. Yeah, I think that would probably be the best thing to do. I know the liaison to Health Care spoke in the Health Care Committee and referenced the fact that the Director of Trauma Prevention is already looking at some of these issues and has, you know, general direction to be looking at this. So, like any more specific information that I suggest you have somebody from that committee come in and chat with you. Okay. All right. When we get this from the House from you on our website page can we have the bill with the changes highlighted in the bill as a whole? Yeah. I'm trying to think of a better way to do that. Yeah, I could just highlight the new language for you so you can see that. Because then we can determine what might or might not be missing based on our original intent. Okay. Okay, let's just say that let's go through this one more time. Sure. You've got seven minutes. May as well use you while we've got you. Yeah. So you want to go back to the Health Care report? Is that what you were thinking? I think let's go, let's start with the Health Care report. Okay. Let me find it. I've got two human services. Okay, time out. Oh, hang on. You have a House Health Care report. But why does it say Human Services? What do you mean? I don't have it. Two different reports, one from Human Services and one from Health Care. Yeah, but it didn't come up. When I say Health Care, I'm getting Human Services. And here, put Human Services and see what you get. You got that too? Yeah, I got both of them. So I have two of the same. Okay. Wait, wait, maybe it's not. Hold on. No, they're both the same. Yeah, they're both the same. So you need to have I have passed by Senate. Yep, passed by Senate. Yeah, and I have Human Services but with that. Just coming in, House Health Care report. So between now and when we go through this thing again, we should probably read what we did and then read what they did. It looks like our reports were way earlier. Yeah. For what then? 2021 is when it's after the next election. People will forget. Why did they put the reports later? Who knows? I know we'll have to get the report or the billing. The first one, there was testimony that that report was required by CMS and I believe it's not required until December 2020. And so they were trying to align the dates more closely. I understand. Yeah, that's understandable. And the second one, I don't recall specifically why that date was changed. Okay. She's got it. She's doing it. Okay. I can speak to why the date was changed. I don't know what to speak about. You have an expert in the way that's why the date was changed. Oh, okay. For the record. Well, as miles of the Green Mountain care board for the record, the date was changed from September 1st to December 1st to align with the quarterly reporting that we received from one care and to give more time for claims and more pressure. Okay. The integration. That's understandable as well. Good. Okay. Thank you. Are we good or not? Almost. I'm going to have you guys do it. I know. You missed me. But the internet hasn't been normal since the little outage thing. Yeah. Should we have a decision to say health care, health may report it. Yeah. I'm going to say that. House health care report. Okay. That's what we want. Yeah. I do the health care report. I'm just going to refresh. Okay. Nope. Oh, it's actually under human services report. That's the health care report. It's the last one and the line up there. Got it. Okay. We're good. No worries. Yeah. I mean, let's just go through. We're not doing any voting or anything. Okay. So the health care report. Section one is the report that is due and it's due to CMS. It's from the agency of human services in collaboration with the board. It's going to the appropriations committees, health care human services in this committee and it's a plan to coordinate the financing and delivery of Medicaid mental health services and Medicaid home and community based services and the all pair of financial target services. That's all the same except for the data. And then the new language is subdivision A2 and preparing the report that the agency is to consult with individuals receiving services and family members of individuals receiving services. Subsection B is the interim status presentation, which is January of this coming year and it's to include update on the agency's progress, the process for the plans development and the identities of any stakeholders with whom the agency has consulted. Go let's see. Section two is the report on integration of services. So that report is coming December 1st of this year from the board and it's coming to this committee human services and health care and it evaluates the manner and degree to which social services including services provided by the parent child center network DA's and SSAs and home health and hospice agencies are integrated into ACOs certified ACOs and preparing the report at the board is to consult with individuals receiving services and family members of individuals receiving services. It's that sentence that's new. That's the change. And then the evaluation addresses the number of social service providers receiving payments through the ACOs and for which services the extent to which any existing relationship between social service providers in the ACO address childhood trauma and resilience building and recommendations to enhance integration between social service providers and ACOs if appropriate. Next is section 2A. This is the budget review process and you've added a criteria subdivision P, the extent to which the ACO provides resources to primary care practices to ensure that care coordination and community services such as mental health and substance use disorder counseling that are provided by community health teams are available to patients without imposing unreasonable burdens on primary care to providers or on ACO member organizations that has not changed. Section 3 is the director of trauma prevention and resilience. This list is all of this individual's responsibilities and subdivision 8 was added by this committee serving as a resource and ensuring new models used by community social service providers are aligned with the state's goals for trauma-informed prevention and resilience that has not changed. Can I ask you if this has changed in human services? No, that's not. That's been retained. Thank you. Section 4 is the report of the director of trauma prevention and resilience development and the director of maternal and child health reporting on the Dulce model and the home visiting programs. Health care did not make any changes to this and human services removed this section. Okay, we'll find out why they did that. Okay. Okay. So briefly, the human services report does. So it strikes that one section. And then in section 1 in the first paragraph it adds a new clause at the very end of the paragraph. So part of this plan that H.S. is working on for CMS would include future plans for the integration of long-term care services with the ACO. Okay. I don't know how that's possible for others to plan for what the ACO is doing since they're an independent organization. Huh. Okay. Well, we'll have to talk about that in a little bit. Okay. Good. Thank you. You're welcome. Now we have to wait for a quorum. To move on. So we've done two of the three bills that we're going to do. So if we, when we have Michael Grady we can finish at 55. Okay. Looking at that, because they're all up on the floor, but we don't have a whole lot of time. So thank you. You're welcome. And I think that amendment's here. Excellent. Yeah, good. Soon it'll be better. I think. Okay. Yeah, good. So we're on break. There's someone that's going to come in. We'll just show us the places. Right. So on page one. Line 19. The Human Services Committee wanted the fact that the commissioner has to request the working group to review chemicals for additional listing. They wanted that removed. They want the working group to review the proposed chemicals for additional listing. Okay. They want to see a power and duty of the working group. With that said, on page two line 16, they want the working group to meet more often if they're going to have additional duties and that they're going to be providing serving and advisory capacity to the commissioner. They want they want them to meet. They've only met once in the past two years because that's all required to do. The House Human Services would require them to meet at least two times a year. Okay. And then on page three in what is being provided as a notice by the manufacturers regarding the chemical by concerning the children product. This is the language that passed the senate. The branding product model and universal product code. If the product has such a code. On page four this is changing the time frame for manufacturers to provide notice right now. They provide notice by annually. So every two years and the Human Services wants them to report annually. So beginning August 31 2020 because that's the next deadline for reporting that the manufacturers will provide annual notice thereafter. And then in the rulemaking section on page five page four going on to page five the standard for the commissioner to add a chemical of high concern. Current law says on the basis of the weight of credible scientific evidence you have changed it to on the basis of credible peer reviewed scientific information and they like credible scientific evidence they looked at the standard in other states specifically Maine and Washington and they think credible scientific evidence is more consistent with those two states. And there is a purpose section in the chemicals of high concern chapter that says the purpose is to maintain as much consistency with other states and so they will. But they wanted to reference or include some reference to peer reviewed study. So on the basis of credible scientific evidence including peer reviewed studies. Can we stop there for a minute? So understanding that peer review is the way that science is validated, scientific data and research is validated. Does this open the door to study from a manufacturer that has not ever been peer reviewed except maybe within the manufacturing company itself to being thrust into the working group or placed into the conversation that may be a biased study. I think the existing standard weight of credible scientific evidence allows for that right now. So your proposal that that's the senate having a peer review condition was the first insertion of that concept instead of standard. So right now. So actually that's a good point and meaning they define credible scientific evidence as being in part like one of it is including being peer reviewed. So it is included as a concept of me Washington doesn't Washington doesn't have that. Included allows for evidence that is not peer reviewed. That's including but not limited to. Yeah. It's a concern. It's a concern. Okay. Just wanted to talk about that a little bit and when we we're going to have to obviously go through this is it on the 4th or 2nd reading. So it was supposed to be on the 4th or 2nd reading yesterday and 3rd reading today because of the length of the debate on a couple of those yesterday they went home at 55 with the condition that everything is going to move through the calendar through all remaining stages of passage today. Okay. So we'll get S-55 tomorrow. We'll have that. Okay. It is up first or second. And I do probably need to be there in a few minutes. Okay. We're not going to keep you. I know it's just about the end here. Okay. Good. So and then on page 6 the current rule the standard in order to for the commissioner to adopt a rule to regulate the sale or labeling of a product requires a recommendation from the working group and that is being struck and changed to consultation that was in your bill. What was also in your bill page 6 line 7 the standard when for that rule is the commissioner needs to be a determination that children currently will be exposed and that certainty of will was always a concern and you wanted that changed in May. You have also struck on page 6 lines 9 through 13 eliminating the requirement that there be a determination that there is a probability to the degree of exposure that the exposure could cause or contribute to one or more of their self-defense. The Health Human Services we agree that probability is too high of a standard but they do like this concept that yes there may be exposure but will exposure lead to one of the adverse effects so they wanted that built in and so they replaced probability with possibility. So my only concern here is sort of the quantitative assessment of the degree and frequency if they had taken out those two phrases and just let possibility of exposure might fit better but we will talk about that. This is fresh and this is language that just stepping back and looking at it in an idealistic policy way which makes perfect sense and then you realize how it can be used. Yes. This is meet for years of delay. So removing on page 6 line 14 because you changed will to may on line 7 you want to change will to may on line 14 and then in one of the criteria for determining if there is exposure on page 7 line 1 they wanted to change frequency to potential and likelihood of exposure and then on page 7 line 16 because the working group this is language that was in your bill the working group isn't recommending or doesn't have that conditional authority over the commissioner so if they provide a consultation to the commissioner and the commissioner doesn't follow it what is their authority. This gives them the ability to submit to you their recommendations to provide a check on a commissioner that just ignores their recommendations. Now on page 8 line 3 this was striking the obsolete date the commissioner has adopted these procedure and process rules the date is no longer relevant and then page 8 line 13 to 17 this was language in your bill about what when should a manufacturer who's introducing a product between a reporting deadline have to provide notice that the product has a chemical high concern to children in it and so is it prior to sale or is it at the next reporting period deadline. The director of the department of health to address that and the requirements for when and how a manufacturer of a children's product that contains a chemical high concern provides the notice when the manufacturer intends to introduce a product for sale between the requirements for reporting. And so that the house left that as is but and they left the director for when that rulemaking needs to be done around 2020 and then there's a requirement on a report regarding implementation of the program that was in yours as well the only other change that they made was in the effective dates they wanted that the the rulemaking to take effect on passage to give them an extra month in order to go to rulemaking. Okay. So some improvements and some and a couple of questions that we have and we'll have to the peer review we're going to have to think about a lot but we'll look at that see what comes over tomorrow. Okay. Thank you very much. You're welcome. You too. We did not want the peer review. We just wanted to have there were indications of possible problems that the commissioner would act on that. At least to identify as a chemical concern. We didn't want the I as standard. Well, peer reviewed analysis is really important. Critical scientific information then if Washington State truly has that that's something we could consider. So let's think about it. Again, to think of what the words actually mean is one thing. Think of how it could be used because it seems to me you could say that there is no credible scientific information without the review. Exactly. By definition of peer review. But then we get into the if I were the lawyer, if I were the Sleesbag lawyer for the chemical guys, I would say the legislature could have said peer review and they didn't. So obviously this means something different. Well, Maine has a definition of credible scientific information as including peer review. That is another avenue that we might take. So let's think about that. I don't want to I know that the houses work very hard on this bill. They want to make it a bill that will get signed and pass into law. But I don't want to pass into law something that has a whole 10 foot feet wide. So we need to think about how to do it. A part of me is just thinking, you know, no matter what we do the bad guys are going to hang us up and they're going to delay. No matter what. So let's just pass the damn bill and see them in court. We want to do it right. We want to do it right. We did. But we don't want good to be the sacrifice for the sake of perfection. Yeah. Well that's sort of what I'm saying. I'm willing to live with that one I can't. As a scientist I have a really difficult time. But I think we're going to court no matter what we do. Oh probably. I think our goal is not to give them any more wiggle room than we can. I think that's my concern about credible scientific information. I think I've said this before in here. The EPA just came out saying that life I say does not cause cancer. All the peer reviewed literature indicates it does cause cancer. If you look at the work that comes out of industry you might get a different that's the kind of debate that we really would like to avoid in our state. You know that life, I've been saying life is safe for years. If you can say it any way you want. Well I've been saying Abednake for years too. I've never heard that. But you know I have always resisted the idea of seeing opponents of environmental protections as inherently evil.  Getting along with the people you disagree with is an important thing. I've got to tell you that thing. Like phosphate. That is evil. It is evil. Detection of that is clearly causes it. We'll know it's okay because industry it's a lot of profits being lost if we tell you. There's a lot of cancer being caused. If we tell the truth it's going to cost us money. Destroys our healthcare system and this is what's wrong and right now the company that produces the most chemicals in the world to expand to Latin America, South America continent of Africa, India and to build an increase of between 15 and 25 percent of their production efforts without having standards in place. And that's one of the one of the things that's going on right now with free trade agreement negotiations across the globe. It's sad because we're destroying our water and our land. We're placing stuff into the food chain that is not biodegradable but is bio cumulative and will cause some debilitating diseases. I'm hearing from people how you I hear from people I respect and trust that 5G stuff is just a lot of alarmism and it's all fine but I once was denigrated as a new cook and the cooks were right we were right and I was denigrated as a climate alarmist we were right we are right and so I'm just reluctant to totally blow these people off people I like and respect and trust they're just really crap on stuff it's aluminum foil ass stuff and there is no problem there just past the damn bill which is probably what I'm going to do but in the back of my mind have I gone over to the dark side of the earth? We'll find out okay we're done we're finished


To talk about some of the market action, we're going to jump over to our man Teddy Keggs at folks. Don't forget, you can check out Teddy's Outstanding Tiger Forex Report right under the Newsletters tab. You can subscribe. It's $97 for the month. He's got new issues every Monday, folks. He's got issues throughout the week when warranted. It comes with a 30-day money-back guarantee. It comes with an archive webinar that you have out there as well. And if you're into candlesticks or you want to learn more about options as well, check out a couple of the great webinars he has under the Services tab, talking about capitalizing on time with calendar stock option spreads and Japanese candlestick pattern stock and option strategies. He's written a book on candlesticks, and we're going to talk some forex though today. Teddy Keggs at. Good morning. Good morning, Tommy. So we got a little bit of a chop in this market right now. We got the dollar at $104. Markets kind of chopping around. Just under the highs we're red today, but man, it's been quite a positive market. Crude a little bit higher. That's one thing that's moving. Where do you want to kick things off this morning, Teddy? Well, if you want to talk about any action, I think we have to talk about crude because as a whole, the FX markets are going kind of flat line, and that's because rates are a flat line. I mean, if you look at the short and the long-term ends of the curve, nothing's really going anywhere, and if rates are a functional value of currency pricing and since rates are going nowhere, currencies are, except for currencies like potentially the US dollar-yen relationship because of the crude relationship. That's where I think we have a little story if you want to go on that one. You know, that's it, and I didn't even plan on it, but I remember the first time you started talking about this, even when I was doing the program with my dad and you were talking to us, right, and you were walking us through kind of the fundamental nature of whether you're a producer, a user, consumer, crude, and how that impacts and the great example that you give of kind of the dollar to the yen. If you could just walk listeners through that, Teddy, when you just make that reference, because I know not everybody kind of understands why what happens with crude is going to impact whether it's the dollar and the yen and why so. Could you walk them through that? Because that's a great one, man. I remember the first time you educated us on that. Absolutely. Okay, so it's not that it's always a relationship, but there are times when it becomes one. Right now is the case. You have a look at Japan. They do not produce oil. They're a big industrial complex, so they use a lot of oil, okay? So they have a demand for it. Oil is priced in dollars predominantly still across the globe. We are one of the biggest suppliers to Japan of oil. So that fundamental relationship becomes integrated in the pricing of dollar and yen relationship, okay? So just how interest rates are a function of any currency, depending on the central bank, oil becomes a factor when it comes to the US dollar-yen relationship. So the fact that not only are they importing, it's not just their demand, but the pricing impacts that currency relationship. So as oil breaks out to the upside, that happens to make the dollar more expensive, which makes the oil also more expensive, okay? So it's a double-edged sword there where it can accelerate a trend or it can decelerate a trend. In this case, with it doesn't matter, right now you have no fundamental dollar strength or weakness because interest rates are kind of going flatline. So that relationship is right, just going nowhere. But now you have the demand function for dollars and oil that changes everything. So for the US dollar-yen relationship, that makes it very bullish. And like especially right now, we're at that 150 critical threshold level for the Bank of Japan, that's where this becomes very, very important because if crude was say $6 lower and rallying up towards $150 because of the price in oil appreciating, it wouldn't be such a big deal. But because we're above the $150 mark right now, that's a big deal because the central bank or the Bank of Japan does not want the US dollar-yen relationship to be above there, okay? So now are we going to see any type of intervention by the Bank of Japan? I don't know, let's see. But I think that especially right now with oil where it's at, we're breaking out to the upside, like I had the $70 to $75 range was holding for a good time. Now I think we're going to raise that range to about basically $80 to $84. And if we get above $84, well then we have a really big deal because then you can see the US dollar-yen if there's no BOJ intervention trading up at 153 to 154. And I'm not talking about like two months from now, I'm talking about like let's say crude rally is $6, $7 over the next week and a half. Well then you'll see the US dollar-yen probably trading at 154 within that timeframe. And that's without any interest rate move. If yields actually go up at the same time, like if you start to see that it's a 10-year and especially the short-term start to hit the hammer the lows, meaning yields go up simultaneously, well then for sure I can't see how you would not see the US dollar-yen trading at least 153 to 154, maybe even spiking up at 155. And that's a currency trade that I think is a very viable trade on the table right now because these markets are in play because everything else is going sideways. You know, I always say that the biggest indicators for the markets are the markets themselves.  But if you want real-time, you know, indicators, look at the markets and there's a correlation between many of them. And that's the trade right there is the crude dollar-yen trade. It's in play right now. It hasn't been in play for a while, but now with oil spiking to the upside and looking to where it really could hold a trend to the upside. And I'm not saying that we're going to go up to 100 bucks, $110 anytime soon. But let's be real, higher highs and higher lows is what? It's indicative of a bullish trend, right? For sure. And listen, that was an awesome wrap up, man. For those that, you know, there's so much good information. If you didn't understand it completely, because I had to hear it a couple times myself, folks, over the years, we archive everything we do. Every interview I have with Teddy, it'll be right on the TFN YouTube page. Search it out because it is so cool, man. And I think a lot of us have gotten a real education in terms of how we've just had such big rallies and pullbacks in yields. And we've seen what that's done to the dollar and how those relationships move, but it's pretty cool the way you just explained it. And it was such a great job, man, so thank you. Thank you. In the same way that just like we're going after yield with our currencies, right, you're using dollars to buy yield in America as it rises or whatever, you're using dollars to buy crude, which we're producing. And if you're in Japan, you're not producing, man, you're consuming it. So you have to take those yen, put them into dollars and buy our crude. And that's it exacerbates things, which is so cool. Yeah, and as I started off, I saw that we were just chopping around, man. Pretty interesting that crude might be the story of $80 and how that impacts things. Anything else you're watching ahead of the inflation data tomorrow? I know, boy, you've been calling it pretty well in terms of the Fed, where they are, where they may end up in terms of not cutting at least or potentially hikes, but you're looking for any of these surprises. How do you trade something like that or do you? Absolutely. I think that you really have to watch out for the ISM number on Friday, that I think that's important. So it's kind of be, it's tough when we're in sideways markets like this, especially for the currencies, because they trend typically most of the time. I mean, writing the forex report right now is not an easy job when you're saying like, sorry, guys, there's no trade on the table right now. And as a trader, the hardest job really is not knowing when to trade, it's knowing when not to trade, and you can't force it. And as something right now, there's no opportunity, the opportunity will come. You just got to be patient. Teddy, man, that was so awesome. I appreciate you walking myself and the listeners through that explanation. I know a lot of people, I'm sure, really learned a lot. And we appreciate the interview, the update as always, and look forward to talking to you next week, man. Sounds good. Take care, Tommy. Thanks so much.


Hello everyone, thanks for coming. So we're here to talk a little bit about KBM and Kemu internals, but before we start on that a little bit about who I am and my background and why you might want to listen to me a little bit. I'm an architect at Red Hat. I mostly work with key customers and partners to develop performance and sizing guides and reference architectures in conjunction with Ceph storage clusters. Prior to Red Hat, I worked at Ink Tank where I did a lot of the same thing, working with customers, building large storage systems. And prior to that, I was in Operations and Architecture at Dreamhost, where I worked on Ceph clusters as well, bringing some of the first few into production. So why did I fall down this rabbit hole of really trying to understand the internals of Kemu and the effects that it had on performance? Well, being that I'm heavily involved in Ceph, and Ceph being the number one sender driver for a number of years for production in open stack environments, talking with a lot of our customers as they're moving their workloads onto their open stack clouds, we were beginning to be approached by them, they were asking us, we want to also use, continue to use Ceph, but we want to use it for our high performance storage too, not just to kind of our capacity tier or for just ephemeral booting of images. And if you look at the open stack user survey, 70% of the application frameworks that are being deployed in open stack environments are LAMP, LAMP based, right? So making sure that MySQL performs very well is very, very important. And so we did a bunch of work with Percona and Super Micro, and developed a reference architecture on running MySQL on top of RBD block devices. And if you're interested in more on that, there's a short link to that reference architecture paper there. But that work has led me into really trying to understand all the different components that make up the Kimu subsystem. And it starts with there's full virtualization and there's a pair of virtualization. And originally, with full virtualization, it was nice because it had the highest compatibility. You didn't have to modify the guest images. But the downside of it was that it was really slow, both in that all the hardware was being emulated, the CPUs were being emulated, and very trap heavy. And so there was a better way to approach this. And the Kimu architecture was very simple at the time. And it wasn't very high performance. And not until the advent of VRTIO and pair of virtualization drivers being placed in the virtual machines were we able to see significantly better IO in the guest operating systems. So now we can actually see relatively good block performance. And this works by there's a VRTIO bus interface between the hypervisor and then there's VRTIO drivers that are inside the kernel of the virtual machine. And then that way there's a couple different methods right now. There's VRTIO block, pair of virtualization driver, and there's the VRTIO SCSI pair of virtualization driver. And you can choose between the two of those. And the way you do that is on your glance image you can set properties on that glance image that say for this particular glance image, if you instantiate a virtual machine from it to use VRTIO SCSI, for example. So as we're doing a lot of this performance testing, the performance testing we're doing was on some really trick hardware. So we had brand new NVMe servers from Supermicro and we had really fast P3700 NVMe's from Intel. And we kept on seeing we were running into a bottleneck when we were trying to understand what it was. And if you look back at kind of an older version of Kimu, the architecture looked something like this where while you had the virtualization of the and multiple threads for the CPU threads, for actually executing the guest code natively with the hardware, your I.O. was still being processed by an event loop. And this event loop could block, depending on the type of I.O. that was issued. To improve on things, this event loop was broken up into multiple components so that it had a separate AIO context so that certain I.Os would have to kind of lock this event loop waiting for I.Os to complete. And that worked pretty well, that improved things. And until a better method came along, and that's the data plane, right? And so with the data plane, you're able to have multiple threads of AIO context. And if you're doing a lot of asynchronous I.O., that works pretty well. And so we're testing with all these different methods, with Verdeo block, with Verdeo SCSI, playing with the different knobs that are available in Kimu. Just with Kimu itself, right? With Kimu itself and RBD. To see what the best configurations were, even outside of OpenStack, and then we would come back to OpenStack, look at how Cinder, or look at how Nova, look at how Glance interacted in order to then program the data plane, program Kimu to be as performant as we could make it. So bridging with OpenStack. Cinder support doesn't quite yet have support for the Verdeo block data plane. There's a blueprint up. So that's something that'll be a work in progress. I was unable to find a way to use the Qs in vector support that's available with Verdeo SCSI that allows you to have multiple I.O. threads through Cinder. And while it could be interesting to add these things to Cinder, one problem that might be encountered is that if you have these separate I.O. threads that are going to be using a lot of CPU to process the I.O., that's going to manifest as CPU steel if it's not accounted for some way. So you would need to make, if we were to add support to Cinder to allow assigning multiple I.O. threads to different volumes using Verdeo SCSI, then we might need to have Cinder and Nova work together to account for those and maybe assign less CPUs if you're using effectively an execution thread for I.O., right? So, you know, only primitively I suppose you could reserve some cores on the hypervisor, but really you would probably need a more holistic approach that does some sort of accounting. Turns out that there's also multiple different I.O. modes in Kimu and actually at the last summit it was hard-coded. You were only able to use the I.O. method of threads. Turns out that the I.O. method of threads is fairly safe. And how that works is Kimu has its own user space implementation of kind of a thread pool and it just does p-read, 64, p-write, 64 calls. And that was just hard-coded in Nova. Now it's the default. I believe there was, I think it was in Mataka that it changed from just being hard-coded to being a default and you could actually override it and use AIO native. The difference with AIO native is that instead of using a user-land thread pool, it uses kernel AIO and I.O. submit a POSIX system call and that can help perform it. It can bring down CPU a little bit in some cases, but it's complicated. We ran tests with both and for our particular workload we didn't see that one performed particularly better than the other. Although we did see that with some means of benchmarking that I.O. submit could block and that could cause the VM to have jitter. So if you were running a workload in the VM and you were pushing it really, really hard, this would manifest as you're running maybe DSTAT and then you would see missed ticks and then it would come back and that was a little bit worrisome for us, especially because we're trying to use those timers for keeping track of how many I.O.s were completed in a given period. There's also a lot of different caching methods, caching modes that are available in KVM and KEMU and understanding how these map, whether you're using just local LVM or just local drives or if you're using RBD can be a little bit confusing. So I've kind of put together this table or I've expanded this table from to include the RBD components but the way that this works is that you can set a cache mode in Nova, right? So in nova.conf you set your just cache mode to writeback and that forces the cache behavior of all your VMs and center volume types to use that particular cache, which is kind of strange, at least from my opinion, from user experience point of view because I don't want to use caching, right? From the benchmarks that I've done, not having caching turned on when you're using an all SSD volume type is better performing than having the cache turned on at all. But for a magnetic pool, it's going to make a lot of sense to have writeback. The writeback is safe. If you have an application that needs to guarantee that data has been persisted to a disk, it should be issuing an F-sync and they should be using a file system that supports barriers and if that's the case, then that'll get passed through and the RBD cache or the page cache on the host system will be flushed and it'll be on persistent media, right? So it's still beneficial in that it's reducing the amount of IOs that are going to the disk in terms of writes, which leaves those seeks available for faster reads. In an ideal world, I think that you should be able to set different cache policies based on your different volume types. So when you're an administrator and you're setting up your different volume types in Nova, I think it would be really sweet if I could say for my gold level tier that's based on all flash media, I want to use direct sync. But on my SSD-based tier storage type, I want to use writeback and Kimu lets you do that. It's just not exposed through the control plane in order to push that down, push that logic down. I get asked about QoS a lot. And Ceph itself doesn't have a native QoS capability. It's something that's being worked on, but it's hard when you're doing distributed storage. But I feel you can achieve adequate QoS through doing appropriate capacity planning. If you know the throughput of a particular storage node, you know it's capacity, and if you collapse those two constraints onto each other, similar to like they have in the public cloud, then your capacity planning is you either add more because your tenants are provisioning it for storage, spatial capacity, or for IOPS. You don't care which. You just need to know that you need more. So by collapsing those two constraints onto one, you don't have to independently track your IOPS and your capacity, because as an operator, it's just making things more complicated than it needs to be. So in the testing that I've done and comparing hard drive based pool performance and SSD based pool performance, if anyone is familiar with the work of Neil Gunther, he wrote a great book called Gorilla Capacity Planning. And he has a model that he's provided, and Gene talks about in great detail in that book, called the universal scalability model. And there's actually tools that you can use by taking data with different levels of throughput and different client numbers and use R to project confidence bands of how much throughput you should expect for certain given numbers of client levels. And it turns out that part of the universal scalability law is a thing called the coherency delay. And the universal scalability law was originally applied to databases and pneumotype systems. But it turns out that the coherency delay matches really well to seek latency. And so client scaling, when you have SSDs, it kind of plateaus because the seek latency is fixed. It's not like spinning media where once you reach a certain number of threads, you have this severe retrograde performance. And so what that leads to is you can do the sort of provisioning that we see in the public clouds. And being able to have that in Cinder, being able to do capacity derived limits for your SSD pools where you give a ratio, so you say, for my volume type, I want to give 30 IOPS per gigabyte of storage that the tenant provisions. Or I want to do 3 IOPS. And if you wanted to do something closer to a general purpose SSD tier, I think that makes a lot of sense for the all flash, all flash-based. Now for the spinning media, I think the static limits are great. Having static limits and volume quotas, I think, is perfectly sufficient for spinning clusters. But for SSD, I really think that a ratio approach works really well. And then, like I said, by collapsing the two constraints together, you don't care whether your tenants are using up capacity or IOPS. You just know that you need more. And if you've done testing beforehand on how much throughput you get from a given system, you know whether or not you can hit those targets. And the architecture paper that I mentioned earlier, we actually did this. So you can create deterministic performance. So our benchmark environment, I mentioned earlier, it was super micro NVMe servers. We had some pretty recent 2650 Haswell CPUs, two of them. And then two Intel P3700 800 gig NVMe's dual 10 gig networking. And we were running RHEL 7.2 and the previous release, not the most recent release, because this work was done earlier this year of Red Hat Storage, which is based on the Hammer release. So this is before Juul. We did do some tuning, though. We actually ran four OSDs per NVMe device. Those NVMe devices from Intel are pretty dang fast. And you really need to fill the queue depth. So we ran actually four OSDs per NVMe device. We tuned TC Malik to use larger thread cache. That's the default now, at least in the downstream versions of Ceph. Other things that were important were TCP, no delay, basically disable the congestion control so that you can have, you don't have to worry about that with your, it keeps the latency down. And then also, really critical when you're running any sort of NVMe is making sure that your kernel supports block MQ, which is so that you can have multiple queues into the NVMe device. So at first, when we were doing the testing in these KVMM instances, we were using FIO with O-Direct. We were F-syncing after each write, using the lib AO engine, and using EXD4 inside the guest. And this is when we encountered the issues with Jitter and the Ms.Tix. And we weren't quite sure that the FIO accounting was, we weren't super confident in the IOPS figures because we didn't know if the stalls that were being caused by the Jitter were affecting the calculations on how many IOPS were being done per second. So in retrospect, and after learning a little bit more, it might be due to using O-Direct and using AO-Native, which happens to not be the default in OpenStack. So maybe if we were running it in an OpenStack environment and not playing around with things, we might not have seen this, and potentially IO submit blocking. But regardless, that was kind of the lower level testing. But the eventual goal was to do MySQL benchmarking. And so we brushed off kind of sysbench. And we set up a config where we had a 50 gig buffer pool, 8 gig, I know DeBeast, made sure it was a fully acid-compliant configuration, I know DeBeat, O-Direct, flushing after every transaction. The guest file system was XFS. We used no A time, no do or A time. We accidentally used no barrier, which is a terrible idea. So don't do that. But as far as the effect on the results, that probably would only affect the results where our writeback cache wasn't enabled, and we actually had better performance when it wasn't enabled. So again, do not use no barrier ever. That was just kind of a mistake. So we reloaded the data before each test. And then we did two different tests. We did 100% read workload, just doing selects.  And then we did a blended workload, where we were doing 70, 30 reads and writes. And we made sure that we were using a uniform distribution, so this wasn't a parrot distribution, because we wanted to have a good amount of the IO hitting the disk. We didn't want everything coming out of the buffer pool, because that's not really testing the storage. Ran it for 20 minutes. So what were the results? Well, as you can see, the original default, fully virtualized, is very slow. That's all the way on the left here. The defaults, just kind of out of the box. You can see that on, we were doing pretty good on reads, just shy of 200,000 IOPS for that particular instance. Reads, I mean, both the writes and the reads and writes were over 5,000. When we set the cache to none, the final two charts here, or with the threads in native, you can see that there wasn't much of a performance difference, especially if we had factored error bars into these graphs. There would probably statistically insignificant difference. As far as cache modes go, you can see that the highest performance was with the cache turned off. So with the cache turned off on an all SSD-based cluster, you saw a lot better performance. And so if you were already running a cluster and you had gone into your nova.conf and set your caching method to write back because your initial deployment was a bunch of spinning media, and then you added a high performance tier because of that inability to have that flexibility in terms of a different cache configuration for a different volume type, you would be sacrificing a decent amount of performance. We didn't see very much of an increase by using dedicated dispatch threads. That's where you can actually have a completely separate IO thread. And I think that if I go back and then I run back through these, you're probably seeing a common theme here in that writes are really stellar. I mean, reads are really stellar. But writes are low. But more interesting is the fact that our mixed read workloads is almost always the same as our write workload. So this is where we're beginning to suspect the Kimu log. So when we know that we have synchronous writes that are blocking the IO thread, the subsequent reads are just waiting for that to come back. And so I mean, any realistic workload is going to be mixed.  So this was surprising to us. We didn't think that this was something that we would see. We even tried increasing the cues by using Verdeo-Skezzi. And we did see much better performance on writes. In fact, we were able to get over 35,000 IOPS from an RBD inside of a guest, but that's a 100% read workload. So the having additional IO threads and being able to asynchronously access or do reads was scaling, was increasing. But still, as soon as we had a mixed workload where there was synchronous writes going on, that was the limiting factor. So we went back and we ran the same tests on bare metal, which is the kernel RBD driver. And then we ran kernel RBD, put a file system on it, and passed it through to a container. And you'll see something interesting, in that the mixed read write workload all of a sudden is not getting blocked. And the only difference here is that we're not going through the Kimu IO subsystem. So kind of in conclusions, our performance is pretty good, right? Like there's been a lot of work that's been put into Kimu to make it perform really well. We're able to, at least on 100% read workload, hit the caps that are similar to what the public clouds do. So the public clouds cap out their instances at 20,000 or 25,000 IOPS that you can't do more than that. And in some cases, we're demonstrating over 30,000, close to 40,000 IOPS, at least on read. I mean, that's still pretty impressive for a single instance. I think the UX is kind of weird for a pair of virtualization devices that you have to set properties on your glance images. I think that if you look at how Google Compute Engine works, they only do VerdiOS fuzzy. I think that it would be really neat if I could say, hey, I want to always use VerdiOS fuzzy for all the instances that are booted as a cloud. That'll just be an operator decision. I don't think many tenants really understand the difference between VerdiOS block, VerdiOS fuzzy, and which one is more appropriate for them, and that they're going to choose the set and maintain separate glance images. Like, am I going to have the rel VerdiOS block and the rel VerdiOS fuzzy, separate glance images with different properties so that I can know I'm probably just going to use one. So I thought that was a little awkward. The OpenStack cache configuration is not very flexible right now, and the impact on multiple volume types didn't seem to be considered, and I think that's just the organic way that it's grown, right? It used to be perfectly sensible when it was Nova and just Nova volume, and we've outgrown that, and we probably just need to adjust to that. The big Kimu lock limits mixed workloads where you have IO direct and synchronous rights. This is a particular evident in I know DB type workloads like we're showing here with Sysbench. The AIO and data plane multi-Q does probably help other workloads a lot, particularly if they're mostly asynchronous, but in the case of databases, I think we're being held back a little bit. So what would I test next? What do I want to test next? I want to try using Verdeo SCSI, and there's a way inside when you're booting the instance to tell it to use block MQ for the Verdeo SCSI device, and then if you have multiple IO Qs or IO threads and vectors for your Verdeo SCSI device, we might be able to see some better parallelization there. One thing that I recently found out about is this thing called Vhost SCSI, and what it does is instead of using the IO subsystem, it actually uses, it goes back to LIO, in the kernel, and uses the kernel for processing the IO. And it turns out that there's some work being done with KRBD and integrating it with LIO, so there may be a way to potentially leverage that work and Vhost SCSI to bypass kind of the Kimu backend with the userland libRBD and just use the kernel implementation of KRBD and not have to do it inside the guest, right? Because we don't want to expose the storage network to the tenant networks, but to have that back Kimu and maybe see if we're able to get the performance levels that we saw here on the metal and the containers in Kimu. And then if this does work, what are the impacts on live migration? Because the kernel CPU, because the kernel is gonna be consuming CPU with all these kernel threads, does that kind of show up? Like if there's a lot of guest IO processing, does that kind of manifest as CPU steel for the guests? And then finally, this is something that is pretty cool. It was done this summer. There's the MyRox, which is a version of MySQL written by Facebook that uses RocksDB instead of I KnowDB as the backend. An intern at Red Hat over the summer made it so that RocksDB can store its right head log and its pages directly into Rados. So you could potentially run a MySQL database without a block device or a file system at all. It would just be talking to a mutable object store. And so you could completely eliminate the double right buffer because Rados is already ensuring atomic updates because it has to be able to support rollbacks anyways. So that could be interesting. So that's all I have for today. Thank you everybody for coming. And if anyone has any questions. Yeah, the question was, so what is the answer when you wanna have different volume types with different cache modes? Is there a way to do it with different host groups or regions? That's a good question, I'm not sure. Yes, sir. When it comes to Virtio Scasia or Virtio BLK, when it comes to guest driver support or any settings which needs to be done in guest OS or is that change is completely invisible for the guest? If it's typical guest OS distributions, not only Red Hat, but others. That's a good question. Yeah, I'm not sure. I think most guests support Virtio Scasia. VDA, yeah. VDA to SDA. SDA, but SDA is supposed to be the normal Scasia, not virtual Scasia. Okay, but maybe you have tested, is it true that the device name at least changes? I thought that it will remain the same because SDA is bare metal Scasia, not virtual. Okay, thanks. Great, thank you.