All right everybody, welcome again to another OpenShift Commons briefing today. It's a wonderful Monday in the world and we are doing AMAs with upstream projects and today we're going to talk with two folks who are the people behind Coverno, a new CNCF sandbox project I believe, Ritesh Patel and Jim Baguadia and I'm going to let them introduce themselves. We're going to have a nice presentation about what Coverno is doing and why and where it fits into the whole CNCF landscape and then we'll have live Q&A at the end. So Ritesh and Jim, take it away. Hey folks, I'm Ritesh Patel, VB Products and co-founder at Nirmata. Hi everyone, this is Jim Baguadia, co-founder and CEO at Nirmata. All right, so let's dive into it. So today we'll talk about, you know, Coverno. Coverno is an open source, Kubernetes native policy management solution that's now a CNCF sandbox project. So here's the agenda. I will, you know, dive deeper into why Coverno, what does it solve? Why did we feel there was a need for something like Coverno and then jump into the technical details of how it works. Now we do have some use cases that we can go over to give you folks of, you know, an idea of the kind of problems you can solve using Coverno. I will discuss some of the other tooling around Coverno, which more specifically that Nirmata offers to help ease the, you know, management of Coverno policies and reporting. And then jump into a demo followed by a quick roadmap and Q and A. So let's jump right into it. Coverno actually, you know, you may kind of wonder what does Coverno mean? And Coverno actually means govern. It's a play in Greek, basically it's govern and, you know, obviously Kubernetes, the origin of Kubernetes, the word is from Greek. So we thought it would be appropriate to kind of use Coverno as the name of this project because Coverno is essentially a policy engine for Kubernetes and policy engine is used typically to govern or for compliance of various aspects of Kubernetes. So that's why the name Coverno, Coverno is a Kubernetes native policy engine, which means that it understands Kubernetes resources, Kubernetes patterns, that it actually is aware of Kubernetes constructs. And it's been, it's designed in a way that it's familiar to Kubernetes users and essentially there is no learning curve. So let's take a step back and understand why do we need policies in Kubernetes? What are policies and, you know, how do they help? So if you're familiar with Kubernetes, Kubernetes, you know, you can configure lots of different resources, whether it's, you know, your infrastructure for Kubernetes, whether your applications or workloads running in Kubernetes, and all of the configuration is essentially done using, you know, YAML, right? And it gets fairly complex, some configuration files could be, you know, 200 lines. And the challenge there is now how do you ensure that the configuration can be managed and is secure? There are also, you know, external tools that, for example, helm and customize that tweak existing configuration to adapt it to environments. So that's how your configuration could change. The end configuration that ends up in the cluster may be different from the source, depending on where your resources are deployed. So, you know, in Kubernetes, there are ways to configure admission controllers. And that's where as a best practice, a lot of the validation for configuration, as well as any modification or mutation of configuration can happen. And that's why, that's kind of the best place where policies, if any, can be or should be applied. So what, what, what policies do is essentially let you govern or, or ensure that the configuration that's being, you know, applied to Kubernetes is secure. It follows your best practices, you know, follows the standards. And, and Kibirno is an engine which does that without the user or the developer or the cluster operator having to build something from scratch or build something custom for their environments. So that kind of brings us to why Kibirno and, you know, touched on this briefly in an earlier slide. But with Kibirno, our goal when we started out was to enable policies for Kubernetes and make it really, really simple, make it really easy for users to understand and write these policies. So Kibirno policies are declarative and very easy to manage. And when these policies are applied, they generate policy results. Basically, these are, you know, violations or any kind of, any, you know, any policies that are violated kind of are reported in the results. So that it's also is Kubernetes native. These policies are can be of different types, you know, you could have a validating policy, which can either be in audit mode or an enforce mode, or, or they could actually mutate an existing configuration by, you know, changing parts of parts of the configuration, or they could, the policy could generate new configuration, if it's in cases where this configuration is required, or it's missing. And that gives you a lot of flexibility to kind of ensure that the configuration of your Kubernetes applications and resources are configured correctly. The being Kubernetes native, Kibirno supports all of the Kubernetes resource types and can also support custom resources, which is a huge benefit so that, you know, now you can use Kibirno across the board without having to worry about which resources are supported and, and which are not. And, you know, being, again, being Kubernetes native, Kibirno leverages and understands, you know, some of the common patterns used in Kibirno, like labels, selectors, labels and selectors, annotations, owner references, and, and, and things like that, right? So that really makes it very easy for somebody to, to, you know, start with Kibirno and start using it with Kubernetes. Now, just a little bit of background on the project itself. So Kibirno today is an open, is open source and a CNCF sandbox project. It, it got the, it, it was added to sandbox projects in November last year. And since then we've been tracking the downloads and it's almost approaching 2 million downloads in, you know, about three and a half months, right? Which is, which is huge. It went to 1 million downloads about three or four weeks ago and now we're almost there at 2 million downloads. So seeing a lot of, you know, interest in the community, you know, a lot of folks are looking at it and like the simplicity ease of use of Kibirno. And, and then there are other projects in, in, in the CNCF and in the community like open policy agent, which, which, which do policy management but Kibirno kind of because of its simplicity has been gaining a lot of, a lot of interest and, and a lot of users. So definitely do check it out if, if, you know, you're interested. So question, when we talk about Kibirno, this question that comes up is, you know, how is Kibirno different from open policy agent? And here are some points of differentiation. You know, open policy agent was, was actually predated. Kubernetes was built as a general purpose policy engine and then adopted or adapted to Kubernetes. Whereas Kibirno itself was designed for Kubernetes. It's Kubernetes native. The policies are Kubernetes native resources in Kibirno. Whereas an open policy agent, the actual policies are defined in a language called Rego, which could, which is something that a user would have to learn to write policies. There is a project called Gatekeeper, which, which works or integrates with open policy agent that simplifies or, or that tries to adapt, adapt Kubernetes resources and, and to, to define policies. But ultimately, the underlying enforcement and all of the, all of the policy processing, everything happens based on, based on policies written in Rego. Also a few other things about Kibirno, which are not necessarily available or possible with OPA, OPA Gatekeeper. So Kibirno is, is, you know, secured by default. You don't need to call out any external systems for any processing. And, you know, because it's an admission control path, this is very important. And also a few other scenarios that Kibirno enables are around, you know, kind of the, if there's in that paradigm where if, if you, if a particular configuration, you know, if you check for a particular configuration and if it exists, you can actually write policies to do certain things. These are very easy to do in Kibirno. And being Kubernetes native and Kibirno policies are also Kubernetes native. It lends itself very well to tools like, you know, customized sound and can, you can use GitHub to actually apply policies across your clusters. So really, just in a nutshell, there are several benefits that you get with Kibirno in terms of ease of use as well as the other capabilities. Let's dig a little bit deeper in terms of, you know, at a high level from the policy perspective for Kibirno or OPA. And we'll go a little bit more details into how Kibirno policies are defined. But here, this is just to show kind of the difference between Kibirno and OPA. On the left hand side, you see the Kibirno policy, which is, you know, if you are familiar with Kubernetes and have used Kubernetes resources like deployment spots, this will be very familiar to you, which vary, you know, straightforward in terms of creating these policies. There are certain keywords like match and validate and we'll talk about those a little bit later. But overall, in this policy, just validates and checks if, you know, the file system is read only. On the right hand side, you see the same policy written in Rego and this would be something that you would use in OPA. So this is just to quickly show a slight difference between Kibirno and OPA. So next, we'll talk about how Kibirno works. I mentioned earlier, Kibirno is a policy engine that runs inside the cluster and it registers, you know, as an admission controller. And then every request that comes in through the API server is a process through Kibirno and policies that are configured in Kibirno are applied to do these requests. So it Kibirno gets an admission review request, it applies these policies, the policies depending on the configuration may generate certain events or a policy violation. And then the response is sent back. If the policy is configured as audit only, then it's really the, it's just, you know, kind of like a report only mode and the request itself is not blocked. But if the policy is configured as, you know, configured to validate and block, then the request can also be blocked if the policy fails. So this is really at a high level, how Kibirno works in Kubernetes. Here is a quick view of the policy structure. So when you're defining a policy in Kibirno, a policy contains one or more rules and each rule contains a match or exclude criteria, which could be mapped to Kubernetes resource kinds, names, you could use label selectors, namespace selectors, as well as user roles, groups and so on. And then you have the various actions that Kibirno can take, you know, that's either validate, mutate or generate. So these are the high level blocks and we'll talk about each of these blocks in the next few slides. So just to map the structure we saw earlier, here we have the same kind of, the Kibirno policy showing the same structure where you have on the top, line number eight, we have the match block where in this case it's matching all resources of kind pod. And then on line number 12, we have validate block which actually does the, you know, does the validation for every pod. And in this case, it checks if the security context run as non-root is set to true. So this kind of gives a quick idea of the policy structure. So validate policy is essentially it kind of matches the fields that are defined in the validate block and determines if your configuration, you know, matches the, or matches the fields and, you know, it is, it's validated for, it validates the configuration by matching all of the fields. So in this case, you could have patterns, you can match, you know, for existence of fields, you can match for, you know, with operators, you can have, you can check if values, you know, are greater than equal to or not equal to. So a lot of the, there's a lot of flexibility in terms of doing various types of matching in these policies in the Vaport for Validate. Next on the mutate policy, so here you can, when you want to mutate an existing config, you can specify kind of a patch, a JSON patch, a strategic merge patch to update the incoming resource. So for example, here if you want to, let's say, add a port or, or, you know, insert like a secret name, you can do that by specifying the patch in the policy, and you can even do that conditionally. So you can even, you can only add a configuration if it's not there, or if, if, for example, you can even have if then condition where for certain port names, you can use forces for certain, only if port names start with certain string, you can, you know, add the, add the value and so on. So this gives you, again, a lot of flexibility to, to manipulate incoming, incoming configuration. And it's actually very important that I use cases where, you know, if you want to add certain labels on every resource or on specific resources, you can now ensure that that's done by creating a mutate policy. A lot of times these labels could be used by other tools for security, for networking, for monitoring and so on. So, so, you know, given through mutate policies allows you to address some of those use cases. And then the, the final type of policy is a generate policy, where you can actually generate certain resources. So, so on the, the example here on the right hand side kind of shows how you can generate a network policy to deny all traffic. This could be your default network policy for your namespace. And it can, you know, it would be created, maybe anytime a new namespace is created. So there is a usually a trigger, which can trigger the generate of, of these, these resources. In this case, we are generating a network policy. And it could also, you could use to clone existing resources.  And these policies are configured. Once these policies are configured and, and, and as they are processed based on, based on incoming requests, there are violations that, that can be generated. So here is an example in Kubernetes to see the policy result. So on the top, you see an example of showing the, of the policy results for a particular namespace. In this case, default namespace, it shows that, you know, nine policies were passed and one failed. And you can get, get more granular details, you know, like, like the example, like the second screenshot here to, to see which policies failed and what exactly, why exactly the policy failed, right? So you can get more details in terms of the violation so that the violation can be fixed. So that's kind of, you have that information. Again, it's all inside your cluster, you don't need an external tool to, to retrieve that information. So this was a quick overview of some of the features that there's lots of advanced features that are being built. And, you know, this is an ongoing kind of, as, as community comes up in more use cases, we're building more and more features. So we talked a little bit about, you know, some of the anchors, operators, like, you know, compare, comparators and, and if then else and so on. You can also have variables. You can actually point to certain, certain parts of your config using JMS path, JMS path. And you can also look up data, external data, you know, from, from config map or through API calls. And this lets, this allows you to kind of check or figure, you know, violations by checking data that may not be part of the incoming requests or the part of, part of the resource configuration. You can also set up deny rules. One thing that's interesting is auto generation of pod controller rules. So if you have policies for pods, they automatically get applied to pod controllers, like deployments, stateful sets, jobs and so on. And then finally, QVerno can also be run in offline mode. So you don't, there's a command line for QVerno, which can be used offline so that you can actually check, check for violations in your configuration if it's in gate or, you know, before your config, check for your violations before your configuration is applied to, to your cluster. So that's definitely useful. You know, you want to, don't want to wait till, till the configuration is in your cluster to check, you can check upfront, address those issues before applying your resource changes to your cluster. So, you know, we talked about a few use cases, but there's more use cases emerging as, as, as QVerno gets more widely used. We talked briefly about security validation and enforcement. There's also QVerno enables you to find the RBAC. So based on, you know, base, for example, when namespaces are created, you cannot create some of the roles, role-bindings, et cetera, depending on the namespace name or labels or any other criteria. QVerno is, enables multi-tenancy in, in Kubernetes again, through some of the automation by generating RBAC rules, by generating network policies, can generate quotas. And, you know, we're also working with various states to kind of enable this, enable more secure multi-tenancy across within, within Kubernetes, you know, for, for your applications. Auto labeling is a use case we talked about earlier. Also, for a lot of, we are seeing patterns like, you know, sidecars being used in, for various, various tools like, you know, for, for, for example, for service mesh and Istio sidecars, it's very common to use sidecars.  So QVerno also can, can, because it can mutate existing configuration, you can create sidecars using QVerno. So you don't have to write a new operator to do that. And then finally, you can have very conditional rules like if this and that for Kubernetes and that opens up a whole set of use cases, you know, and for your, in your, in your Kubernetes deployment. So it's very flexible and, you know, you're happy to kind of, if you have other use cases that we have not covered or not seen, it would be great to hear about those and see how we can help with QVerno. One area where, you know, we really see a lot of interest today is around, you know, pod security. In the past, pod security policies have been defined using PSPs, but PSPs have been marked for deprecation by the, by the community. And there are other options and alternatives being discussed and we are part of those discussions. But today, if you want to, if you are interested in, in the pod security policies, QVerno provides a very easy way of achieving that. We already have a set of policies, you know, and are on the QVerno website of it, which kind of address all of the requirements of, you know, for, for pod security and they are based on, on levels, you know, like we, like the PSPs and, and some of the discussions that are happening in the community. So it would be, you could, should, if you're interested in alternatives to PSPs for, while, you know, communities working on, you know, the next iteration of PSP, it would be great to check out QVerno and let us know if that addresses the security needs. A lot of folks in the community that we know are already using QVerno for pod security.  So before we jump into the demo, the one thing I want to quickly touch on is, you know, as, as QVerno gets used in Kubernetes and it's, as it's more wide, widespread, you know, we're seeing some requirements around managing policies at scale across clusters, across multiple clusters, ensuring, you know, consistent deployments, and, and so on. And so we've created, you know, the cloud-based solution at Nirmata to help manage policies at scale. So through Nirmata, you can actually deploy policies across clusters, you know, using GitOps, but that's a concept of policy groups. You can define different policy groups, like, say, for pod security or for multi-tenancy. And then these groups, these policy groups can be applied either automatically to new clusters or they can apply, be applied to specific clusters. So you can manage it very easily instead of having to deal with each cluster individually, right? And then the ongoing challenge of keeping your policies up to date and automating that deployment can also happen, you know, very easily with with our integration with QVerno, with the integration of Nirmata with QVerno. And then QVerno generates a lot of violations, right? So depending on the size of your clusters and the number of pods and the number of applications running, could have tons of, you know, policy violations, you know, failures and so on. And policy reports are available in the cluster, but through our integration with QVerno through, you can actually get a very granular report of the violations and not just get the report, but then also understand what the problem is and then, you know, either get help on remediating that problem or filing, you know, tickets to kind of get that problem remediated, you know, from the application on and so on. So through the integration, we complete the full lifecycle and of managing QVerno policies and getting visibility into QVerno violations at scale, right? If you have multiple clusters that you're managing, this becomes very, very important. Next, we'll jump into the demo. So I'll stop sharing and let Jim take over from here. There we go. Yeah, so what I'm going to do is I'm going to just demonstrate, you know, how you would, as a new user, get started with QVerno, what you can do with it. We'll look at a few different examples. We'll look at, you know, like Ritesh mentioned, pod security policies get a lot of attention. So we'll start with those, see what QVerno can do in that area, and then also look at some more advanced policies. So I'll just go, you know, just again, going to the QVerno site, if I go to documentation and we'll click on installation.  There's a Helm chart, but for this, what I want to do is just install using a command line so we can kind of look, take a look behind the scenes and see what's going on, right? So I'm just going to copy this command, which is a single line to pull down all the YAMLs from the QVerno Git repo and install into my local cluster, right? So if I do kubectl, get namespace, you'll see I'm just using Docker for Windows here and on this, I'm going to run, you know, this command line that we copied and have it install QVerno, right? So we see there's a bunch of CRDs, that's all of these that came in about like six or seven different CRDs, including policy reports. One interesting thing I want to mention about policy reports is these are not unique to QVerno. In fact, the policy reports are being developed, you know, in collaboration with the work group policy, the policy working group in the Kubernetes community. And the intent is to make these same CRs available to any other tool that wants to report policy results in a Kubernetes cluster, right? So this provides a standard way of reporting policy violations independent of the engine or the tool that's producing it. And there's some work being done to take, you know, things like kubbench and also adapt it to these policy reports. Falco is being discussed as another candidate, et cetera. So once these CRDs are installed, we also see that QVerno created a service account. There's also a few other things done. And what's also interesting is there's some web books created, right? So now if we look at our namespaces again, we should see a QVerno namespace. If I do, you know, minus hand QVerno, and if we let's say do pods, we should see that we have a single pod running, which is what we expect at this point, right? So now let's go ahead and, you know, I don't have any policies. And if I do, kubectl get C-Pol, which is short for cluster policy, it will say I don't have any resources. Because the CRD is installed, I can actually now use something like kubectl explain. And I can say policy, so QVerno both has a cluster policy, which is C-Pol and just policy, which is a namespace policy. And if I say policy spec, it will tell me, you know, because it's integrated again with Kubernetes, I can see right here what I can put into a policy. So I see I have a background option, which is a Boolean. I have a set of rules in the policy, like Ritesh was explaining. And then I have a validation failure action, which can be set to either enforce or audit, right? And if you go further, so on, it's very straightforward. And of course, if you have your, you know, Visual Studio Code or other things integrated with Kubernetes, you get all this help and, you know, schema, et cetera, now, also in your IDE, right? So makes it easy to write policies, check the syntax, things like that. But here I can see that this is the description of rules. But so far, I don't have any policies. I just have QVerno installed. So let's go ahead and, you know, get some policies, right? So I'm going to go back to the QVerno site, and we'll click on policies. And I see I have pod security, best practices, and other. So let's go to pod security. And like Ritesh mentioned, here, the way these policies are organized is based on the pod security standards. So if you're not familiar with those, these are great to review and look at. Because what the Kubernetes pod security standards help you do is they organize the pod security controls into three categories. So there's privilege pods, which is basically unrestricted. There's baseline, which is typically used as a default. So this is minimally restrictive. And then there's restricted, which is the highest level of security itself, right? And then it lists every control within the pod security security context structures, and allowed settings, et cetera, for those. Now the, you know, the cap which is being developed, which is most likely going to be called pod isolation policies, is also going to be based on pod security levels. So again, it's important and pod security standards. So it's important to kind of keep these in mind. And then you'll be able to apply these, one of these levels to namespaces in this upcoming, you know, cap, which will be pod isolation policies. So anyways, going back to the Kibana policies, as you see, they are organized to default. So this is actually a baseline. It got renamed from default. So this is the baseline setup policies. And this is restricted. If I go into default, we'll see, you know, each policy that's, you know, part of this, if I click on any one of this, like for example, let's say the required default proc mount. So it is checking both for init containers and containers, that if a proc mount is specified, it has to be default. It cannot be changed from that setting, right? But there's other, of course, several other policies. If we go back to, you know, the restricted and look at some of the more stronger or the tighter control policies, you'll see things like require, you know, that run as non-root, which is pretty important for non-privileged pods to make sure both init containers as well as containers are always run as a non-root user, right? So let's go back, you know, to this main pod security page. And what we have over here, it's showing me a one-line command. So it uses customize, it pulls down policies from this git repo. And then it's just using kubectl to apply them, right? I'm going to take that and let's see what happens if we apply to our cluster. So I'm going to, you know, this takes a few seconds to pull down all of the yaml's. And once that's down, what we should see is it will apply into our cluster itself, right? So we see about like about 10 or so policies got applied. And if we go back and look at the cluster policies, we should see that these are now enforced. Now what's interesting is if I go, let's actually, you know, look at this, if I go to one of these policies, if you notice when I was browsing through the policies, most of them are written at the pod level, right? But as Ritesh mentioned, one of the things Kivarno does by default is because it understands the relationship between pod and pod controllers, it will, let's take a look at that, you know, root policy. Kivarno will automatically generate policies for different pod controllers. And of course, you can control the settings. So if I say kubectl get, you know, let's say cpaul, I'm going to do this and do minus o yaml.  So you're, if you notice what happened is when I applied this policy, it was just written, the rule was written to match a pod. But in addition to a pod, what happened is Kivarno automatically generated policies for, you know, most of the standard pod controllers. And again, this can be, you know, tuned, you can specify different annotations to control, which you want to, if you want to restrict, for example, only to deployments and not to demon sets or something like that, right? But here, what I wanted to make sure is now we do have all of these generated. It's this one's for a cron job. This other one is for all of these, you know, the more standard pod controllers as we see. So pretty neat that I have now all of this installed. So let me try and run a simple workload. You know, I think I have an nginx pod. So I'm just going to say create minus f. And let me check if this is in my temporary. Yeah. So, okay. So immediately, Kivarno is saying, and because these policies were set there in force, it's saying that running as root is not allowed. It's telling me what exactly got violated. And it's saying that run as root must be true, must be set to true. Run as non-root must be set to true, both for init containers as well as containers, right? So that was since that was violated in my YAML, it's basically blocked this particular configuration itself. So now let's try something a little bit more intrusive, right? So I'm going to actually use this site, which is pretty handy. If you haven't seen it, it's the site is Bishop Fox, and it's called bad pods. And as the name suggests, these are pods which are misbehaving, right? So pods which are misconfigured to allow all sorts of, I guess, to basically be open where if you want to allow the host bit or access to host network or other namespaces, all of that is fairly open, right? So here I'm going to go into, let's check and see which one we want to do. Let's go into everything allowed, which is probably the worst you can do, right? And we'll take a quick look at the YAML. We'll actually go to the deployment just to see how that works. And we'll look at it. So this is basically allowing different, it's saying privilege true. It's mounting host, which is also a bad thing to do. It's all, and then allowing all of our name spaces to true, right? So let's grab that. And what we want to do is we'll go to the raw YAML and we'll run that in our cluster, right? So I'm going to now do kubectl create minus F and we'll give it that whole YAML and see what happens. So now we see a lot more warnings or a lot more errors coming back from our admission controller, which is what we expect. So it's saying that, hey, don't allow host namespaces, don't use host path. It's a don't privilege. So all of this got blocked, which is, and of course, the other error we saw before, which is run as non-root. So this is how straightforward and simple it can be to set these policies. These policies are very flexible. You can tune them based on ELECTROTATION's explaining. A lot of different selectors. And I have some examples in here. So this is actually a different policy. It mutates or this is generating a network policy. But if you look at another policy like which also can mutate things. So let's, yeah, this is an example of a mutate policy. It matches also pods, but you can write your selectors based on several constructs, including namespace labels, et cetera, right? So very flexible, very simple, you know, to apply some of these. One other thing I want to quickly show is how, you know, Ritesh also mentioned some features like being able to select and get external data, right? So very often you want to write a policy, but you want to then, you know, leverage data from things like config maps, which are very natural incriminates. In fact, you know, in the community, one of the policies that I was just working with somebody to develop is to make sure that workloads, workload identities are protected based on the image name. So, you know, what the author did, which is pretty nice, is use the config map to make sure that a certain service account can only be used for certain images, right? And they're managing this, the data through a config map, but then it's a very generic policy to do that. Another common example is to, you know, make sure your ingress is unique within a particular, you know, cluster, right? Your host names for your ingress, I should say, are unique. So Keverno has this ability to use James Path and also to do API look calls. So this is a combination of using those two features. So just going through the structure of the policy, this policy matches ingresses, and then there's a context, which, where you can populate different data and different variables. So you're, what it's doing is it's saying, it's doing a call to, you know, to the API server, which is this API call construct. It's getting all the ingresses from the API server, and then it's applying this James Path on it to extract out all of the hosts. And then it's checking if that host is, you know, the host from the request object is already, is included in that list of existing hosts, it's going to deny the operation, right? So fairly complex logic, but pretty straightforward, you know, once you kind of get the hang of how this is structured. And the interesting thing is, you know, if you use kubectl and, you know, just through the command line, all of this can be tested very easily. So I can do kubectl, I think it's get minus minus raw. Yeah. So if you do kubectl get minus minus raw. So you're, I don't have any ingresses in this cluster, but you can kind of see what that looks like. And even if you want to test the James Path expression, you can use JP, which is the command line for that. And if I go back to my policy, let's grab what this looks like. And of course, if I had some ingresses, this would maybe make more sense. But if I apply this right now, it will come back with an empty string, right? Because there's no, there's nothing. So it just came back with an empty list over here, as you see. If I had any ingresses with hosts, it would show me a list, a string list of those ingress hosts. And then in my policy, what I'm doing is I'm checking to make sure that the host, which is coming from the request object is not, you know, already used within the cluster. So, you know, this is a simple example, which shows again how you can combine some of these. The other thing, you know, which is very interesting is there's, you know, policies, I think Ritesh mentioned also for multi-tenancy. I can show a few examples of, you know, those. What we're seeing is, first of all, adding labels to namespaces is a very common use case.  We're going to be doing this at the Cloud Security Day for Kupkan EU, where we're going to talk about how Kiverno and HNC can work together and manage both namespaces.  So how you can kind of allow those controls. So you're, anyways, in this policy, it's pretty straightforward. It's saying that if there's a namespace created, except by cluster admins or by the HNC manager, go ahead and inject, like, the user info. And then you can also do more complex things, like you could say, okay, maybe I want three types of namespaces. I want a small, medium, large. And based on that, now I'm going to configure, you know, different quotas and different things for my tenants, right, for my users. So all of that can be fairly easily done. You can also generate very fine grained roles and permissions. So only the person who requested that namespace, the owner will get permissions to then delete that namespace. Things like that can be then, you know, generated on the fly. So these are Kiverno policies to do those kind of things. So we'll share this repo once, once, you know, we have this finalized for the session. But this is just a quick example of how now you can use the power of, you know, leveraging labels, and then based on automatically generating different, you know, configurations for those namespaces. And of course, you can also add like validation logic to make sure that only the right settings, like in this example, it was requiring that each namespace only have a certain, you know, can only be used with a certain suffix, right, so dash SM or medium or large. And if it doesn't have those, it would be rejected itself. So these kind of things, you know, you can combine, you know, validate, mutate, and generate to get exactly the right behaviors you want for your cluster. And once these policies are set, they can, they're very much data driven. So, you know, the configurations itself can be easily automated in there.  And something one of our community member created, and that is made available, this is also an open source project. He created something called a policy reporter, which takes this policy report and also provides a nice graphical UI on it. He's also got the ability to push this to Rafauna Loki and Elasticsearch and create other notifications. But really nice example of how, you know, you can now view this graphically, you can share this inside a single cluster. So just wanted to highlight that. So if you're looking for a single cluster tool to, you know, visualize these policy reports and even create notifications when these get created, definitely check out this policy reporter project. And, you know, there's also work in progress, by the way, to take more Kiverno metrics and push these to, you know, to Prometheus, which could then be, of course, displayed in various UIs and dashboards like Rafauna and others as well. But this is more focused on the policy report. The other work that we're starting now is to build, to push engine metrics and engine statistics out as well. All right, so let me stop there. I know we have about five minutes left for the hour. So we'd love to see if there's any other questions, thoughts, comments that we can help answer.  And if you want to reach out to us, feel free to reach out on the Kiverno Slack, which this is the on the Kubernetes Slack, the Kiverno channel itself. Well, that was one thing I was actually noticing is that your documentation is awesome. So congrats on that, because I think that's one of the things that drives communities and really helps people adopt. We have a couple people on the call that had some questions. Paul Maury, if you want to pop yourself out of mute and onto the screen and Kirsten, newcomer. Yeah, so I heard a lot about pods. I didn't hear whether you can write policies around custom resources. So like say that I have a custom resource for my own bespoke controller that creates pods. It contains something that's like pretty close to a pod spec. Can I write a Kiverno policy on it? Absolutely. Yes. Yes. So one of the things right from the beginning Kiverno has supported is also full custom resources. And it does help if the custom resource schemas are structural. So if they're structural, what Kiverno can also do is can validate the various paths and fields for that custom resource policy. If they're not that validation doesn't occur, but Kiverno will still apply the policies. Okay. And how about if I have like an aggregated API, so can I write policies against APIs that another API server might bring in? So you can make calls to an aggregated API server, right? So through the API called construct, if you, for example, if you want to pull metrics, things like that, that would work. If there are resource definitions, yes, then those can still be, you know, looked up and then policies can be applied to those. Okay. All right. Thank you. Sure. There's one question. Kirsten in the chat, Mikey's asking, do you need a CNI to enforce the policies? So for network policies, yes. If you're generating a network policy, you will still need a CNI to provide that runtime, you know, kind of to make sure that, you know, the right segmentation exists on the network layer. But for other policies like pod security, et cetera, those are applied directly with the API server and Kiverno itself will block configurations that are violating, you know, the policies you set. Kirsten, take it away. Sure. Thanks. Oh, Mikey had a follow-up. Is the, yeah, go ahead. Is the mutate policy in cache until validate is run? Yes. So there is a, there is a ordering in, you know, the admission control. So the validate will occur, you know, at the end. So all of the mutation logic will occur. And then once that's done, then you get a chance to sort of validate. There's also a way to re-invoke admission controllers. So Kiverno by default, when it installs itself, it tries to be the last admission controller to receive. So there are, you know, there are some customers who will use both OPA and Kiverno for various reasons. And in that case, you know, you could still run one particular policy and then register to read, receive any mutate results. Great. Take it away. Thanks. So I had kind of a question about network policy generation as well. How do you inform the generation of that network policy? Are you using config data to inform that? Or is there, you know, or is that optional? Kind of, how do you decide on what's going to be in that policy that's generated? Right. Yeah. So the few things, right? So the trigger to generate the policy could be the creation of a namespace, could be the setting of a label or the setting of an annotation itself. So one pattern is if you're using, if you want some variations and you want to control, you know, like for example, what exactly gets generated, you could use labels and based on labels, generate, you know, have templates for different types of network policies. And that would be, yeah, one quick way of, you know, of changing that. You can also do things like, because Kivarno can look up the namespace, can match namespace selectors, you can say that if for certain namespaces and if there is an annotation on that namespace, then based on that, maybe you want to trigger different network policies. So then you would have two different generate rules, one for each. So, you know, a few different ways of managing those kind of variations and handling that. Great. Thank you. That's helpful. And I know we're pretty much right at the top of the hour. I'm actually, I have two questions, but I'm going to pick one of them. Do you have thoughts on kind of policies or ways that we as a community might move forward to kind of apply policies to respond to events in the runtime? Yeah, great question. Right. And something that we have been thinking about, there's been discussion also in the policy working group about what other triggers, so admission controls and configuration changes, of course, are one trigger, but it would be very interesting to also have a set of, you know, defined runtime triggers. There's also discussion, you know, about authorization checks, right? So somebody in the community was asking about, you know, that there's a self-subject review request that gets sent out from the API server when there's an authorization check. So, you know, for having some more granular policies on that. But, yeah, to your point, like if there's other runtime events, even those could be potentially used for triggers. But it's something that it's not supported right now in Kiverno, but we're very open to that and interested in hearing more about the use cases and how that could be standardized. Great. Thanks so much. That's it from me. All right. Well, thank you guys for coming today. It's wonderful to have these upstream talks on Mondays because I really didn't know that much about Kiverno before this, that I was pretty OPA-centric in my understanding of policy management and such. So it's really helpful to have the context set. And so I think everybody appreciates that as well as Jim, awesome deep dive and demo there today, too. So, well, definitely you're in the sandbox now. Yes. The road ahead, more production users, more people using it, more feedback on it, hopefully incubation and graduation into the real world today sometime soon. Absolutely. And we'll definitely get you back on as it matures and we get further down the path. And congratulations on getting into the sandbox and getting it to this state. Again, if you're interested in this topic, these guys are pretty active in some of the SIGs and the CNCF as well as, as I mentioned earlier, the documentation is rock solid. So good on that. That's as a community person, that's one of my key tests, whether we can keep something going forward. And definitely when you get a new release, new features, new functions, reach out to us, Ritesh and Jim, and we'll have you back. So good work. Good luck. And thank you. Thanks again for sharing the information with us today.


Thank you Dominica. So I've had the privilege of studying high-performing technology organizations since 1999 and those specifically those are the organizations that had the best project due date performance in development. They had the best operational availability and stability in ops as well as the best posture security and compliance. And so our goal was always to understand how did these amazing organizations make their good to great transformation. So why would we do that is because we want to understand how other organizations could replicate their amazing outcomes. So you can imagine in that 18-year journey there were many surprises but by far the biggest surprise was how it took me into the DevOps movement which I think is urgent and important. The last time that we've seen any industry being disrupted to the extent that our industry is being disrupted was probably in the 1980s when it was revolutionized through the application of lean principles and I think that's exactly what DevOps is, is the technology value stream when you apply those same lean principles. So in the next 30 minutes what I want to do is share with you what I've learned since the Phoenix project came out in 2013 and it is impossible for me to overstate how much I've learned. So what I want to do is share with you what those biggest learnings are and some ways like these are things I wish I had learned before we actually put out the Phoenix project in 2013. The first one is to what extent the business value that DevOps creates by applying DevOps principles and patterns. And so this is work that we did with Dr. Nicole Forsgren who is here somewhere in the room, Jezz Humble, a DevOps handbook co-author as well as Alana Brown from Puppet. And so what we found is that the high performers are massively outperforming the non-high-performing peers and this is based on four now going on four years of research spanning 26,000 respondents or will be by the end of the year. So surprise number one is how high performers are just getting a lot more done. They're doing 200 times more frequent deployments and so that could be deployments of code or that could be deployments of changes in the environment. And more importantly they can complete those deployments 2,500 times more quickly. Notice how quickly can we go from a change being introduced into version control, right? Not just for development, that's for everybody, right? Version controls for everybody. Do some sort of test process, do some sort of deployment process, so it's actually running in production so customers are actually getting value, right? High performers can do it in usually minutes, worst case, hours whereas lower performers might require weeks, months or quarters. So it's not just that they're getting more done, they're getting far better outcomes. When they do a production deployment, high performers are one-third as likely to have that change blow up and cause a several outage, a service impairment, a security breach or a compliance failure. And when something goes wrong, they can fix those issues 24 times more quickly. In other words, the mean time to store was 24 times faster. So this was such a decisive finding when it first came out in 2014 because it showed us that, you know, this deeply held and common experience that we had in general, the larger the size of our deployments, the more things that go wrong and the bigger creators that we make in the data center, right? The only way that we can get these sort of reliability profiles is to be doing smaller deployments more frequently. So this has been validated year over year. Last year we found another dimension of quality, which is that because high performers are integrating information security objectives into every stage of everybody's daily work, they're spending one-half the amount of time remediating security issues. And because they're able to control unplanned work, they're able to deploy nearly a third more time on planned new work, right? The higher strategic activities versus the lower value firefighting, which is probably the first page of the Phoenix project, right? This is what firefighting feels like. In 2014, you know, we found that not only did high performers have better IT performance as measured by deployment frequency, lead time, mean time prepare, change success rate, they had better organizational performance. They found that high performers are twice as likely to exceed market share profitability and productivity goals. And for those nearly 1,000 organizations that year that gave us a stock ticker symbol, they had 50% higher market cap growth over three years. So this last year we found another marker of organizational performance. In high performers, employees are 2.2 times more likely to recommend their organizations as a great place to work to their friends, as measured by the employee net promoter score. And so this is just a great proxy measure for organizations' ability to hire and retain great talent, right? So I think all of these gives us a better ability to sell DevOps within our organizations. And Nicole and I will be working on the 2017 State of DevOps Report. I can't tell you a lot about it, but it's freaking awesome. Just when you think you've learned it all, there's more to learn. So surprise number one was just to what extent high performers were outperforming their non-high performing peers. Surprise number two is how DevOps is as good for operations as it is for development. One of the cases they got a chance to study with Jez Humboldt for weeks was this case study back in 2008. And it's a Facebook chat launch story. And some of you may roll your eyes saying, wow, what could be so interesting about that, right? Chat servers are what undergraduate CS students write as part of their sophomore year of college, right? And so although that's true, what you may not know is that chat is inherently an order-and-cubed algorithm. And so at Facebook, N is 70 million simultaneous users. So that was actually technically, it was considered one of the most massive technical undertaking that Facebook. It took them one year to do. It was one of the largest project teams ever assembled. And so there were two technical practices that blew me away. One was the notion that they were testing in production for, well, so how did they use that year? So as soon as they constituted the chat team, even on day one, they were checking their code into a shared source code repository. And anything that was in the source code repo in trunk would be migrated to the production environment at least once per day. And they would do it in the middle of the day, 2 p.m. Pacific time, right? Not at midnight, not on Friday, right? Not working on weekend. They would do it in the middle of the day. And then the second thing was that they were using every active Facebook browser user session as a test harness. And the reason they did this was so they could simulate production like loads, even at the earliest stages of the project. And so the result was when they dark launched this one year later, they went from zero users to 70 million users overnight. So that's a dark launch. The launch was simply changing a configuration flag, right? And if something went wrong, they could just undo that configuration. But for me, the more interesting practice is that notion of a daily deployment, right? That they don't do it at Friday at midnight and make people work under horrendous conditions all weekend long to get things running before customers notice on Monday morning. And I think the best verbalization of this came from Nathan Schemek. He told me in 2013, it was actually at Jez Humboldt conference, Flocon at the Bar, he said, as a lifelong ops practitioner, I know that we need DevOps to make our work humane. He said, over the course of my career, I've worked on every holiday on my birthday, even worse on my spouse's birthday. And even on the day my son was born. So I think some of you may have friends, you know, have been in that situation, right, we're out of a sense of duty or obligation or maybe they because they didn't have a choice, right, they've had to do this. And some of you may be like me, where you've been a part of the leadership that I've created these inhumane work systems. And I think what makes DevOps have so much significance is that we now know that there's a better way, right, it doesn't have to be this way. Unless you think that this is only possible with open source hippie companies like Facebook, right, you should know about this case study from Scott Prude from CSG, the largest bill printing company in the United States, they're publicly traded. And if you get a paper bill from a Comcast charter communications direct TV, chance there comes from one of the two bill printing plants in the US from CSG. And this thing is, in my mind, one of the pathological worst case architectures of what can do DevOps on, you know, this bill printing application runs on 20 different technology stacks, including dot net thick client then client J2EE, COBOL, assembler, COBOL, mainframe, COBOL, mainframe assembler, mainframe VSAM, mainframe DB2, right, it's all in there, right. And so to execute a deployment, it required 20 simultaneous deployments on 20 different technology stacks for it to work, it would take them 14 days to execute, right. So, you know, they were, so they over the course of the year, they moved to a DevOps transformation, and they went from two releases a year to four releases a year, but success was predicated on the notion of a daily deployment. Every day, a team spanning DevTest and operations would deploy into a UAT environment. What were the outcomes? Within a year, incident count went down by 90%. Meantime to prepare went down by 98%. But most importantly, the code deployment lead time, right. In other words, how went from 14 days down to a day, right. So 14 days of a release team trapped in a war room, right, trying to get things running, right, with executives coming in every hour saying, are we done yet? Right. To which they would have to honestly respond, no, we're not done yet, we have 13 more days to go, right. 14 days within a year, it was done by 1pm on day one, right. And the Xbox has come out, right, because there are no life side incidents. So it's great for DevTest and operations, but also it's great for the business and customers, because often they can get the value of the features in half the time. So, yet there's another side. One of the more interesting patterns for me in watching DevTest, this is the notion of developers being put on page rotation. Patrick Lightbody said in 2011, he said, what we found was when we woke up developers at 2am, defects got fixed faster than ever, right. And Werner Vogels said it even was succinctly, if you help build it, you must help run it. And so I'm very well aware that jackasses like me, showing off jackass slides like this, is probably mobilizing an entire generation of developers to hate DevOps, right. They will sabotage every DevOps effort they see, because they would say, we did not become developers to wear pager. Pagers are for ops people, right. The whole reason they became ops people is because they're like pagers. And so, although I will recognize that there is an internal consistency to that logic, I think there is a more compelling narrative. And that comes from Tim Tischler. For many years, he led the DevOps Initiative at Nike. And he said, as a developer myself, the most satisfying point of my career was when I got to write the code, when I got to test it myself, when I got to push into production myself, when I got to see happy customers when it worked, and when I got to see their angry shaking fists when it didn't work, and when I could fix it myself. He said, not only, you know, the point is, you know, if I didn't have to open up a ticket and wait a day, the point is that not that I could have fixed it faster, which is true, the point is I could have learned something, so I didn't make the same mistake the next time around. And what he said was that, you know, our inability to self-test, self-deploy, self-fix has diminished over the last decade. And paradoxically, he's actually being put on page rotation that allows us to get that joy back. How am I doing this? Am I being too cavalier about that claim? Kind of cool, right? It's great for ops and it's great for dev. And I think it's because, you know, ultimately we're both engineers, right? And increasingly as ops professionals, I think we're going to be using development philosophies, development practices, increasingly the same tools as developers, and ultimately it's because we're both engineers working in the value stream together. So, surprise number one was the business value of dev ops. The second surprise was just how great dev ops is for both development and operations. The third surprise was there's this measurement that looks very tactical and easy to dismiss, but I actually think it's probably one of the most strategic measurements of any technology organization. And specifically it's code deployment lead time. So, even though in the dev ops community, I think the one metric that we love talking about is deploys per day, right? It's kind of embedded into the dev ops community, right? Flickr, we did 10 deploys a day every day. You know, it was per the famous Allspaw Hammons slide in 2009. But in the lean community, especially in manufacturing, that is obviously not their favorite metric. Their favorite metric is lead time.  And what we found in our benchmarking work, spending 26,000 respondents, is that that property applies to our work as well. So in manufacturing, they would probably measure lead time as how quickly can we go from a customer order or raw materials at one of the plant to finish goods leaving the plant. In our world, in our research, we specifically measured lead time as, you know, the point at which change and they're introduced into version control, right? Through tests, through integration, through tests, through deployment, so that customers are actually getting value. And I think that begs the question, why do we start the lead time clock there, right? Why don't we start it earlier when a feature is accepted by development or even when an idea is first conceived? And the big learning for me was that it's the point of which change introduced into version control that is a dividing line between two very different parts of the technology value stream. So to the left of being put into version control is design and development. So the nature of design and development work, like the Facebook chat story, right, is often we're doing work for the first time, maybe never again to be repeated, right? So the lead time of that kind of work is longer and it's highly variable, right? Because we never get a chance to practice it. So I think that's just a fact of life. Everything to the right of changes being committed into version control is we want the exact opposite characteristics of work. So that's testing and operations. We want testing and deployment to be happening all the time. We want it to happen mechanistically, repeatedly, the same way every time, and we want it to happen quickly. So code deployment lead time simultaneously predicts the effectiveness of testing and operations, but it also predicts how quickly can we create feedback for design and development. In other words, if I'm a developer and I make an error and I only find out about it nine months later during integration testing, then the link between cause and effect has surely been lost, right? This is where we get into the blame game, you know, who changed what? Not me, it must be that person, whatever. In the ideal, that error should be detected within minutes because that's when we kick in quick automated testing, right? And so that not only creates fast feedback for developers, but it also predicts how quickly can design and development get feedback from external customers. You know, you can't do a lot of experiments if you're only deploying once a year, right? So code deployment lead time is a great predictor of testing and operations effectiveness, as well as how quickly we can create feedback to design and development. So that was learning number three. So surprise number one was business value of DevOps. Two was how great it is for Dev and Ops. And three is this tactical measurement is actually not so tactical at all. In my mind, it's probably the most strategic measure of any technology organization. So prize number three was Conway's Law. And so I noticed like around 2012, it was very difficult to go to any DevOps event and not have someone talking about Conway's Law. And so for those of you who aren't familiar with it, I think the most popular incarnation of it was actually framed by Eric Raymond. So he wrote the great book Cathedral on the Bazaar about open source, but he's also responsible for the Devil's Dictionary. And his definition was this, if you have four groups working on a compiler, you will get a four pass compiler. And this is based on a famous experiment that Dr. Melvin Conway did in 1968. But so I think I kind of intellectually got that, but I wasn't, I'll be honest, there's no way that could have actually explained back to you how that would impact how we design our work and how we execute that work in the DevOps value stream. So one of the biggest aha moments was for me was just seeing Conway's Law at work in this famous story about Sprouter at Etsy. So let me just tell you the story and show you how Conway's Law actually is the backdrop of it. So back in the battle days at Etsy in 2008, in order to ship any piece of business functionality, there were two teams that were required to have to do work, right? You had the devs working in the front end and PHP, and you had the DBAs making changes to the stored procedures and Postgres, right? So these two teams would have to coordinate Marshall sequence, prioritize, blah, blah, blah, blah, all right then. And so they said, this is a problem, right? So their solution in 2009 was to create Sprouter. So Sprouter stands for stored procedure router. The idea was we're going to be able to enable devs and DBAs to work independently and they'll meet in the middle inside Sprouter. The problem is, is that now we went from two teams having to coordinate and sequence and Marshall work to three teams having to coordinate sequence and Marshall work. And as they said, this required a degree of synchronization and coordination that was rarely achieved. Every deployment became a mini outage, right? So as part of the great Etsy transformation, they said, all right, where we have to go is being able to fully empower developers to independently make changes just by making changes inside of PHP. So they created this PHP ORM, object relational model thing so that they wouldn't have to make changes to the database directly. And so the end result was not only did reliability go up because, but lead time went way down, right? Because I think, and this kind of shows how the goal, one of the goals of DevOps is to fully enable small teams to independently develop, test and deploy value to customers, right? And so I think this is a great example, at least it was for me of how Conway's Law can hurt us, 2008 to 2009, as we went from two teams to three teams and how Conway's Law can help us as we went from three teams down to one. So the real lesson for me was that the organization and our software architecture must be congruent, right? It's not enough to shuffle teams around. You know, we must also have an architecture that enables those teams to independently create and deploy value to customers, right? And we can't do that if all the teams are tightly coupled together. It means that every time we wanna do something we have to coordinate with maybe hundreds of other developers, testers and ops people. Okay, and by the way, incidentally, that becomes almost impossible when we organize our teams by the technology. If we have 20 different teams, DBA's here, oh no, Postgres's there, MySQL there, Oracle there, J2E there, Windows there, right? You mean everything has to have a lot of handoffs, so. So surprise, one, business value of DevOps. Two is DevOps is great for Devon ops. Three is code deployment lead time. Four is Conway's law has a lot to do with the outcomes that we get. So the prize number five is that I think when historians look back at the DevOps movement and technology in general, I think they'll say DevOps is probably a subset of something much larger and that they would call it probably dynamic learning organizations. So Dr. Steven Spear, he wrote one of the most famous Harvard Business Review papers and it was a paper called Decoding the DNA of the Toyota production system. And that was based on his PhD dissertation he did at the Harvard Business School. And as part of that, he actually worked on the assembly line on the production floor of a tier one supplier for six months at Toyota. And before the Toyota executives let them do that, they said you must first work in a big three auto plant for 30 days, right? Essentially saying, you will not understand the lessons that are going to be imparted on you until you work in a more conventional plant. So he extended that work beyond the Toyota production system to helping build a safety culture at Alcoa, to engine design at Pratt & Whitney, to design and operations of the US Naval Reactor Corps inside the US Navy and so forth. And he said, while designing perfectly safe systems is likely beyond our abilities. And by the way, there's no work that is more dangerous and complex than the work that we do potentially. Safe systems are closely achievable when four following conditions are met. So what I wanna do is share with you what those four conditions are. And I'll just highlight some of the technical practices that this might remind you of. But I wanna highlight on one of those capabilities because when I took this workshop at MIT, it just hit me that there was a big blind spot that we had within the DevOps handbook authorship team. In fact, I would blame Dr. Spear for about a two year delay in the five years that it took to get the DevOps handbook out. So Dr. Spear asserts that there are four conditions that most exist. One, you must see problems as they occur. In other words, any sort of assumptions that we have that are incorrect must be quickly revealed both in the design and operations phase of any complex work system. And so that's like assertion statements and code. That's like production telemetry. That's like all the telemetry they wanna put everywhere so we can actually see is the system behaving as we think it is, so we can actually correct it. The second capability is that when bad things happen, we must swarm it not only so we can restore service faster, but so we can create new knowledge. And so the paragraph of this principle is Toyota and non-cord, right? When something goes wrong, you pull the cord, the entire assembly line stops, and they do it 3,500 times a day, right? And essentially what they're saying is that we need to make systemic fixes then and there because if we don't, we're gonna have the same problem 55 seconds later, right? And so that's the notion of the daily work around. So daily work arounds happen in our world but because our work takes longer than 55 seconds, it's just less visible, right? But it is just as destructive. So in our world, this would be like continuous testing, continuous builds, continuous deployment, dropping whatever takes when something goes wrong, helping peer review other people's code because getting them into production is actually more important than whatever I'm doing right now, but in tomorrow it might be the other way around. I might need someone else to peer review my code, right? Because lead time predicts effectiveness. But the real big surprise for me was capabilities three, there has to be some mechanism where local discoveries can be integrated to create global greatness, right? In other words, how do we elevate the state of the practice so that genuine learnings are created and integrated everywhere? So I wanna share with you, and then number four is leaders create new leaders, but let me share with you like what capability three is all about. For me, this was the most profound. Like what are ways that we propagate learnings in the DevOps community? One is the notion of a single shared source code repo, right? And I think this is so important for operations and information security. The whole idea is that we put our best expertise into code so that anyone who pulls from it can inherit the best known understanding and expertise of the entire organization, right? The most famous example, this is the monolithic source code repo at Google. They have, every engineer has access to all the Google properties and everything gets executed through a continuous deployment pipeline and build systems inside the repo. Only one version of each library allowed. Contrast actor friend of mine, said I'm at a large bank and he said of the 93 versions of Java struts, we are running 92 of them in production, right? So consistency and conformity, blameless postmortems, this idea that when something goes wrong, we create the condition so we can talk about problems. As Bethany Macri from Etsy said, prevention requires honesty. Honesty requires safety, right? How do we make it possible to really create an accurate timeline of what actually happened so that we can actually talk about honestly what the right counter measures should be so we can ideally prevent those bad things from happening again. If we can't prevent, at least enable quicker detection and recovery. Chaos monkey, right? It's like eventually you run out of things to talk, if you stop having several outages, you don't have enough blameless postmortems. So you talk not about just customer impacting incidents, you talk about team impacting incidents of the seven safeguards that were designed to prevent a customer impacting incident, six of them failed, right? And if you run out of those, eventually you have to create your own failures, like chaos monkey, right? Where we like Amazon, I'm sorry, Netflix randomly kills production compute instances all the time in production. By the way, did you know that they only run that during office hours, right? In other words, you don't actually wanna wake people up needlessly at 2 a.m. You do it when everyone's in the office, just like when you would do a deployment. Learning days, DevOps days, internal technology conferences, another way where we can actually have people who are creating greatness be able to spread and propagate that and set the control norm that this is what we want within our workforce. And certainly I think open source is a part of that. Just one little side note on this, one of my favorite quotes that I think, talk about, think about it all the time is this. You're only as smart as the average of the top five people you hang out with, right? And so I think it's organizations like this where we can actually create the peer group, you know, where we can actually learn. Okay, oh, such great story. No, I actually put these slides in for you, but I don't have time. All right. Surprise number six is, I think there's this misnomer that DevOps is just for the unicorns. You know, that's Google, Amazon, Facebook. It's not for the horses. Large complex organizations that have been around for decades or even centuries. And this has been my area of passion for the last four years, which is studying not so much the unicorns, but you know, how DevOps principles and patterns are being used in organizations have been around for decades or maybe even centuries. And so we're going to the fourth year of a conference that we call the DevOps Enterprise Summit. And the goal is really to collect these learning. In fact, there are 48 case studies in the DevOps Handbook. 30 of them are from large complex organizations and they almost all came from this conference. We asked leaders of technology organizations to tell experience reports. Here's the industry we compete in. Here's my organization. Here's where I fit in the org chart. Here's the business problem we set out to solve. Here's what we did. Here's what we learned. Here's what the problems that still remain. And the reason for that is that as adult learners, we don't learn so much from theory and what people say we should do. We learn from what people did. So we can sort of conclude the learnings that we need. And so, by the way, we did one in London last year and what was interesting about London was just the age of these organizations. Barclays was founded in the year 1634, right? UKHMRC, Her Majesty's Revenue Collection Service was founded in the year 1200, right? So I don't think there's any code that goes that far back but there are certainly traditions and values and practices that go that far back. So there are many, many awesome DevOps outcomes, right? There should be no doubt that large complex organizations are achieving the same outcomes that the unicorns have been achieving. But there's one thing that is astonishing to me which is the level of courageousness that are being exhibited by these leaders. I think every one of them was given some degree of air cover but I think almost every one of them, at some point in their journey, wildly exceeded the air cover they were given, essentially putting themselves into some degree of personal jeopardy. And so the question is why would they do that? And I think the reason is that every one of them had a sense of absolute clarity and conviction that what they were doing for their organizations was needed not just to survive in the marketplace but also win in the marketplace. And as a little example of what courage looks like, I got to shadow Heather Mikman for many years. She was a senior development director at Target and I noticed a certificate on her desk and it looks like this PrintShop Pro type thing, right? The certificate and it says, to Heather Mikman for Lifetime Achievement Award for Annihilating TEP and LARB. So I asked her what is TEP and LARB? TEP stands for the Technology Evaluation Process and LARB stands for the Lead Architecture Review Board. So whenever you want to do something novel and scary, like say use Tomcat, right? You would fill out the TEP form and eventually you would get the rights to be able to pitch the Lead Architecture Review Board. And so you walk into a room and there's all the dev architects on one side, all the ops architects on the other side. They pepper you with questions, they start arguing with each other and they assign you 50 more questions and say come back next week, come back next month, right? Her reaction was why no one on my team should have to go through this. In fact, none of the 2000 engineers at Target should have to do this. In fact, why is this even here? And she said, no one could really remember. But there was some vague memory of something terrible that happened 16 years ago, but the details have been lost. And so some months later, they actually abolished the TEP and LARB, right? And I think that is like one of the markers of these people driving these transformations. I wanna actually have now learned how to actually more specifically talk about this. I was talking to a friend, Dr. Stephen Maynard. As part of his PhD program, he was studying transformational leadership and I asked him, what's that? And he rattled off these characteristics and my jaw hit the floor when I heard them because in my mind it actually exactly verbalized the behaviors and values I've seen in these leaders. So I just wanna share with you what those are. The first one is inspirational motivation, they say. Can you articulate a clear vision, inspire passion and help get other people on board? The second one is idealized influence. Can you be a role model, set the example? Can you be a lifelong learner and encourage that around others? The third is individualized consideration. Can you coach others, enable others, keep lines of communication open, recognize other contributors? And the fourth one I love, this notion of intellectual stimulation. Can you challenge status quo? Do you have this relentless need for improvement? In other words, just because we did it for 16 years the same way, this suddenly unacceptable now and I'm gonna focus, that's where I'm gonna focus on. How do you empower decision making and so forth? We did this little experiment where we asked about 100 people to take a MLQ assessment and it turns out that we found that the DevOps Enterprise community self-identified as transformational leaders and that we actually integrated that into the 2017 State of DevOps report and holy cow, exciting news and findings to come. So with that, why do I think this is important? It's because I think over the years I've thought about DevOps a lot of ways in terms of what is the mission and I think I've settled on this. IDC, the analyst firm says there's about eight million developers on the planet, eight million ops people on the planet and at best, I think the wireless opnists could say we're at 0.5% adoption. You take all the unicorns, you take the segment within the horses and so that says we have 99% left to go and I think the real goal of that is how to get every one of those 16 million engineers to be as productive as if they were working at a Google, Amazon or Facebook and there's no doubt that when we do that we will unlock trillions of dollars of economic value per year and that's not gonna happen in unicorns, that's gonna happen in every of the largest brands in every industry vertical. So I think that's the mission at hand and yes, many people still joke that yes, it took five and a half years to get out but it's out and again, the thing I'm proudest of is like the 48K studies. Most of them are from large complex organizations and I thought it was definitely worth the time that we took. So if you're interested in a 340 page excerpt of both the DevOps Handbook and the Phoenix Project, all the videos and slides from DevOps Enterprise are online.  Don't take a picture, don't write it down, just send an email to realgenecomentsenderslides.com subjectline DevOps and you'll get an automated response in a couple of minutes. So with that, Domenica, thank you so much and I guess I get to hand it over to Nell. Thank you.


Hello everybody. Welcome to the Alabama 4-H ESPN page. This is your host, Warren J. Williams. And my next guest is one of my local coaches who is the coach of the Ufala High School Girls Basketball Team, Coach Jameet Clayett. Mr. Jameet Clayett, could you say hello? Hey, hello. How you doing? My name is Jameet Clayett. Ufala High School here at Varsity Girls Basketball Coach. Okay. So how you doing, Mr. A. Coach Clayett? I'm doing well. How about you, sir? Doing good. Doing good. You ready for that season? And we're ready. We've been working hard. So, hey, we're ready. We actually got practice here in about two more hours. But, hey, we're ready. We're working hard and looking to come to the big goal this year. That's good. That's good. So, in this interview, I have a couple of questions for you if you don't mind if I ask. All right. All right. So the first question, Ashley, what was your major in college? My major in college was physical education. I have a bachelor's from Troy University, where I graduated in 2013. Then two years later, I decided to go back to school and I obtained my master's in athletic administration. And currently, I'm back in school again to get my second master's, which I'm interning currently at the moment here at UFALA Higher School. My second master's is in instructional leadership, where I can be an administrator in the school system. I actually be graduating in December. I'm in my last intern. Well, in my first intern class now, my second intern will start in the fall. I'll be graduating in December from the University of West Alabama in instructional leadership. That's nice. I didn't know that. Well, class of 2020. Moving in silence. Moving in silence. Hey, I know what you're talking about. All right. Well, next thing. Next question is, what is a day like as a coach? Wow. First thing I'll say, it can be stressful if you live, but you know, a day like as a coach, man, to be honest, I was just speaking with a former player who I've coached in the past, man, just giving her some words of encouragement as far as, you know, what to major in, because she's headed to Wichita State. And I was telling her what you're going to major in. And she was like, I don't know. I said, well, what you like to do? She was like, place basketball. I said, well, if you can't play it one day, you know, that ball is going to stop dripping. So you might well, you might as well be a coach. She's like, yeah, yeah.  And I told her this, I said, if you can do something without pay, then it's not a job to you. So, you know, I've been coaching basketball for maybe 14 years. And I'll say seven pay you seven unpaid. So I did it for seven years for free. So again, it's something that I enjoy doing it. But the life of a coach man is rewarding because you're able to help kids through sports, discipline kids through sports. I love every minute of it, got a passion for it. You know, again, basketball teaches life skills. I'm very big on teaching life skills. My team that I have currently, they'll tell you, I use the game of basketball to teach them life skills.   Teams that's going to beat you. You're not going to win every single game. I don't know a coach in this world that has won every single game that he has coached just like life. You're not going to win every single battle. You're going to have some speed bumps, you're going to have some obstacles in life.  So, man, you know, the life of a coach, man, it's rewarding, man, to get phone calls from every day from all types of coaches from the college level, high school level. So, man, you meet a lot of people networking.  Hey, you coach here, you coach there, man. And, you know, it's a great feeling. It's really rewarding. Yeah. I consider basketball one of the best, I guess you could say social sports. This is where you can really make friends out on the court. Everything's competitive. Yeah. Well, next thing, what is the best thing about your job that you like to do? Wow. The best thing, that's a good question. The best thing about my job, again, is interacting with my players, getting more personal, building relationships. That's the biggest thing because, you know, basketball versus football, where football, you've got maybe 50, sometimes 50 to 100 football players out there. You're going to get no kid, but basketball, you're talking about dealing with 8 to 12 kids or 13 kids, whatever. So, you begin to form a bond with them, a relationship where you can get personal with them, man. And when you get personal, you know, it's about helping them become productive citizens in the world that we live in here. So, you know, it's basketball, man.  You gain a lot of distance skills because, again, everything is not going to go your way. You gain some motivation. You know, you got to play a lot of games. So, you're fatigued, you're tired.  Life is about adversity, how you respond to it. You know, when things are going well in life, guess what? You don't have any problems. You're living a good life. But boom, the minute you come to a speed bump or the minute something happens and you face adversity, how are you going to respond to it? So... Oh, yeah, definitely. There's a lot of players in the NBA and in college level who always go back to their coaches saying they had to build a good relationship. They always keep training.  Definitely, man. Like I said, building those relationships, that's the most exciting and enjoyable, you know, thing to me is building that relationship and seeing them basically get somewhere that they never thought they could. You know, I pushed them, pushed all my players and things that they say they can't do, you know, they find out that they can't do. You know, things where they lack confidence, they leave with confidence. You know, self-esteem, low self-esteem, they leave with confidence. You know, so the thing is, man, you know, building relationships is the thing that really, really gives me as far as a coachman and just really makes me feel good when I see that smile and I say, hey coach, I did it.  I had a kid that I never thought, you know, who I thought never would get it.  You know, if she had an attitude or I thought she wasn't getting in and she had problems with whatever. But later on when she went to college, she sent me a text message. She was like, coach, I just want to say thank you, you know, for being hard on me. Thank you for this with me. Thank you for teaching me life skills and the things that you taught me because I'm way out here in this part of the United States and the first thing I could remember is the things that you taught me on and off the court. So that goes back to building relationship.  So it's very important, you know, building a relationship. That's good. That's nice. All right. So this question, I feel this is a very good question. What advice would you give a youth interested in your job as coach? Wow. The best advice is number one, patience. You know, even with coaching in sport, you got to be patient.  You got to develop and you got to build your culture, build your great program. When I say patience with parents, you know, parents, it's got to work both hand in hand, patience with the coach, coach patience with the parents. You know, you're going to have parents that, you know, want their kids to play.  Well, my child is going to play, but you got to let her know that it's different from rec league to high school. And then so you got to be patient with the coach because again, a coach, I don't, I believe it takes a coach three years to build a program, makes the program his, you know, his or hers. So, you know, being patient with administration, things that you want, the things that you need for your program, you might not get it firsthand, you know. You know, it's a couple of things that I didn't have my first year, but as I began to be patient, the things that I needed, the things I asked for, I began to get it, you know. So, you know, being patient in this sport, also being passionate. Love what you do. Again, nobody wants to work or nobody wants to do anything that they don't love. And so, you know, in anything you do, you got to put some love in it. You know, if I didn't love the sport of basketball, I didn't love coaching, I wouldn't be doing it. You know, do something you love to do because of, because again, it's not a job then. It becomes a hobby. And everybody loves hobby. Everybody in this world have a hobby. You know, so basketball is a hobby to me. Coaching is a hobby to me. I enjoy doing it. I've done, I've coached it for free. And now I'm coaching it and getting paid for it.   Being confident. You got to be confident in yourself as a coach. And when I say confident, I'm saying, you know, you're going to hear things from the outside, you're going to hear things from the inside, but you got to be confident and you got to trust the process, your vision, your values, your goals, what you have as a coach and what you have for your team. You got to be confident in it.  Just got to be confident and be strong and just keep it moving forward. I definitely know I need help with my patients. I snap out of my brother and he makes me back. Oh, yeah. All right. So, next question is, who inspired you to do better in life? Who gave you that push in order to work as hard as you have, you know, from, I guess you could say, high school to where you are now? Wow. Definitely have had several people play an important role in the man I am today. But if I could really pinpoint and stick my hand on one thing, it would be my mom. 7th grade here, you follow city schools in that more middle school. A tragic happened in my life and I was in 7th grade, my mom passed away. So, when that day struck, you know, there's some things that I don't want to do. I want to quit in life. You know, I want to stop going to school. I was just pressed. I was sad and I was mad. You know, mad at God, you know, for taking my mom away with a young age. I didn't know where to turn, didn't know where to go. But I met a young man by the name of Corey Jones, Boys and Girls Club director, who was very influential in my life and kind of steered me, kind of guide me. You know, I was a member, I was 13 years old, and you follow Boys and Girls Club. And he, you know, he poured into me, the staff there poured into me and that became a turning point in my life. And along with that staff and just thinking about my mom, that gave me motivation to, you know, if mom was here, she would want me to do it this way. If mom was here, she would want me to be successful. If mom was here, she would want me to graduate and go on the college. So, you know, it was tough and difficult that mom wasn't there for my 8th grade graduation. It was difficult that mom wasn't there for my high school diploma. It was difficult that mom wasn't there for my college degree. So every degree that I get, I dedicate this for my mom. Every degree that I get is for my mom. It's my motivation to do what I love to do. So, you know, she would definitely be along with Mr. Corey Jones and the You Follow Boys and Girls Club staff definitely pay the role in my life to be where I am today. I get what she's saying. I get it. Because my grandma, she died in February and I was there. It just happened. But, you know, got to pick yourself up.  But all that couch, you better keep working. Plus, yeah, Mr. Corey, you know, Mr. Corey. Because I had to end up taking care of raising me after mom left. Oh, yeah. Oh, yeah, that's what he said. I know. But yeah, Mr. Corey, I know he's helped a lot of people. He's very wise. Especially when he was at the Boys and Girls Club. Oh, yeah, most definitely. All right. So, last question. Oh, yeah. Okay. All right. So, who do you think is going to be the hardest team you're going to face this season? The hardest team? I'll say this right here. The most, I don't feel like nobody's hard for us. I feel like it's a challenge. I feel like confident that we're the better team this year. And that goes back to being confident when I told you before. So I said the most challenging teams will be, I will say, I'm going to some tournaments that's in Hoover, Alabama, so I'm playing some really good teams up there. Also going down to Mobile, Alabama. I'm going to play some 6 and 7-8 schools down there. So those games will be challenging. Also, we're moving to a different area this year with three new teams. So, again, that area will be a little bit challenging for us. But again, I'm the type of coach. Hey, I respect all fear none. So, you know, we're going to take it one game at a time. And hey, the results will speak for itself. Hey, you're right. You're right. You guys got a good team. Appreciate it. Yeah, I could see y'all been working hard. I can't wait to see y'all progress and see the record y'all hold. You know, got to get that championship. Oh, yeah. It's been a long time coming. Well, Coach Clyde, that was the end of questions. I appreciate you coming on here, speaking your mind and answering these questions. I liked all these answers. Very inspirational. And just before you, and before you leave, what would you like to, I don't know, as an inspiration to kids? What would you like to leave as an inspiration for kids going through probably what you've been through or just going through life in general? Gotcha. Well, I'll say this right here. The best motivation that I can give to a kid, the best encouragement is everyone can succeed in their own way. You know, and that's something that model everyone can succeed.  Whether you're working in a janitor in the lunchroom, you be that best lunchroom.  If you're a lawyer, you be that best lawyer. If you're the doctor, you be that best doctor. Everybody has their own niche where they can succeed at it. So be good at it.  There's going to be obstacles. There's going to be adversity. Everybody has, I don't know, one person in life that hasn't been through any speed bumps or adversity or obstacles. The question is, how are you going to respond to it? Always have a plan, because guess what? Your plan A is not going to always work. If plan A was going to work with me, I'll be in the NBA. That's everybody's dream. So NFL play, NBA, you know, so you got to have a plan B. What's your plan B? How are you going to respond to adversity when plan A don't work? You can't just give up. If I didn't have a plan B, then my plan A would be nothing. You know, again, when Travis struck with me, I had a plan B.   Again, how are you going to, that it goes back to that word adversity. That means a lot, adversity. How are you going to handle adversity? So, and we have a hashtag. I say here with my team, my basketball girl is pound outwork everybody, pound outwork everybody. If you're in the classroom, outwork your classmate. Hey, if you're on the football field, basketball court, if you're on the baseball field, outwork your teammate.  When I was in college, I would get up at five, five, 30 every morning because I wanted to beat the sun up. I believe that, you know, I believe that people, if you want to be successful, you got to work in the morning and get a head start as a sleep.  A lot of times people like to sleep in and call the students, nine, 10, 11, 12 o'clock. Hey, I'm up at five and six a.m. in the morning. I'm working, doing schoolwork. I'm writing down my goals. I'm writing down my visions. You got to work when everybody else is asleep. So, you got to pound outwork everybody in every aspect of your life. I know what you're saying. I woke up at, I wake up every day around six. I woke up today at six, 13. I get to work every morning around seven a.m. I wake up about five, 30, six o'clock. I don't set a long clock. My passion wakes me up. Me coaching basketball is my passion. I'm trying to believe, hey, I don't have to set a long clock. My passion wakes me up. What I love, I wake up every morning, love what I'm doing. You know, the moment that I don't love what I'm doing, that's the moment, hey, I'll give it up. But again, I don't set a long clock. My passion wakes me up. Again, I'm ready. I'm in that mode. I'm ready to go out working. To me, some people be like, I wake up at eight or nine o'clock. To me, I'll be like, dang, that's late. I don't like myself if I wake up at nine o'clock. It's just like wasting the day. I say, live life to the fullest. Wake up at six. Get your life started. And I think that's the end of it. Well, thanks, Coach Clyde, for your time.