I'm William Dennis and thank you for coming to my session on building a no-less Kubernetes platform. First, a little background about me. I'm a product manager in Google Cloud where I work on Kubernetes engine. In 2019, I co-founded GKE Autopilot and that's what I currently work on. This is a new mode of operation for GKE, which I'll be talking a bit about today. I'm a very big supporter of the Kubernetes open-source project and in 2017 I also co-founded the certified Kubernetes conformance program, which is still used today to ensure portability between vendors and distributions of Kubernetes. I'm also writing a book published by Manning called Kubernetes for Developers. A little bit about this session, I hope it's going to be useful for all of you as you build or build on Kubernetes platforms of your own. In this session I plan to give you a behind-the-scenes look of the creation of GKE Autopilot, which is a fully managed platform for Kubernetes by my team. I'll be giving throughout the presentation some arguments for and against the idea of having nodes in a so-called node-less platform. At the end I'll talk a little bit about some of the future possibilities that can be enabled by this design. Firstly, let me do a quick audience poll. I'd like to know, raise your hand, who believes that it's possible to have a node-less Kubernetes where there is still technically a node object? Raise your hands. Got a couple? Not too many? Who doesn't? Everyone else? Yeah? And third option, who believes I was just adding a bit of controversy to my KubeCon talk to spruce up the marketing a bit? All right. Well, the truth probably lies somewhere in the middle of those three. But let's dig in. So what does, as we look at building a fully managed Kubernetes platform, I think it's worth taking a step back and looking at a traditional Kubernetes platform. And I'll be using GKE as the example here. So a traditional Kubernetes platform basically consists of two APIs that you need to interact with as the developer. The first one above is the Kubernetes API. That's what you're here for, right? That's what you're here at KubeCon for. That's what you use Kubernetes for.  You want to maybe represent a Redis database or a MariaDB database as a stateful set. You want to create jobs for your workloads, et cetera. That's what you're here for. And then you have this other API under the line, which is the platform API, whether it's GKE or one of the others. And with that API, you have to configure the cluster in order to serve those Kubernetes objects. You might have to create nodes of a certain size or with certain capabilities. I believe that an ideal fully managed Kubernetes platform would just be the Kubernetes API, where you interact everything at that level using, you know, KubeCTL using YAML files, basically. So what does it mean to build a fully managed Kubernetes platform? Well, when I was looking at this problem in 2018, I wrote a position paper and I had three headings. Let me just quickly share them here. The first one was that a node list or a fully managed Kubernetes platform should still be Kubernetes. Secondly, the containers should be able to utilize unused reserved capacity. In other words, allow for bursting. People come to Kubernetes. One of the reasons they come to Kubernetes is to be able to pool their resources and potentially burst when needed. And the third thing is I wanted to make sure that we price this in such a way that it supported continued usage.  And on node visibility, I had a suggestion just to make them visible, while still hiding certain bits like maybe the VM. I'll be digging into that in a bit. So back to that first point, though. The first point being that no less Kubernetes or fully managed Kubernetes should still be Kubernetes. Why is that important? So when we approach this problem of let's build a simpler to operate, a simpler to use Kubernetes, a lot of people came to me and said, well, William, Kubernetes is hard. There's a lot you have to learn to get started. There's a lot you need to do just to deploy an app. Maybe at the same time we should simplify that and create a cut down or a simplified experience so you can deploy things more easily. I think that would be a mistake. And the reason that would be a mistake is that it misses the point of why people were choosing Kubernetes as so successful. And I believe the reason is that it's an orchestration layer designed for professionals. These are professionals, like they might be running a massive website for a Fortune 500 company, for example. And they're professional, right? They need power. They need flexibility. One day someone might approach them and say, hey, I need to run a Redis database in this, a state for workload that you have to run. Kubernetes can handle that. I believe that the power behind Kubernetes is that flexibility and the scalability. So simplicity of the Kubernetes layer was a non-goal. Only simplicity of operating this cluster is the goal. Now, when the team looked at this, we had a couple of different operational models to consider when thinking about how to actually run pods, how to run pods on this fully managed product.  We'll provision all the infrastructure for you. So with that in mind, the team looked at a number of different models that were actually handling that compute. The first one was actually to use Borg. So I don't know how many people are familiar with Borg. It's a container orchestrator developed by Google and used internally and was the subject of an academic paper which you can find online. And it actually formed the inspiration for Kubernetes itself many years ago. So one of the options for running your pod containers for us would have been just to run them as so-called tasks or Borg. One of the advantages of that system is that it's massively multi-tenant and we would have like very rapid scaling and many other benefits. The second option was to reuse more of GKE and run each pod in its own VM. The benefit for that is that by being a lot closer to GKE it would be more compatible with various different infrastructure. The downside of that approach though would be that it would limit the pod sizes that we could offer in this platform to just a range of VMs that we have. You wouldn't be able to do something like a 5.25 CPU pod on this system because we don't have a VM of that size. The final option was just be like GKE, the existing product that we're looking at. Just be like GKE standard. Run multiple pods per node and basically don't change it too much. The benefit of that when we looked at it was that it would also then support other Kubernetes constructs that actually rely on nodes such as Damon sets, pod affinity, node affinity, things like that. In the end the team chose to make it just like GKE. One of the other nice things about this is that we were able to offer like a wide range of pod sizes scaling from a quarter of a core all the way to 28 cores in core-to-core increments. We'd be able to support something like a 17.75 vCPU pod without having to scale it up to a predetermined size. It also provides for maximum compatibility. That was one of the first foundational decisions we made on the path of building this product. The next question was around visibility. If we're trying to create a fully managed Kubernetes platform, one that's so-called nodeless, what does it mean when you go to kubectl get nodes? Should it list all the nodes that we actually have there? Should we just group that together in one and just return auto pilot node and just sort of hide all that detail? In the end we went with just list the nodes, be transparent, show the user what's actually happening under the hood even if they most of the time don't have to care about it. Second decision we looked at was what about the actual inner workings of those nodes? Should we be transparent about what shape those nodes are? Should we tell you if it's like an Intel or an AMD or this type of machine or that type of machine? Should we disclose how much of the allocatable table has been used? This was actually the subject of a lot of debate in the team. Some people were like, no, we should hide it, right? Because you shouldn't have to care about this, so why should you know about it? But in the end we actually went with full transparency. The reason is that we want you to trust us that we're going to do the right thing with your pods when you schedule them in this platform, but we're going to let you verify what we did as well. So if you want to poke under the covers and see exactly how things landed, it's all there fully transparent. I don't think we literally don't hide a single field compared to the GKE sound product. The other question was, can users access the VM object separately outside of Kubernetes? So at this point it's probably worth mentioning actually I showed a diagram before that developers use two APIs to interact with this system. It turns out there's actually a third one right at the bottom, which is the VM API. The reason I didn't show that before is typically people using Kubernetes don't have to care about the VM API. It's 100% managed for you, but it happens to be there and you can interact with it. For example, you can SSH into a node. So going back, one of the problems we had there was should we allow access to that object? Decision was no. In this case, we will completely eliminate that API since it's not necessary at all for the developer. So those were some of the key design decisions going into this. Now I'm going to cover a little bit about how we actually implemented this. Okay, so we built this product using various components that already existed in GKE. The way GKE works and the platform that we were dealing with is that nodes of the same configuration are grouped into a semantic grouping called a node pool. So for example, up on screen there, I have a node pool with eight VCPU cores and 16 gigabytes memory, and there's another one with four and four. And so what happens is if there's existing space on one of those nodes, the pod will just get placed. Okay, so let's say we have two panning pods here, one that can fit on one of these existing node pools and one that can't. So for the ones that can fit, there's a component called the cluster autoscala which will actually look at existing node pools and extend that node pool to add an extra node that can handle that pod and therefore that pod will be scheduled. In the case of the pod that didn't fit in any of the existing node pool definitions, we used a separate component that exists in GKE called the node auto provisioner and that component is capable of creating a new node pool definition, in this case one larger with 16 CPU 32 gigabytes memory in order for that pod to fit. But the final step is actually that cluster autoscala is then responsible for actually creating a node in that node pool to run that pod. So under the hood, that is what is happening, that is how we built it and how this system works and how it actually actuates on the user input is because we've pretty much eliminated the actual node API and the node pool API so there's no way for users to specify those things, we derive everything from the pod spec. So one of the simple examples is the resources needed, so things like the CPU and the memory, we derive that from the resource request of the pod. Another example would be node features and this is an interesting one. In the past, if you wanted to have a bunch of nodes with different features like for example you wanted to use spot compute, you would typically go ahead and create a node pool that was a spot node pool and then you would target that spot node pool with your pods. With autopilot, we flip the script there and you actually specify the requirement just in the pod spec and we will actuate on that and provision a node that can then handle that pod spec. Which I think is actually a really nice design because it means that all of the configuration of the hardware essentially, right, of hardware properties like this should be a spot node and in the future potentially other things like this pod needs a GPU, all this is done at a pod level, at a workload level, right where or the rest of your configuration is. You don't have to kind of do this multi-pass where you design your pods and you write those specs and you go and figure out the nodes that can run them. There are a couple of other additional components that we used I won't go into too much detail here but we used a component called release channels to keep them updated and node order repair to replace unhealthy nodes. Okay, so that's how we provision the resources to manage the pods. Another aspect of the implementation of this platform was an admission webhook. So we had to achieve two things here. I mentioned that we built the system to have a very wide range of resources for the pods but there were still some limits. The pod CPU needs to be between a quarter of a core and 28 cores and there's a ratio of CPU to memory. So what we do is we use a mutating webhook to look at the pod requests to ensure everything is within the range and if it's outside of the acceptable value we will actually just mutate the pod and fix it for you. When we do that we emit a warning if you're using cube CTL and we also write an annotation into the pod spec that basically logs what we changed so that you can kind of audit that. The second part is we have a validating admission controller and the validating admission controller is designed to enforce policies that prevent users from running admin level workloads on the nodes. Now why is that needed? The reason we need to restrict admin level workloads and basically preventing root access is that we want to offer a fully managed platform where Google SRE team basically responsible for running these nodes. What that means is we can't really offer you the users direct access to those nodes at a root level because then people can potentially go in modify the kernel, change bits and pieces and we essentially just lose the confidence to actually manage that thing for you because we don't know what's happened to the node. So it's important for us that all the nodes kind of look the same or at least have very well known properties so we have an admission controller to enforce those policies. What did we pick as our list of enforced policies? This is the list. Some of the simple ones are and I already kind of mentioned was limiting privilege pod. So this is a pod with the security context of privilege equals true. This basically gives you almost like root access on the node so we reject that. We also reject some Linux capabilities like sysadmin. Interestingly though, many Linux capabilities are actually still offered in this product so ptrace was actually requested by one of our security partners. They wanted to be able to use ptrace to inspect running processes. We looked at that, we felt like it was actually quite fine to offer so we added that into the list that you can actually use. Other things we clamped down on are stuff that directly relates to the node. The goal here is to build a node list product so we don't really want people using host port and running a container say on port 80 on the host because then if you try and schedule another container also with port 80 we can't curlicate that on the same node it kind of breaks our bin packing model and impacts the platform. So we had to limit that, the limited host networking which is also fairly highly privileged. Host path where we've also restricted although you can mount var logs in read only mode which means that you can actually as a user just have like a daemon set like scraping logs that's totally fine. As far as node affinity keys are concerned we restrict host name because again we don't want users thinking about nodes we don't want people targeting specific nodes but we allow many other node affinity keys like the zonal topology or regional topology things like that. One thing we didn't restrict was the ability to run a container as the root user. You can yourself restrict that using the open source pod security admission. We didn't actually need to restrict that because the security boundary of this product is still the VM. It's not actually a multi-tenant system at all. The VMs are still 100% your VMs, your nodes. So we didn't feel the need to actually restrict this and if we did restrict this from a usability perspective like half of all Docker images wouldn't run so that would be a problem too. So far as I've been talking about this implementation one of the interesting things is you could have actually done every single thing I've described yourself. You can use the cluster autoscaler you can use the node-order provisioner component you can write your own mutating webhook admission controller you can do all that you can literally build exactly what I just described yourselves today on GKE's data. So if that's the case why do we even need this other product? Aside from the fact that it's a bit challenging creating these mutating webhooks and so on. Well obviously one of the benefits of us doing it is that it's all preconfigured in a nice package but that alone is probably not enough. There's a couple of things that we add with this product that is sort of hard to do yourselves and that is the billing model is different it's request based so we charge based on the pod request rather than the nodes. Obviously that's not something that you can change as a user. Probably the biggest selling point is that by creating a fully managed platform with nodes in a known condition for the first time we were actually able to add node SRE under this product. So how the traditional GKE Kubernetes platform works is the nodes were kind of a shared responsibility model. Google would take a lot of responsibility for it but the users would also be responsible and that was because people could go in with read access and change them and it was sort of hard for us to know what had changed. So with this product since we've eliminated that we can actually for the first time offer SRE. Essentially that's kind of the bargain that you have when you use this product. If you're willing to give up that little bit of control about not being able to modify the node which I think hopefully for most workloads it's totally fine to give up. If you're running MariaDB you shouldn't have root access. What you get in return is a more fully managed system with us being in the SRE. The other thing is that we, as I mentioned we eliminated completely the VM API and there is no visibility on these VMs. The VMs are actually still there. They're exactly the same. They have a different prefix. So if you look at the cube CTL get node output you might notice that the autopilot node has the prefix GK3 instead of GKE. The virtual machines are actually still there they're still there in the product but the API is removed. This has the benefit for users that particularly security conscious ones they don't have to worry about things like SSH because there is no SSH into these nodes so it's like more lockdown. That's another advantage of the product. Okay, so I covered the design and the implementation of how we built it. Let's look a little bit now at the result. Where did we land? What does this look like? So at the beginning I showed this diagram of the two APIs essentially users have to interact with to get Kubernetes in order to use Kubernetes and even this diagram with that extra VM API at the bottom which you typically don't have to use but it's there and you might have to care that it's there. So with autopilot what we're able to achieve is we essentially shrink the entire GKE API surface area down to one command which is create. You just create it, you connect it to cube CTL from that point on you are 100% interacting using the Kubernetes API. So it's really a kind of a... I like to think of it as a very pure Kubernetes platform. The API you're using on this platform is just the Kubernetes API. There's no node pool API, you don't have to configure scaling, auto-scaling, anything like that. You just interact with it through Kubernetes. From a UI standpoint, obviously the UI representation of that API is also pretty nice. There's just three fields and you can create basically a production-grade cluster that builds is an arbitrary name. Okay. Now for the meat of this talk I guess. What are the benefits realized from all this? From this design of building this fully managed platform which actually looks a lot like the traditional GKE, right? It has pretty much the same nodes under the hood, the same multiple pods per node, a lot of the same things. Other than the fact that that potentially made it a little bit faster for us to build, what's the benefit to the user? What's the benefit of this design? And I hope this is relevant to all of you, particularly if you're also building your own Kubernetes platforms, maybe this might be of interest as well. I think the first benefit is that it enables really granular pod sizes. Because we're bin packing pods onto machines, we don't have to shoehorn the pods into VM sizes. So I did cover that earlier, but you can basically create like a 21.25-core pod and just slot it in there. We'll run that just fine. And then you can add a quart of a core and another one core, whatever you want to do. It's very, very flexible. And I think that matches what users want. Other nice things about keeping the node object or the node scheduling concept in this design is that things like pod affinity and anti-affinity continue to work. These are important concepts that come from the Kubernetes API. And if you remember back to my original proposal, I really wanted this to look like and be a fully capable Kubernetes platform. Well, it's not fully capable if you eliminate things like pod affinity and affinity. Pod affinity might be used, for example, if you have like a front-end pod and a back-end pod. And you want to say, I want these pods to always be together on the node. Well, it's hard to do that if you don't have nodes in a fully node-less system. You can't do that. This actually works quite fine with our design. We will ensure that that constraint is satisfied. Similar story with node affinity. So we offer zonal affinity. So if you have a zonal resource and you want your pods in that same zone, you can use that. One slide. I guess drawback of this system is occasionally you might want to separate workloads in one pod per node system. You don't have to separate them because they're always separated. But for that, you can actually just use the Kubernetes constructs of tolerations and node selection. So again, the Kubernetes API already has the language, the syntax to describe these things, and we can honour that in this managed product that still has nodes. Pod spread topologies work. And the last one here is Damon sets. Damon sets is often overlooked, I think, when it comes to fully managed platforms. But they're actually really important. Damon sets don't make much sense if you only have one pod per node because the whole point is you want to run an agent on the node. The good thing is with this proposal, with this design, you can still have a Damon set because there are still nodes. With one catch. And the catch is that Damon sets are typically used to actually modify the node. So how does that work in this system? We have, on one hand, it has nodes so you can theoretically have a Damon set, but on the other hand, we limited some of the administrative functionality which means some of the use cases of that Damon set no longer apply. The compromise we reached here is that we looked at well-known products and solutions out there in the community, out there commercially available, and we decided to allow this specific solution so they could continue to work on autopilot. So many security, logging and monitoring solutions that are commercially available, like all the people you see out there on the booth floor. Most of those solutions still work and if they don't work, they can probably come and chat to me and we can make it work. The reason we could do that is, as I said earlier, the security boundary for this product remains the VM. So even though by allowing certain partner workloads to have elevated privileges, the security boundary remains the same. It's still the VM. This is not a multi-tenant product, so we didn't have that concern. The main concern when we were building this from a design perspective was we had to make sure those nodes are supportable. We didn't just want to open the floodgates and let all kind of node modifications happen, but if it's a known set and professionally supported software, we felt like that was safe enough to offer from a supportability point of view. So it still allows daemon sets, and I think that's important because if you have a workload that you're running on-prem or you're running on another cloud provider or on GKE standard and you want to migrate it around, you probably have, in fact, most of our customers have at least one daemon set workload they just want to run. If we didn't offer this, essentially you would have to go in and add that whether it's security or logging or whatever. You would have to add that functionality to every single pod, which I think is a really big burden on developers. So that is actually one of the really overlooked benefits I think of still supporting nodes and still offering a multi-pod per node system. Other benefits, and this mostly kind of helps us as we work on the product further, but other benefits of this design are the fact that the infrastructure features, it's very similar to GKE, so a lot of things just work. One really good example is stateful set. So right out of the gate, on day one we were able to support persistent volume claims using block storage resources, which means you can run MariaDB, Redis, any other stateful workload just works because it's the same infrastructure. We didn't have to reinvent the wheel. When we look at adding additional features, things like new machine types, maybe if you follow the Google Cloud naming pattern, like N2 or C2, things like the highly optimized compute machines, stuff like that is actually going to be a lot easier for us to add because we chose this model because it's basically sharing that same infrastructure. Hardware like GPU, local SSD, again because these are just regular VMs under the hood, it's a lot simpler for the team to add support for those features. And finally it's possible to offer burstable class pods. So I mentioned this right at the very beginning as one of my three kind of objectives, and that was the ability to utilize unused capacity in the class to buy pods. Why do I think this is important? The reason I think this is important is that if you have like three or four pods all running on a node, it's likely that at some point some of them are completely idle. Let's say it's a web-serving application. You may have three pods that are completely idle. Now if a request comes in to one of those pods, you ideally don't want to constrain it to just the resources that that pod requested. Now certainly that pod needs to be able to handle that request within whatever SLO you have with the resources it requested, but wouldn't it be nice if opportunistically it could just burst, use the capacity and serve that request a little bit faster, delight the user a little bit more. I feel like that's a really important feature, and it is possible with this design. It's really only possible with this design because if you don't have multiple pods on the node, if you don't have this concept of the node, then you can't really pull that capacity for bursting. We don't actually offer this yet in the product that we built, but it's something that we're looking at, and I'd like to share a quick kind of design of how this might work. So if you look at guaranteed class pods today, the CPU requests equal the limits, which means there's no bursting. Traditionally in Kubernetes, you could set a much higher limit than the request and thus create a burstable pod that can scale up when there's extra resources. The problem with the design that we came up with is take this example of three pods that totally have requested five and a half cores and that are running on an eight-core machine. The problem is in a traditional Kubernetes model, those pods would be able to use up to eight cores, which could be a bit of a problem for us. It might allow for some gaming of the system where you could try and convince autopilot to create like an eight-core node, run a one-core workload on it, try and prevent other pods getting on there using some technique, which is potentially possible, and thus kind of get like 8X your resources. That would be a problem for us. So the ideal design would be, well, what if the user could burst within the paid-for capacity on the node at any given time? So if we clamp the bursting, in this case, to five and a half cores, then we would be giving you, you know, the capacity that you're paying for, which would be ideal. The only problem is it turns out that's a bit hard to do with the way Linux, the completely fair schedule that works in Linux today, it's a little bit hard to do that. So one idea that we're actively looking at is to potentially kind of round up the nearest integer number of cores by just turning off the unused cores and allowing full bursting within, in this case, six cores. That's just an idea we have. Stay tuned for that. What about the downsides? I did mention that there would be pros and cons. A couple of the downsides, I believe, of the design that we came up with here is that there is a potential for allocatable inefficiency in this system, right? It's possible if the user is creating and deleting a lot of pods of all different shapes and sizes that you could end up with very much underutilized nodes and that requires the team to build additional features like defrag to kind of correct that. So that's a bit of extra work that we end up taking. It doesn't affect the user. The user doesn't care about that, really. It's kind of more just a problem for the infrastructure platform to solve. The other potential issue, obviously, by running two containers, two pods on the one node, you can potentially have resource contention. Although we do have a way to solve that for users, they can still separate the workloads when needed. And a couple of downsides of using a multi-single tenant platform, like we do, as opposed to a fully multi-tenant platform, is that it's hard for us to add hot standby capacity. So one of the really nice benefits of a multi-tenant system is you typically have a massive resource pool shared by everyone. You can just... Individuals can scale up and down very quickly. We don't have that ability with this design, so that is one drawback. Adding a new pod, if it needs a new node, can take between 60 and 80 seconds. And there's a little bit of greater operational complexity on the platform side.  But again, most of these downsides, by the way, kind of just a burden on us, I think just makes a couple of different problems a little bit harder to solve, but hopefully it's solvable. Okay, so in summary, and this is the takeaway here, I guess, that I'd like to try and convince you, hope I did convince you of this, is this node-less and does that even matter? Well, when I started this project back in 2018, node-less was kind of synonymous with fully managed Kubernetes. And I guess maybe the point I'm trying to make here is that it's not a good synonym. I maintain that this design is operationally node-less. It's node-less in the sense that you don't care about the nodes, you don't have to think about them. But I do believe there is a benefit, as I've hopefully outlined, of having nodes existing as a scheduling concept when needed and when relevant. So that was our journey. I hope that was useful. Yeah, I hope it's of interest to learn how the team went about and built this thing. Maybe it can inspire you as you're building your own products and services with Kubernetes. And with that, I'd love to take any questions. We have a couple of minutes. I'd be happy to continue the debate on Twitter, over a beer or however you want to do it. There are a couple of microphones. If anyone has a question, let me know. Thank you. Question? Can we have the mic go live, please? Testing. Okay, lovely. Hi, I'm the moderator. I'll be doing questions online if there aren't any, but there aren't any at this point. I just wanted to let you know that we have some time until our schedule in time, but we'll do a couple of questions. We can go just a few minutes over. Okay, sounds great. Please, take it away. Well, I hope you can hear me. Well, the company I work with is subject to a lot of audits, and they require us to split nodes in different subnets in order to segregate them on a network level. And also, they require us to install certain software and binaries on these nodes. Basically, we have to be in control of the machine image that we use to spin on the nodes. And I wanted to ask if this is capable, if you are capable of doing both of these things, actually. I don't know if the subnet separation thing is possible. That's something I'd have to probably take offline and do a deep dive with you. With the security components that you're running, I would like to think that it is possible. We do work with a lot of security partners to make sure that their solutions work. So unless it's like a homegrown, home-built container, I would say the answer is probably yes. It either works or can be made to work with this system. The other thing, by the way, is as you go to these auditors, hopefully you can position this, and we would help you, right? Position this as a more secure platform to begin with, right? Because things like SSH are completely eliminated from the nodes as well. I don't know about that because they were pretty reluctant of allowing us to go to the cloud in the first place. I'm not really sure how cool they will be with that. But yeah, then the other question that spikes up is, RDAV, for example, softwares, enabled through demon sets, or is that the way you do it? Which software, sorry? For example, antivirus softwares. Right, yes, you would install that with a demon set. All right. Provided if the demon set is using privilege, which it probably is as a virus scanner, it would need to be specifically allowed by my team, but we've already done about, I think, eight or nine solutions and there's room to grow that. So yes, it would be through a demon set. All right, thank you. And apart all the good things or benefits of the DKI autopilot, we are active users of, happy users of the DKI autopilot, but the restrictions, we come up with not being able to deploy things like secret store and these kind of drivers. Do you have any plan to overcome those issues that implicitly these kinds of designs imposes? Because, yeah, we want to use both, for example, for secrets and we weren't able to do that by using standard CSI drivers instead of creating direct connections to those both clusters and so on. Okay. I believe Vault might actually work now because I know I've tested it and I think I got it working, unless you're using a different configuration. Your own, let's say, the Google Cloud Platform GitHub repository has one secret store and there is a limitation of host path and privilege to actively being used. Yeah, I think when it comes to these restrictions, basically we have to either, if it's a kind of a partner, like a well-known container, a well-known workload, we can potentially allow list it. The other option we have is, if it's like a technique we have to enable, we essentially have to kind of productize it. So if you need a driver, it might be the case where we just have to offer the driver as a feature, where you can just turn on that driver. We should connect and continue this conversation, but I do believe Vault should actually work. I know when we first launched the product, we didn't offer mutating webhook support, which broke like half of the types of workloads, like Vault and things like that, which need to mutate workloads. We did add that in recently, so that was a restriction that we never really designed to be in the product. It was kind of like collateral damage. Okay. So we fixed that. So yeah, I do believe that Vault should actually work in particular, but we should follow up. We dropped you direct. Please, please. All right. Thank you. Cheers. All right. Last question, I think, then we might have to wrap. Yeah. Hi. Quick one. Yeah. I saw a couple of times the restriction on CPU increment being 0.25. Is that on purpose? And if so, why? Good question. It's a decision I think we should actually revisit. The plan was to start a little bit conservative, I guess, because it's much easier to relax these restrictions over time. And I believe that the theory was, and I was part of the decision, and I'm still trying to remember kind of like, why did we actually do that? I think it was just so that when we packed the pods on the nodes that are kind of, if they're known kind of sizes, like the little Tetris kind of diagram, we can kind of slot it in a little bit better. In hindsight, I'm not convinced it's actually needed, to be honest. So yeah, we might take another look at that. Yeah, I mean, would you like to just have like, just anything, like between, we probably need a minimum, but yeah. All right. That was a thumbs up for the people online. Great. Well, great questions. Thanks a lot. Like I said, let's keep the conversation going too. Thank you.


The results of this study suggest that there is no significant association between the 308g per a polymorphism in the TNF-alpha gene and type 2 diabetes mellitus. This article was authored by Ali Hassanpour-Decorty, Majid Dusty, Faize Kiani, and others.


Tonight the DuPont Company brings you Builder of the Sioux starring William Holden and Marsha Hunt on the Cavalcade of America. But first, here's Gain Whitman. If your ski clothes and other winter sports garments no longer shed rain and snow because the water repellent finish came out of the cleaners, you'll appreciate the protection offered by garments made of fabric treated with DuPont Zeeland Durable Water Repellent. Zeeland continues to give protection even after many washings or cleanings. So look for the Zeeland tag when you buy jackets, ski clothes and outdoor sportswear. Zeeland is one of the DuPont Company's better things for better living through chemistry. And now William Holden as Charles Harvey and Marsha Hunt as Katie Burroughs in Builder of the Sioux on the DuPont Cavalcade of America. Nobody today remembers my name. It's Katie Burroughs. And only a few remember Charles Harvey, the man who, more than any other in the 19th century, changed the face of America. I can still recall the first time I saw him. It was Thanksgiving Day, 1852. Mother brought him in at dinner time. Katie, this is Mr. Harvey, our new boarder. Well, how do you do, Mr. Harvey? Sit down here, won't you please? Oh, thank you, Ms. Burroughs. This is Dr. Lash. How do you do, young man? Now don't you get sick, Mr. Harvey, because all you'll get from him is Calamel and Jalop. That is, if you can spare your time on his checker plane. Over here is Major Morris, our army engineer. How do you do? Now just help yourself to the turkey and cranberry sauce, Mr. Harvey. There's plenty of it. Oh, thank you, but I had my dinner on the boat, Ms. Burroughs. I would like a cup of coffee. I'll bring you some. Oh, Major Morris. Yes? Mr. Harvey says he's read about that canal survey you're making for Congress.  And potatoes, Major? Oh, are you an engineer, Mr. Harvey? No, no, I'm not, but I've read some on it. Look at Ms. Katie. Disappointment written all over her face. I should warn you, son, Ms. Katie has got a dream man. Some big strapping fellow who'll come along and build a canal between Lake Superior and Lake Huron just for her.  And you too, Major. But one of these days, you'll look up from your checkerboard and there it'll be, all dug and finished. You won't laugh so hard then. You see, I told you, she's hepped on that canal business. Ms. Katie, those surveys convinced me they might just as well try to build a canal across the moon. Why is that, Major Morris? Do you ever hear the big freeze, young fellow? A big freeze? From December to May, this suponentular is frozen tight as a drum. Nobody can get in or out except on snowshoes and the nearest machine shop is 200 miles away. You let the right man come along, Major, and he wouldn't stop for that. Yeah, there she goes again. Major, reach me some of that turkey, will you? Do you know what a canal would do to this wilderness, Mr. Harvey? Why, it would open up the whole west all the way from Lake Superior to the Atlantic Ocean. But every time I mention it, they laugh at me. I know how you feel, Ms. Burroughs. All my life, I wanted to build things, and everybody laughed at me, too. All his life? How old are you now, young fellow? 20? 21? Oh, no, no, I'm older than that. I'm 23. 23, he says. A gray beard, he's 23. Ah, what's your business, anyhow? You wouldn't be another schoolteacher, would you? I'm a traveling salesman, Dr. Lash. A traveling salesman? Oh, too bad, Ms. Katie. I reckon you'll have to wait for the next excursion boat for that dream man, yours. Mr. Harvey, I'd like to show you the Sioux country. I know every inch of it. We could go now if you want to. Well, I'd like that, Ms. Burroughs. I'd like it very much. Come on. Here's your coffee, Mr. Harvey. Oh, see the force, will you, Ma? We'll be back before supper. Well, I never. Where are they going? Can't you guess, Ma Burroughs? They're going to build the Sioux canal. Ms. Tilt, you can see the Sioux St. Marie river in the rapids.  All right, let's go. Look out for those briars. You know, this country makes me think of home. When I was a kid, I was dying to be an engineer, and I used to build bridges across the creek behind the house. Oh, real ones? Bridges you could walk on? Well, the first time everybody laughed and said it would collapse. And did it? So I got hold of old Blossom. She was our coward, took her across. And after that, I suppose everybody used it. Yes, but that didn't keep them from laughing just the same the next time. I suppose not. Well, here we are. There's the river. You see that tiny stretch down there? Oh, yes, I see. Those are the rapids, only a mile long. That's where they'd have to build the canal. Ms. Burroughs, I have a confession to make to you. The canal is one of the things I came up here to look into. Well, from the way you talked at the table, I thought as much. But back east, everybody says it can't be done. They said that about your bridge, too, didn't they? Ms. Burroughs, I'll tell you something else. You know, you're just the kind of person I wanted to meet him. Well, if I thought that canal was possible... I know it is. Well, I might get backing from Mr. Fairbanks, my boss. Then why don't you write him? Tell him about it. Well, what could I tell him? Well, tell him the company that built that canal has the right to claim the greatest ore-bearing land in the world. Oh, now, wait a minute. I'm not an engineer, Ms. Burroughs. You took Blossom across that bridge, didn't you? Yes, but that... Did she care whether you were an engineer or not? All she asked was that the bridge didn't collapse.  Look, Mr. Harvey, why don't you try it? It's a tempting idea, all right, but... well, I hardly know where to start. Well, first, you have to go before the Michigan State Assembly. Do you think they'd give me a grant? Why not? They want the canal and you want to build it. I want to build it more than... more than anything I ever wanted to build in my life. All right. Then the next step is budgets. Will you help me with those? Of course, I'd love to. Machinery, labor, stone. Uh-oh, stone. That's a big problem. Everybody says you'll have to haul it in from the east. But this stone right here, I'll show it to you. Look, the whole top of this hill is limestone, solid limestone. Limestone? Well, that means we can keep the budget down. Oh, this is wonderful. You don't know what it means to me. Oh, Miss Katie, I... you're so... you're so... Yes, Mr. Harvey? Miss Katie, I... I think it's... I think it's going to snow. I... I think there's a blizzard coming. We'd better hurry home. The State Assembly met in January and Charles Harvey was off to Lansing. Then, for six dreary weeks, I heard nothing. To one day, a trapper brought a letter in by Snowshoe. It said... Dear Miss Katie. Miss Katie. I think the Assembly liked our plans for the canal. Our plans? But opposition arose from Major Morris. The project is completely impractical. Congress has allotted the State of Michigan 775,000 acres of public land to be awarded the construction company on the condition that the canal is completed in 10 years. 10 years. Now, Mr. Harvey proposes to build locks 350 feet long, though the longest vessel sailing the Great Lakes measures less than 200 feet. It's a pity that Mr. Harvey's not an engineer. If he were, he would know that the construction he plans would take at least 20 years. In fact, as an engineer, I don't believe you'll ever get it done. That's all I have to say. Mr. Chairman, may I... Yes, Mr. Harvey. Quiet, please. Mr. Harvey has the floor. Gentlemen, I will sign a proviso that my canal must be completed within two years. If I fail... If I fail, the loss will be solely my own and that of my backers. The State of Michigan will still have eight years left in which to authorize a smaller canal built by anyone who wants to. Major Morris, do you consider that fair? I suppose so, since there is no other builder at the moment. Then, gentlemen, if there's no objection, I propose that authorization with a two-year proviso be granted, Mr. Harvey. Objection? Mr. Harvey, the opportunity for a canal is yours. And though I don't see how you'll do it, I hope you win, my boy. Then, for months on end, no word, no sign from Charles Harvey. And for me, the winter days seemed suddenly empty and dreary. Even Mother noticed at a diniton. Casey. Oh, yes, Ma. I thought you were helping me set the table. Oh, I'm sorry, Ma. Katie, your mind's wandering. What are you thinking about? Oh, nothing, Ma. It wouldn't be about that Charlie Harvey, would it? I was remembering a walk we took just before he left. He was just as shy as the first day I met him. Once he turned to me and for a minute, I thought he was finally going to say something. What did he say? Miss Katie. Just Miss Katie. Katie, is it the canal you're interested in or Mr. Harvey? Oh, Ma, it was so much fun working with him. Those walks we took together and the nights we sat up, planning and figuring. You're not falling in love with him, are you, dear? I don't know. What if I am? Because it won't do you much good, honey. What do you mean, Ma? Katie, that boy has only one interest, the canal. There's no room for you in his life. But he might at least have written me. Just don't count on him, dear. That's all I mean. Try not to care too much. Please, Ma, I'd rather not talk about it. Forget him, dear. If he ever does come back, he'll finish that canal and that's the last you'll see of him. What, Katie, honey, what's the matter? Come here to me. Leave me alone, kid. She says all I want is leave me alone. Spring that year did not really come till June 1st. And that was the day Charles Harvey returned to the Sioux with the first boatload of men to build the canal.  One day in August, when I was down to talk again... What happened? The scaffold of the coffer dam. It collapsed. Oh, no! Fire. Fire, how did it happen? Are you hurt? I'm all right. But there's somebody under those timbers. Here, you men. Let that beam. It's Johnny Cassidy, sir. Johnny. Johnny, where are you hurt? Seems to be my side, sir. Now, don't try to move. Let me see it first. I feel like I can't hardly breathe anymore. Give me your knife, Ira. I'm going to cut this shirt away. Katie, get Dr. Lash at once. Tell him to hurry. Ribs are broke. His whole side's crushed. It looks awful, Dr. Lash. I think I'll move this king over here. It's your move, Ben. Watch yourself. Dr. Lash? What? Oh, please. Johnny Cassidy's in terrible pain. Oh, a little pain won't hurt a man. Go ahead, Ben. I'm waiting. But you've got to hurry. You know, Ben, ever since that traveling salesman pup came up here and started his canal, everybody's got to hurry. Hurry. Listen to me. There was a time, Ben, when you could wait till the spring thoughts came to stake yourself acclaim a land. Now, if you don't sneak off in a roar and blizzard, you'll find somebody from the canal company's been ahead of you and stole it. Wait a minute. I don't care what you think of Mr. Harvey or me, but Johnny Cassidy's dying. Do you hear he's dying? You've got to come and help him. What you do, Miss Katie? You go to my medicine cabinet and take that man a dose of Calamol and Jalop. That's as good as anything for, what is it you say, his whole side's crushed? You'll move, Ben. You were listening to William Holden as Charles Harvey and Marsha Hunt as Katie Boroughs in Builder of the Sioux on the Cavalcade of America sponsored by the DuPont Company, maker of better things for better living through chemistry. When Charles Harvey came to the little frontier settlement of Sioux St. Marie, Michigan in 1852 and talked of building a canal that would open up the Great Lakes to the whole country, I felt that here was a man with the courage to see his ideas through. He accepted the challenge to complete the canal within two years, but trouble arose when a collapsing scaffold crushed a workman and Dr. Lash refused to leave a checker game to help him. Early the next morning. So they're leaving, are they? How many? At least a hundred, Mr. Harvey. They said they won't stay on a job where there isn't a doctor they can count on. I've sent for a doctor from Detroit. I'm going to keep him here. Oh, I wonder what Dr. Lash will say. He's no doctor, Miss Katie. What do you mean?  Has no diploma, no medical degree, nothing. I checked up on him last night. Oh, no. If Johnny Cassidy dies... Is he feeling any better? Come in. Oh, hello, Ira. Good morning. I better go see if I can do anything for Johnny. Excuse me, Ira. Yes, sir. You know, it's the timber crew, Mr. Harvey. We've run into trouble. More trouble? What kind?  Well, what's wrong with that? Land belongs to us. Yeah, but to get there, we go up a road through an unclaimed strip.  No trespass. What? We started snagging those trees aside to take our cutters through. Well, just then, Dr. Lash came out of the woods with a gang of saloon tufts. Dr. Lash, huh? Yes, sir. He said that land belongs to him. Since when? Since yesterday. Said he put a claim in for it. Now, what difference does that make? We've got the right of way through that strip. Well, I told him that, but he yelled, you tell that salesman pup to keep his men off my land. Oh, he did, did he? Well, sir, I didn't want to pick no fight. We've got trouble enough in this camp. Does he think I'll let a quack like this? Listen, Mr. Harvey, you start fighting the townspeople and you'll lose what men you have left. Mr. Harvey, Johnny Cassidy, he just died. Ira, go down to the camp and call the men together. Tell them I want to talk to them. Men, I've called you together to tell you that Johnny Cassidy's dead. He died because Dr. Lash couldn't be bothered to help him.  I've got a good man with a medical diploma coming here from Detroit. He's bringing hospitals and equipment with him. But I want you to know that all these years while Lash has been defrauding you, he's kept a real doctor out of sue. Now, wait a minute. None of that. I'm going down to talk to him now and I think when I get through, we won't see Mr. Lash around here anymore. Winter came and the short days crackled with frost. Every foot of canal had to be wrenched by brute force from the frozen land. I don't know how we ever lived through that year, but by fall of 1854 the big locks were nearly finished and in October the canal company sent an engineer to inspect them. Harvey, who made these topographical surveys? Why, those are the original army maps, Mr. Brooks. Major Morris made them with Congress back in 1852, I believe. Why? Well, they're wrong. Dead wrong. He must have made them at flood level. Let me see. Look there.  Do you know what this means? They're locks, boy. They're too shallow. 11 feet of water instead of 12. But he's supposed to be an engineer. You'd at least think he'd know how to measure water. You know, boy, I think maybe we can get away with 11 feet. And after all the mistakes, not yours. It's Congress's. You're not legally responsible. No, no, no. I can't do that. Contract calls for 12 feet, Mr. Brooks. I'm going to tear those bottoms out and make them one foot deeper. But Harvey, think of the cost. It's a lot of money in our budget, aren't we? You haven't time. The first ship has to go through that canal in the middle of May. You start ripping those locks up now and you'll ruin us, Harvey. Mr. Brooks. My contract says 12 feet of water. And that's what it's going to be. Every foot of that canal had to be deepened. In February, a diver was sent down to explore an obstruction in the lake channel. Major Morris' map showed it as a sandbar. The diver said... Sandbar? Mr. Harvey, that's a rock ledge. Six foot thick and an acre wide. In desperation, he set out on sleigh to consult engineers in Saginaw. Harvey, the only solution we can see is another cofferdam. That'll cost money. Lots of money. How much? At least 100,000. All right. We'll do it. When can you start? Start? Not until the April 4th, Mr. Harvey. Well, how long will it take? An acre of six foot rock and ten foot of water. I figure that'll take you at least a year. He looked like a beaten man now. Night after night, I could see the lamp light in his office window. Day after day, I could see the lines deep in his face. And one night, I decided to do something about it. Mr. Harvey? Mr. Harvey? Is that you, Miss Katie? What are you doing in your office at two in the morning? I couldn't sleep. There's a moon outside. A walk would do you good. Come along, won't you? No, I've got to do some thinking, Miss Katie. I keep feeling there must be some way to crush that rock. Please come with me. We'll walk down to the river. No, no. I haven't time. There's only one month left. Oh, come on. The fresh air will help you think. Well, all right. I'll go with you. I don't know when I've seen the moon so bright. Yes, yes, isn't it?  The wind feels so soft tonight. Yes, yes it is. A big steel bar, say 20 or 30 tons, pointed to pierce that rock, but heavy enough to crush it. Look, there's the old steam ship whore. When I was a little girl, I used to love that place. Where do you get the steel bar? Can't hold 20 tons from Detroit on snowshoes. I used to come down here in the early spring to watch the ships when the ice broke out. Well, there must be some way. There must be. Wait a minute. Look, Mr. Harvey, you can't go on this way. You've done your best, but it's... It's just no use. The canal is doomed. Everybody says so. For your own sake. Please, give it up. I can't. I can't give it up. All right. Let's not talk about it now. Not on a night like this. It's too beautiful and quiet out here. Yes, the whole world seems to be asleep. Everybody but us. Look at that moonlight and those boats. Just lying there, waiting for the first spring thaws. Come on, let's go down to the edge. Wait a minute. What is it? Those boats. What's the matter? Suppose we borrow their propeller shafts. Propeller shafts? The first steamer from the east could bring them new ones. What for? I mean, lift the boats out of the ice and... What are you talking about? Steel bars to make the steam punch with. Oh, Miss Katie. Miss Katie, I... Yes, Charles? I can do it. I know I can. I can crush that rock. On May 10th, just nine days before the two years expired, the first boat went through the canal with 100 tons of iron ore. All Sioux St. Marie was there to cheer. Charles Hardy and I stood at the last lock gate. Captain Wells. Oh, hi there, Captain Wells on the Illinois. Hi there, Mr. Harvey. How are the locks, Captain Wells, on the channel? Mr. Harvey, we sailed right through like a fish. You built a good canal, the Sioux. I built. Captain, I want you to meet the real builder of the Sioux. Oh, Katie. Katie. Yes, Charles? Come over here. Captain Wells, you know this girl, don't you? Well, sure. Everybody knows Miss Katie. Well, it won't be Miss Katie Long. I want you to meet the future Mrs. Harvey. And that's the story of the building of the Sioux Canal. But it's much more than that. Yes, it's the story of men like Charles Harvey and women like Katie Burroughs who built the roads and canals and bridges of this country until finally a continent had been spanned. Yet never a bridge factory canal nor city was built, but at the outset someone had called it impossible. It's this enterprising spirit that we proudly call America. Now here's Gain Whitman. Rats cost you money, plenty of money.  Estimates put the figure as high as half a billion in the United States alone. And that's a lot of rats. On American farms, rats destroy as much food every year as 200,000 farmers can produce. For that matter, there are plenty of rats in any city. I heard not long ago of a fellow in one eastern city who thought he had four or five rats in his place of business. He put out bait, dusted with a rat poison, and in the morning he found 139 dead rats. Today, thanks to scientific research carried out during the war, there is a new poison that is unusually effective for brown rats. It's the one that killed the 139 rats I mentioned a moment ago. The new poison is called N2, spelled A-N-T-U. It may be used in the ordinary ways, but in addition, it may be used in a new way as a tracking poison. Perhaps you've never heard of tracking poisons. Here's how they work.  All you have to do is sprinkle a tracking poison where the rats will walk in it. They lick their feet, and that's the end of them. While N2 is a special poison with a common rat and is not as dangerous to other animals as most rat poisons, it is a poison and a powerful one. Pets and farm animals, including chickens and other fowl, must be kept away from it. And it goes without saying that children must be kept away from N2, as from any other poison. The DuPont Company supplies N2 to exterminators for professional application in commercial buildings and homes, and to compounders of trademarked rat poisons. If you'd like to know more about N2, write to the DuPont Company, Wilmington 98 Delaware. We'll send you, free of any charge, an illustrated 12-page booklet, How to Get Rid of Rats. Remember, just write to the DuPont Company, Wilmington 98 Delaware. And, too, Alpha Naphthyl Thyrouria is one of the newest of the DuPont Company's Better Things for Better Living, through chemistry. Next week, the DuPont Cavalcade brings you Basil Rathbone and A Chance for Jimmy, the story of Francis Henry Brown, a young doctor in Boston, who started the first children's hospital in America. Dr. Brown believed that children were not just small adults, but should be handled with special care in a medical center where children would be treated as children. A medical center where research could be carried on about their particular illnesses. So be sure to listen next Monday at the same time to A Chance for Jimmy, starring that outstanding actor, Basil Rathbone, in the Cavalcade of America. The music for tonight's DuPont's Cavalcade was composed by Arden Cornwell and conducted by Donald Borey. Our Cavalcade play was written by Paul Peters. William Holden will soon be seen in the Paramount Picture, Blaze of Noon. In the cast with William Holden and Marsha Hunt tonight were Bess McCameron as Mrs. Burroughs, Robert Dryden as Dr. Lash, House Jameson as Morris, and Cameron Proudham as Ira. This is Ted Pearson inviting you to listen next week to Basil Rathbone and A Chance for Jimmy on the Cavalcade of America, brought to you by the DuPont Company of Wilmington, Delaware. This is NBC, the National Broadcasting Company.


Not a lot of players can say that they play for the team they support so I guess that's kind of Makes me more passionate, you know to play for the club and stuff So it's definitely good to play for the club you support. Yeah, soccer has always been like a dream of mine. So Say that come true is kind of rewarding. Oh, yes.  It's great to get a reward for him especially with so much support Like the fans and biggest fans in the ALE So I'm just looking forward to next year. It's very close. You know to my house my home and everything my family and friends They all come to the games and watch so it'll be very special for me to play in front of them with all the work That they put in all the tactical, you know the advice and you know when you train with the first team as well All the players, you know they give you great advice and you know just training with them every day and stuff You realize how much it helps you get better stronger fitter and like just better with the ball, you know awareness and everything Yeah, definitely like Santa and Bridget and that they've helped me like in training whether it's like where to be what runs to make That kind of thing like they've just given me that helping hands. So yeah, it's good amazing the intensity is so high Like the players are so so good. They'll world-class players and just so so nice as well Just good people special moment for me, man special moment Yeah, just that from that feeling, you know that I got called up and you know just stepping out into the pitch it was just specials, you know felt rewarding and got a few touches on the ball and Yeah, hopefully more to come wouldn't say unexpected, but there was a lot of people that could have got it So I was pretty proud of myself to get that and see if I can keep taking that fall into the rest of the season Hopefully next year. Yeah, he's such a great player and to play with him and get that combination going That's helped my game as well. I'm just No, he's we're very close mates and everything and off the pitch as well and just on the pitch You know it helps me out so much and like we play well together We combine well and we just read each other. So that's why we've you know together We've been doing pretty well this year get on the field as much as possible play as many games as I can and score some goals for the club and Hopefully maybe get picked in the young soccerers team. So they're the goals for the next 12 months or so You know try and get as many elite games as I can you know play to the best of my ability Obviously get stronger as a player and hopefully you know get another two three years at this club