OK, well, we're about to meet out. Hey, everybody. Welcome to This Week in Cloud Native with this episode number five. Good to see you. Good to see you. I'm saying hello to somebody who's coming to say hello. Good to see you. Darkside21. Awesome. Welcome, welcome. All right, so if you're in the chat and you want to say hi, just drop a note into the chat. And I can highlight it, ask your questions, anything else like that. Just let me know. Happy to raise you up in this way. If you are here for the first time, one of the most important things to understand is that I keep a weekly notes document. This is a carryover from something I did in TGIK, which was, thank goodness, it's Kubernetes. And so I've created for that, I've created a team account. And let's just go ahead and check that out real quick. Let me set up my environment so you can see it. So this is the Week in Cloud Native. If you go to hackMD slash at TWICN, you're going to see this week's notes along with any previous notes that came earlier in the session. These notes are editable by you. If you sign in, then I can see who's actually putting content up. If you don't sign in, you won't be able to see it there. But if you do sign in, then you'll be able to edit the content directly. Just notice something I want to change here. Give me one moment. So this is where I keep my notes and everything else like that.  Hello. Good to see you. Welcome, welcome. So this will be an interesting session. But yeah, let's dig into it. So as you know, there's a ton of really great content on Cloud Native TV. I'm just one of the many shows. There are playlists for each of the content. So if you missed a show or you want to make sure that you catch a show, you can actually go check out the playlist. And all of our sessions are captured and then upstreamed into a YouTube playlist so you'd be able to see them later if you missed them. So don't worry about that. There's lots of content that happened. There was actually a really great CNCF face-off with Matt Stratton. It was actually, I can't remember what it was, like the Rogues Gallery versus, I can't remember what the other piece of it was, but it was a really funny show. You should definitely check it out. It was really a kick. Search magic with Siam, LGTM with Rockhood is doing OPA. And I think that Spotlight is also doing OPA. So lots of OPA love this week. If you're not familiar with OPA, it is the open policy agent. So there's lots going on there. Thank you, I'm doing great. Great to have you here. What else do we have? Oh, you know what I just realized? I wanted to change this real quick because I'm realizing it is redundant. There we go. Nope, that's not what I want. You get to watch me do this kind of stuff live. Perfect, there we go. So if you want to go to one out of our notes, you can see here in the bottom left-hand corner it says hackMD.atTwicken. That's where our notes are located. We have the cognitive classroom with Kat Kosgrove 100 days with Anias and Fields tested with Kazzlen. All of these great shows, definitely go check them out. One of the things I wanted to point out this week, there was a lot of hubbub of course on Twitter because that's what hubbub, that's what Twitter is for. It is a hubbub factory. There's always something going on. And this week, one of the things that was raised up was the API removals in version 122. Now I covered this in 116 as well in a session very like this where we talked about like basically how to identify those manifests that have been deprecated and things like that. And I figured I would dedicate this episode to that same sort of a piece of it. And we're gonna get into that here in just a little bit. But if you want to jump ahead or read ahead, you can always jump over to the Kubernetes blog that's blog.kh.io. And the first one here is a really well-written and well-thought-out post from Krishna Killari at Amazon Web Services and Tim Bannister at the Scale Factory writing a blog post on Kubernetes API and feature removals. I don't know if it's feature removals per se, but like API removals, portions of the API that are being removed from the generally available API inside of your Kubernetes cluster when you deploy version 1.22. And we're gonna kind of play with that and see what that looks like and how that feels during this episode. But if you want to read ahead, this is where you can find out information. So the version 1.22 release will stop serving. This is an important distinction. Stop serving the API versions that we've listed below. These are all beta APIs that were previously deprecated in favor of newer and more stable API versions. So some of the stuff that's been removed are things like the custom resource definition in this particular path, right? And we're gonna play with this a little bit. We're gonna get hands on with it. So if this is confusing to you at this point, don't worry, we're gonna get more into detail. I'm gonna show you just exactly what this means to you in your cluster during this episode. That's really my target for this episode. The API service, the token review, subject access review, certificate of signing request and lease.  This is probably gonna be the one where people really get caught out is ingress, is no longer going to be available at extensions V1, beta one and is now or at the networking k8.io of V1, beta one. So we'll see what path it is available at and we're gonna dig into like how that works. So the Kubernetes documentation covers these API rules for V21, 22. And that's actually in the deprecation guide, which is a really great one. And you can actually see when things are removed and what things were removed at what versions. So if you wanna understand more about it, the docs are a great source of information. Looks like the next set of removals are going to happen in one, two, five. But that is some of the content that we're gonna dig into this week. One of the other things I saw, I mean, of course, one version 1.21 is out. So the next version of Kubernetes is gonna be the one that removes those APIs. So if you want to look into the change log, but difference between what happened in 120 and 121, this is always a good thing I try to spend an episode on. I'm not really not gonna do that in this episode, but right now, I'm just reminding folks, this is actually how you would go about reading the release notes for a given upcoming release. And so if you're looking to adopt version 1.21, this is where you will actually look for information. And it has a lot of really important information. Some of the major themes that are happening, some versions of customized or upgraded from version 203 to version 405, people who have used KubeKettle customized, you can see why that's an important one. The default container, hello. Good to see you, thanks for saying hi, hello. We have immutable secrets and config maps that now graduates to GA, which means the API path will change. Whatever you see graduates to GA, you know that there's going to be an API change, right? That's one of the ways that you can determine that. Structured logging in the KubeLit has changed, generic, fmro volumes, but there's lots of good content here. And one of the other pieces I really wanted to call out was this section of the release notes for every release. And one of the really important bits is this piece here, urgent upgrade notes. And I really like how they actually call this out. I feel like this is repeated every release of Kubernetes know you really must read this before you upgrade. So this tells us information that is important to us if we're going to upgrade from a previous version of Kubernetes to the current released version of Kubernetes. So definitely read this stuff if that's you, or if that's something that you're gonna be working on. So this was just a quick reminder of where to go and find information regarding upgrade paths and release notes for Kubernetes clusters. And so the release notes for 122 haven't been cut yet, but I imagine portions and pieces and portions of them are already in there. And we're kind of getting a little bit of a preview into what that looks like here of the deprecated API migration guide. All right, what is next up in our notes? I did see a new CBE being announced. And the way that I track this stuff is I'm actually a member of the security announce group. If security and Kubernetes is important to you, I recommend that you become a member of the security announce group, right? And if you click on over here, it's a Google group and you can go ahead and join that Google group and you'll be able to get notified when things interesting and security-wise have happened. This particular one is a security issue was discovered within Kubernetes that could enable users to send network traffic to locations that they would otherwise not have access to via a confused deputy attack. It's a low severity. It was assigned CBE 2021, 25740. It gets into the details and I really like how CJA has done this and how I'm sure that many of the members of the security security are responsible on this one as well. It says, if a potential attacker can create or edit endpoints or endpoint slices, and that means that you have the ability to define that object within Kubernetes, then they can potentially direct a load balancer or ingress implementation to expose back end IPs the attacker should not have access to. Importantly, if the target's network policy already trusts the load balancer or ingress implementation, network policy cannot be used to prevent exposure from other names fixes, potentially bypassing any security controls such as load balancer source ranges. This issue is a design flaw that cannot be fully mitigated without user-facing changes. With this public announcement, we can begin the conversations about a long-term fix. So if this, I mean, if you're making use of load balancer and ingress, which you probably are in your production clusters, this is probably something to be aware of and be aware of that those things could actually be changed. So there's information on how to mitigate it. It's basically changing the permissions for that particular object or those objects, the endpoint and the endpoint slice object, how you can detect if anything has actually used this. And there was an additional advisory, a similar attack as possible using the ingress implementations to support forwarding to external name services, which are one of the valid services within Kubernetes. This can be used to forward two services in other namespaces or in some cases, sensitive endpoints within the ingress implementation. So really good update from the, from security. And if you're interested in understanding more about this vulnerability, put this here. And one of the things they always add here, it's kind of at the bottom, oh, it was actually Rob Scott sent on behalf of the Kubernetes product security committee. So awesome stuff. So there's a link to the GitHub issue. If you want to know more about it, or if you want to follow along for the resolution of this particular CBE, definitely check out that issue and you'll be able to see those things. So pretty cool stuff. Lots of good information coming out of Cube Weekly and also out of this week in Kubernetes it's coming right up. And then the next thing I wanted to get into were these two pieces here. So this was the Twitter thread that I was describing earlier. It started with Kat Cosgrove putting out a Twitter thread saying, you need to pay attention to the release notes for Kubernetes version 122. I know nobody likes reading release notes, but you gotta, and in this version, multiple beta APIs will be removed, not deprecated. They've already been deprecated or we wouldn't be removing them. And this is a removal of those APIs from serving. And we're going to play with that during this session. That's one of the things we're going to dig into. But one of the comments to this thread that I got a lot out of was actually from Stefan. Stefan is a core contributor to Kubernetes and he makes a few valid points about how to understand this particular state that we're in when we're going to stop serving a particular API inside of Kubernetes. He says that APIs go away in 1.22 but understand that existing objects created with old APIs are not. And what that means is that if you're in version 1.21, you can still create those objects using those same versions. And those objects will then be inside of your cluster's database, inside of SCD, stored as objects. And when you then upgrade that cluster to version 1.22, that means that those objects are still defined within Kubernetes. And we'll talk about how and why that works when we get to the playtime part of this session. How's it going, Dev? Good to see you. Welcome. So there is no object version of V1, beta one in the cluster concept. Objects are converted on the fly. And that means that there is an abstraction layer in the API server that is responsible for defining that mapping between the API objects that you create and the stored object that is stored inside of SCD. And this is gonna be one of the fun things that we play with today. We're gonna look at the difference there and see how that works. Hey, Pop. Good to see you, Pop. But suffice to say, what that means is that as we change API versions that are made available or unavailable within the Kubernetes API server, those API versions are not generally stored as an object within SCD. It's actually just the core object itself that's stored in SCD. And the abstraction that the Kubernetes API server provides is how we surface those objects either to post when you're going to be creating those objects or whether they are gets or reads when you're actually trying to keep kettle get or update those objects. That's actually how that bit of it's going to work. We're not in a situation necessarily where we store the API version that it was created under. All right. So that's actually pretty interesting stuff. And we're gonna play with this. And I like how it says, this is magic. It's not magic. It's software, you know, it's all right there. When storage versions are removed in a release but Etsy still has objects of the old storage versions, QBPI server fails starting up. That is actually incorrect. You have to touch the objects to convert them for the upgrade. And the Kubo storage version migrator can help. Read the release notes. And there's going to be documentation on how to do that. And then the last thing that Stefan calls out is this piece here, which is where I'm gonna spend the most of the time in playing with this stuff. Because I feel like this article by Evan Cordell, who's somebody I worked with at CoreOS, great guy, really gets into, like really deserves more time, right? Because I feel like this article really gets into some of the internals about how Kubernetes API server works and how those API versions actually are surfaced and manipulated. When you do a kubectl get or a kubectl apply or kubectl, those sorts of things. When you're working with the API server directly. And so my plan today is to kind of get into this. This is going to be the play time. We're going to walk through some of these scenarios. There's actually, I think this blog post does an incredible job. And so if you want to skip out and just read the blog post, that's the thing that you can do. But I really feel like a lot of the points that are made in this blog post are worth kind of exploring live and we're going to play with them. All right. But there was one more thing I had, which is something that I learned. And I wanted to play with you and I wanted to share this with you. It was one of the things on my side that I thought was actually kind of interesting. So I'm working with ISOVALENT, which you already are aware, I'm sure. And I'm working as their field CTO there. And so I'm having a great time at ISOVALENT. And my work computer is a Linux laptop. And this is generally true whenever I'm not working at a place that doesn't allow that. But right now I have a Linux laptop. And one of my goals was to get to a place where I could do better work with like BPF trace and those sorts of things. And the way that BPF does that bit of tooling is that it needs header, not header information, it needs a BTF symbol information so that it can understand how to map events that happen in the Linux kernel with context around what's related to that particular event. The way that it does that is that it actually holds, you could consider it kind of like a dictionary of symbol data. So if you've ever built a debug package for a package in Linux, right? You could create a debug package and that debug package will hold the symbol data for the package that you're going to debug. And that way when you're doing a trace on that application to understand where it's spending its time or where a particular bug has happened or you're digging into a crash, that symbol data will actually allow you to kind of like really kind of in a more human readable way understand what's happening with that particular data. So BTF is very similar. It holds that type data for events so that we can use BPF to map what's happening at the type with at an event layer with what's actually happening inside of the kernel when a particular syscall or a particular event happens as a kernel layer. That was a lot of information and I hope it made sense. But if it didn't, feel free to reach out to me and like let me know that the thing I said doesn't make any sense. Like you can hit me up on Twitter or you can hit me up on the Kubernetes Slack. There's lots of ways to get all to me. But I'm Maui Lyon everywhere. Suffice to say what I was working on, I wanted to get to a place where I could actually do BPF trace and those sorts of things from my own Linux kernel that I have running on my laptop. Now what I noticed was that the Linux kernel that I was operating didn't have access to BTF information or BPF type information. And there's a couple of different ways that I can solve this problem. One of the ways is to build a debug kernel and then to grab, yes, the link to the Evan Cordell is definitely in the blog post. But you know what? I can actually just put the link in the chat, I think. Let's try it. Hopefully you can all see that. Did that work? Also, hi, Diane. Good to see you. So the way this works is that you can build like a debug kernel and that debug kernel will basically build a version of the kernel that gives you access to all of that type data. And then what I found a new config option in the Linux kernel, well, new to me anyway, I'm sure it's not like super new, that gives you the ability to define that you want the kernel to make that BTF data available inside of SysFS, which is really cool. So let's look at that real quick, because this is actually the thing that I kind of learned this week. So begin here. So we move into, so I've got my Linux kernel. I'm using Arch as a Linux as my operating system for my laptop and it seems a little contentious. So I think a lot of people kind of chuckle about the whole like I use Arch kind of thing, but that's not really where I'm coming from. I like that it's actually very flexible I like the packaging system and I like the AUR stuff. But let's dig into how this would work inside of Arch because that was actually the problem that I had to solve. So this is the actual build package for the Linux kernel that is shipped that works and it is most compatible with the version of laptop that I'm using. I'm using a T14S. So it's got an AMD chip, it's a Ryzen chip and I'm using the Zen version two chip, right? So it's the, I can't really like a Cezanne family or something like that. Regardless, this kernel has a lot of the optimizations for the AMD chip that I'm using on my laptop. And what I had to figure out was how to make it so that I could build that BTF information. So let me first, let's just take a look at where that data is actually being made available. So it's in this SysFS underneath kernel, BTF. The setup here, we can see these are actually all of the BTF data that are actually stored for not only my Linux kernel, which is right here, but also for all of the modules that have been compiled for the kernel. So anything I compiled as a module, BTF symbols are also generated and made available here. And when these things are available and when these things are made available inside of the SysFS, that means that BPF trace and other tools that leverage libbpf can just make use of those symbols directly without having to go look anywhere else. They can immediately find the symbols they're looking for and then give you really good contextual information about what's happening on an event level for BPF trace at the kernel layer, which I thought was mind blowing, really, really cool stuff. So if we do it that way, then we don't have to worry about like actually making that BTF symbol information available to some other application. Say I wanted to run BPF trace inside of a container or if I wanted to like run some other application that would need BTF symbol data within a container or something else that was going to share the same Linux kernel as my laptop. So if I were to do Docker, excuse me, Docker run bash ls sys, we can see that we have that same data available inside the internal. And so what this means to me, this is actually the thing that I was playing with on that very beginning was I needed some way, I wanted to play with some tracing information in a Kubernetes cluster, but I wanted to be able to leverage kind to be able to do that. And so to be able to get across that finish line, what I needed, what I was missing was making that BTF symbol data available to my Linux kernel directly and then any containers that are also associated with that same Linux kernel. So anything running inside of a kind cluster or anything, any of the Docker container that I run locally on my laptop will not have access to that same data. And I don't have to make a file available to that to BTF trace. I could actually just build it right into the kernels sys fs, which is so cool. Anyway, that's what I wanted to show you. So the differences that I made between my, the upstream kernel that is available to my laptop and my own are pretty, are not very substantial. The difference was it diff or the master configured. All I had to do was add these three lines and actually only two of them are applying right now. So it's config debug info and config debug info BTF. And this makes my, that makes that sys fs piece work and that makes it so that BTF symbols are available to anything that would actually try and leverage BTF inside of my laptop at any layer as long as it's not a virtualized layer. So really neat stuff. Anyway, that's what I wanted to share with you on that piece of it. And now we can kind of get back into that incredible blog post. So six things, I love the call out here too. This made me chuckle as soon as I saw it. 16 things you didn't know about cube APIs and CRDs. Number 15 will shock you. That's so great. So this is actually, I mean, I think I had known this but I hadn't really thought about it before. But what this says is not all Kubernetes APIs have controllers. So the thing that might be modifying state for a cluster may not be a controller, right? The thing that updates that object may not be an actual controller necessarily. It could be some shared context that is updating status. So let's read this a little bit and see what it says. The QBVI server is largely responsible for the storage of API data into a backing store. And typically that's at CD, as we know. The Q controller runs a series of reconciliation loops. I consider these to be controllers because it's a bunch of controllers that are just really kind of using a shared cache. And so it's a more efficient set of controllers than you might think of for like controllers for CRDs, for example. But in this particular case, because they're using a shared cache, they can kind of get back and forth to that data very quickly. An example of that set of controllers that happen at the controller manager, for example, when I create a Kubernetes, when I create a deployment, if I do a kubectl apply or kubectl create deployment, then that creates a deployment object. And then there's a series of controllers that are responsible, that are being managed by the controller manager that progress that deployment object to a set of running pods, right? And so for example, there is a deployment controller that is responsible for breaking that deployment into a replica set. And there's a replica controller that is responsible for breaking that replica set into a set of pods that have been defined. And then we pass the control or lifecycle of those pods down to the kubelet, or actually no. Once those pod objects are created, then the kubescheduler kicks in, kubescheduler associates those pod objects that have not already been attributed to a node with a node. And then once you have a complete pod object, the pod object exists, it's been scheduled to a node, all of those things are checked off the list, then the kubelet sees that that pod object has been created and is associated with itself, and then goes about managing the pod. Again, in a controller kind of way, right? There's a reconciliation loop that happens on the kubelet that is responsible for ensuring, A, that we know about all of the pods that are associated with this node, B, that we understand the lifecycle, where we are in the lifecycle of each of those pods, and that we're reporting where we are in that status to the Kubernetes API server so that those objects can be updated. And this is, I think, where Evan is going with this, right? Is that when you go ahead and create those things, there are different pieces that are responsible from updating, perhaps, some parts of that object over time, right? So the deployment controller may not be the thing that marks itself as progressed, and that might be the replica set controller. That pods are active or unactive is actually something that's going, or active or not active, or something that's going to be kind of bubbled up from the kubelet to the pods status object, and then to the status object as a controller, right? So what information is reported at the API server when you do kubectl get deployment, whether you have the, whether you've reached the current running state of that object or not, can be updated by different sets of controllers, not by one particular one. So let's take a look at how this works. Clear. I go ahead and create a cluster here. So let's take a look at what he did. So he had, I think it was just doing the single node. Let's bring up our cube cluster. Thank you, Dan, for updating that. I thought I'd put the link in there, but if I didn't, thank you for putting me in there for me. I appreciate it. And thank you, Dan, for the great comment. Makes me happy. Glad to, I'm glad you're here. I'm glad everybody's here. I'm gonna do a kubectl create cluster or create deployment dash in.  I just created a deployment object. And if I look at that object, events, there's a couple of different things that happen. You know, some of which are, oh, here we go. So we see the deployment object get created and then we see the replica set object get created and then we see the pods get created. So here's the deployment object. Here's the replica set. Replicaset controller, basically creating two pods. Then we have the pods themselves being pulled and created, starting the container ngenix. And this is being reported usually by the node that is reporting it. Pulling the image and creating it. And so what we see there is that like those controllers are responsible for each of those controllers are responsible for that. The command I just ran was kubectl get events. And I was just targeting the namespace. In fact, like deployment, I think you can do, I always forget if there's a way to actually get kubectl events for a specific object. I think there are. I'm used to just looking at, here we go. You know what it is? Describe deployment. So these are the events. This is the event data happening at the deployment object itself. And so this is an event data that is actually represented inside of the object itself. And then if you look at the kubectl events for the namespace, then you'll be able to see even more event data for all of the things associated with that deployment, right? And so what we saw was a deployment be created two minutes and 30 seconds ago. We then saw a replica set telling us about things that have changed, right? So we've got a successful create for the pod. And then we saw the pods being started up. And we're kind of getting a little bit out of order, right? Because we're seeing like two minutes and 29 seconds ago we're seeing the pole image. And then two minutes and 23 seconds ago we saw the creation of the pod. So we're seeing, actually we're not seeing out of order. We're seeing the order of events. So a creation of a pod object is different than showing up on the kubelet and kubelet reporting. Okay, I'm pulling the image so that I can actually start that thing up. All right, so I'm gonna do kubectl describe deployment. Click in. And then we're gonna look through some of the data that is being presented by the deployment object. Right now I'm only looking at the deployment object. I'm not looking at pods. I'm not looking at replica sets. I'm just looking at the thing that I created which is the deployment object. And it's interesting stuff because I have a lot of information that is made available to me here at the deployment object that might have been set or made available to that deployment object through some other controller, right? Or has been bubbled up by some other controller. Replicas for example, right? I did say when I created the deployment that I wanted to replicas but the deployment object is not responsible for creating replicas. It doesn't understand that, it doesn't understand necessarily that those replicas have been created but it can surface the answer to that question, right? So it's surfacing that I requested two replicas that two have been updated, that there are two total and that there are two available. And this information is made available to me via the deployment object when I'm doing kubectl describe deployment I can see that data but this information does not come from the deployment controller. It's bubbled up to the deployment controller from or to that object from the replica set controller which I think is really kind of the important point that Evan was getting to in his blog post. So let's move on to the next one here. So that was the point of it that was pretty interesting. And then he has some other examples, right? So controller manager has a deployment object subject access review is one that actually exists but it's actually the abstraction for that is entirely within the API server. So this would actually just be a function call within the API server when you do a subject access review although we create an object to hold state for that object. The thing that is actually manipulating that object is actually going to be the API server itself because the API server handles the authorization piece of it. So there's no controller necessarily inside the controller manager even or on the kubelet or any of those things that is responsible for managing that subject access review object. When you do a, when you, so when you create one, you're actually only seeing the, we're only capturing the state of that leveraging a function within the API server itself. Kind of interesting stuff. So we return the controller manager off we would be able to see the object get created. So here he's actually pointing out that like that the subject access review gets created. And so there is an object inside of the API server that is known but then there's actually a status, right? Whether it's allowed or not allowed and that status is set by the API server itself. So instead of actually just creating the object and letting some controller handle it, the API server itself is handling that object that status update. That's pretty neat. Now here's another one I think I personally had internalized but I had never really thought about it in these terms. And I think Evan does a great job of making this very approachable and understandable. And that if you've worked with the Kubernetes API at all, one of the things that you learn about is this idea of GVK group version kind, right? And this is gonna be really relevant to those removed or deprecated APIs because group and version are part of what gets deprecated or removed from the API server in these circumstances. But the kind persists, right? We still have an ingress object that you can define in version 1.22. But what's different is the group and version. The path to that object that gets created is different. So we're gonna dig into like how this works in this article. And I think he does a great job of describing it. But the other piece that I think highlights is that you also have group version resource. And these things, these three coordinates identify APIs, right? And so when we think about custom resource definitions, we're defining group version resource and we're defining whether that resource is scoped to a namespace or to a cluster. But this is a resource, it's an API that is extended, right? And then the objects, the representation of objects that are consumed by that API, that's GVK, group version kind. So if I defined a new CRD for pieces, and I might say my group and version are, I don't know, trying to think of some really famous New York, Italian pizzeria, right? As the group, right? So another practical example is it might be like isovalent.com or yourcompany.com or cluster.com, any of those particular groups that you define. And then the version is where in the API versioning scheme am I for this particular API that I've extended, right? So is it an alpha, is it an alpha API? Is it a beta API? And what version or where in the life cycle of that versioning scheme am I? So the very first one, I might call it V1 alpha one. Rouse, there you go. Thank you, dad. I mean, thank you, pop. So the group could be Rouse and the version could be V1. Actually, that's probably V1, right? Cause like these guys are very mature and making pizza. And then the resource object might be pizzas. But when I create a pizza, like say I want a, just an amazing pizza that has like basil and tomato and just like really just like, I want to go for like the OG pizza. I want to go for like, you know, really good, like the really good just like base upon which all things are built pizza, right? Then I might create that object and I have to create an object of kind pizza. And then I can say what the ingredients would be, basil, tomato, the really amazing dough and that's going to be my, that's, I mean, it's some cheese and that's going to be my pizza. And so this is where the GVK part comes in, right? When I define a pizza, leveraging that API made available to me by a resource, then we can actually, let me create the GVK. We create that object and that object is represented by the group version kind. All right, so let's take a look a little bit more into how he defines it here. And the example here saying, let's take a look at a single object, right? And so in this particular case, we're looking at the, an RBAC object that's been defined kind of by default within your cluster. Oh, nope, this is actually one that he created because it's inside the namespace default. So this is a, this is a roll. Interesting, why is that namespace there? Because he created the roll within the namespace. So this is a namespaced roll that has been generated that he generated directly. And you can see that the group is RBAC authorization kx.io and the version is V1, right? So this is the actual object that's been created. And this is how this was created, right? So if I do kubectl create roll and default, we're going to give it a name like pod reader. And then we're going to verb equals get list resource equals. This is actually just using the kubectl command line to go ahead and create that same object that we see in the blog post. And if I do kubectl get roll pod reader, that's OEML, we should be able to see that basically that same output that we found on the left and it gives us the ability to play with this stuff, right? We can actually just create that object directly. Now, this is an object, like I said before, and this is actually one of the things I thought was really distinct about the blog post, right? This is the group, this is the version, and this is the kind. And this is defined here because this is an object that has been created. If you're extending kubernetes to understand a particular CRD, then the group inversion might be something else. But this particular group inversion is made available via the core API. There's nothing in the objects that tells you what the resource is. The kube API server maintains a mapping of resource types to kinds. As do kube clients like kubectl or client go. This is called the rest mapping. In these client projects and related docs, once you know the resource, there are simple rules for building URLs. APIs, group, version, resource, name. APIs, group, version, namespace is a name. If it's a namespace object, if it's a cluster scope object, you'd be able to see it here. If it's a namespace object, you'd be able to see it there. Now, if you want to explore that stuff live, or if you wanna kind of dig into playing with this pathing information, you can actually look at this information directly if you just turn on kubectl proxy, right? So let's just say kubectl proxy. And now I'm serving the entire API server in a kind of a read-only mode at 1-2-7-0-0-1 call it 8001. It's not read-only. You can do all kinds of stuff there, but let's go ahead and pull this up. So in the, let me make this a little bit bigger. So in the blog post, you know what, I'm gonna clean up my browser tabs here. I know people don't generally do that, but I'm a wild person that way. Let's do some stuff. So we see APIs, group, version, resource, name, right? So let's see if we can find the resource associated with RBAC. So I know that this is gonna be under authenticationk8s.io because that's where the RBACs, actually it's gonna be an RBAC. Well, where would it be? That's a great interesting question. So our object is a role. And actually we can see from the header information, it's gonna be RBAC authorization. So if we look inside of here, we can see RBAC authorization. So this gives us that same path that we talked about in the blog post, right? If I wanted to understand for the role, what the path would be, it would be under slash APIs, group, version, resource, name. So my group and version are RBAC authorizationk8s.io and my version is v1. So let's go check out that path and see what we see. So I'm gonna go ahead and grab this URL, copy. And I'm gonna paste it against the RBAC proxy that I see here. Inside of here, this is that rest mapping that Evan is referring to, right? So these are the things that can be created. So this is the resources that have been defined. You can define a cluster role binding and the kind or objects that can be defined for this particular resource is a cluster role binding. The resource for cluster roles and the object or kind that can be defined is cluster role. And these are the things that can be done to that particular object for this particular rest mapping. So when we create this object, these are the things that can be done with it. If we cruise down a little bit further, I think we can see roles, right? And this is an interesting one because roles are defined not at a cluster scope, but at a namesake scope. So in this rest mapping, we can see that this is a role, is represented by a kind or an object called a role, and it is a namespaced object, meaning that it can only be created at a namespace level. I can't create a cluster wide role with this object. There's a different object for that. But since this one's namespaced, it can only be created within a namespace. And that's why when I did my kubectl command to create the object, which I think I made go away, but when I did the command to create the object, I had to specify a namespace in which to put it. And then the verbs associated with this one, it can be, you can do a create, a delete, you can delete a whole bunch of them, you can get them all, you list them, you can patch them or update them, and you can also watch for them. Now, in this rest mapping, we also have storage version. And this is actually kind of an interesting one. Let's go ahead and decode this because I think it's kind of interesting that it's in base 64. I thought that this would actually not be, but go. Whenever I see base 64, I kind of almost always want to just decode it to see what it says. A helpful rest. I wonder if I messed something up here. Oh, storage version hash. I'm silly. Up here at the top. So for the group in the v1, we can see what the preferred version is that we're going to store it as or make it available. Let's go back to our blog post here. So this is actually how we can see it. And there's actually another great doc in the docs that says API concepts document that gives you information about how this stuff works. Thank you for coming by, Diane. See you next time. All right. The next one up is objects are accessible and all API versions, which I think is another really important one. And I think we're probably going to have to hop through these a little quicker because I think we're taking our sweet time about it, but I feel like there are some really succinct points that we should make first, that we should get out of this, right? So objects are accessible at all API versions. And this is actually a really useful one for understanding how to convert objects and how that conversion process works as you handle it upgrade, right? Let's take a look here. So we create an ingress at this newly to be deprecated version, networking.kh.io if you want beta one. And then we do a get or a create of that. We'll see that object get created. And we can see that instead of, and then it's made available to us on the get at this new version. And that means in this particular case, it's calling out the preferred version as what he's referring to here, right? So when we looked at the group version, right? We can see that the preferred version, so anything that when somebody just does a random get of roles, then I'm going to provide them the result as mapped in this way. But if I had a different preferred version, then even though I might have compatible versions, like in this particular case, I support creating, it's supported to be able to create a role using V1, beta one or V1. You can create either one, but when you ask me for the results of that object that was created, that I'm only going to provide that data unless otherwise specified as that particular version. You can find out what the preferred version is by looking at the group. And if we're looking at it here, but he also calls out another really cool command, which is this one, control C, you can get raw APIs. And so in our example, let's just do networking. Okay, it's to IO. So that's the group. We can see the version that is being made up, that is preferred as V1, just like they were talking about before, right? The version of the object stored in SED may differ from the submitted version or and the preferred version. For every group and resource, the Kubernetes API server knows the served versions, the decodable versions, the storage or encodable version and the preferred version. If you submit an object to the API server and it doesn't match the storage version, the API server will convert it to the storage version before storing it in SED to be more precise. At first it converts it to an internal version before converting it back to the storage version, but this is an implementation detail. Now this is where like some deep magic happens at the API server. And I think it's worth understanding that this happens because it's actually pretty neat. Trying to think if we can actually have an example here to walk through it, but maybe he's actually answering that for us. So for example, in cube 120, the preferred version of ingress is V1, but the storage version is V1 beta one. This can be seen by looking at the data the API server stores in SED directly. But he's created the ingress object and he's going ahead and grabbing the content, the data from SED directly, leveraging Auger. And we can see that the API version stored inside of SED is a networking k8s.io slash V1 beta one, or V1 beta one. Now that's a little bit contrary to something we said before, when like the object that is stored, the content that's stored in SED doesn't know about group version and kind. And I want to clear that up a little bit. So I think where the disconnect is happening is that we say that the object doesn't know anything about it. Now, while that's true, it means that if you look at the object that's stored in SED, there is some content that's stored about it, right? So we can actually have a preferred storage version or a storage version that is represented by that storage version. But that storage version isn't necessarily the entire API that is exposed by that extended API itself, right? It's just whatever is necessary to store the object in SED and these can differ. And this is I think where Evan's point is coming in, right? So in 120, for example, we stored only that data that was made available inside of that beta API. But in the newer version, the stable version, the content might have changed. It might have required new fields. And so there's some conversion that has to happen. Now, if we cared, if we get caught up in it here, what that means is that we can easily get into a position where the data that's being presented when you do a kubectl get of an object might be materially different than the way that that object is stored. So let's say I created that pizza that we talked about before. And before I didn't particularly care about what the contents of the dough were or a particular topping. Like I didn't even make it available that you could put pepperoni on a pizza. And so I didn't really have like the topping section built out. And so to make the topping section, to build out the topping section, I needed to change the API. So if I created a version of pizza at V1 beta one that didn't have toppings, and I made that my storage version, then the thing I'm storing in SED wouldn't actually have information about toppings at all. And so even if I were to make a new extended API that had that topping information, I would need to have some way of defaulting or making available the toppings option so that that object could be stored with toppings as we move forward, right? And so the storage version that is represented inside of SED might be materially different than the version of the API that I made available via the CRD. I know that wasn't super easy to follow. Was a little bit confusing, but suffice to say the storage version is usually the preferred version, but it can be overridden. So in 120, for example, the storage version was different than the preferred version. Changing the object stored version requires writing a newer API server or a newer CRD depending on what you're working with here, right? So if we wanted to change the object storage version for something that was built into the API server, like that role object that we talked about before, we would need to change the API server because that's actually where we're defining the API for a particular version of Kubernetes. In this example, we create an object at V1 in Q120 which stores it at V1, beta one. We update the API server to version 1.21 and check in SED that it's still stored as V1, beta one and then we overwrite the object with a new API server to see it stored at V1. So check this out. This is how you might migrate or how you, earlier we talked about, I'm gonna pause this for a moment. Earlier we talked about the very salient point that Stefan pointed out, right? Where in if you create an object at a version that will be removed from API serving, how do we actually convert that object on the backend to make it available so that we don't just like abandon those objects that don't work anymore, right? And this is actually practically how it works. And this is a great example, right? So we have a cluster, its version is serving V120.7 and then they go ahead and create a new ingress object using V1. And we can see because we're still running at Q120 that when we go through the QAPI server version at 120 the object is stored in SED as V1, beta one. So it's that same SED CTL command and we can see that it's stored as V1, beta one. So now with that API server operating at V1.20 this is the only way we're gonna store these objects. Now he's migrating his API server to version 1.21 and we can see that the change has been made. We reconfigure SED using the same storage backend. Now, even though you couldn't, you wouldn't be able to create this object at V1, beta one anymore. You would still be able to see that it's stored that way because this object came from before the cluster was upgraded before the API server was moved to version V1.21. And what this is doing is it's just doing a straight replace of the object. And we can see now that the version that we do a Q, if we do a SED get, we can see that the stored version gets updated. And this kind of happens somewhat organically within Kubernetes, which is actually pretty neat. So it means that anytime you modify that object and the storage version has changed, anytime you modify that object, the object will get stored as the new preferred storage version. So if I created a whole bunch of resources like deployments, for example, at V1, beta one, and I haven't modified that deployment, but my cluster has moved forward in versions, right? So I'm no longer a 116, I moved to 117, I moved to 118, I moved to 119, what have you? The, those objects need to be touched or modified in the cluster before you move to the next version so that we can store those in the new storage version. And this is where that, those release notes that Stefan was talking about before. You have a couple of different options. So you can actually run a keep, got a command that just goes through and does a replace on every object in the SED data store. And it's an intensive thing, but you can even like, you know, stash it, right? You can go buy API paths. Let's do all of RBAC, let's do all of ingress, let's do all of networking, let's do all of, you know, deployments and replicasets and all of those different CRDs that we want to modify. And this gives us the ability to go through and touch all of those objects and get them stored to the new version before we move to the next version of Kubernetes. Now you might have heard like in the upgrade guide or read in the upgrade guide inside of Kubernetes that you want, that you need to ensure that you don't skip any patch versions when upgrading your Kubernetes cluster.  The steps that are necessary to do that are that you bring up a 117 cluster that might have changed the storage version for those objects stored in your data store. And then you touch all of those objects to make sure that they're now at that new storage version and then you can upgrade to 117, right? So you go 116, 117, you upgrade all of things to the new storage version, 118, upgrade all of things to the new storage version, 119, upgrade to all of things to the new storage version because that's the guarantee, the API guarantee that we have is that we'll get you from old version to new version but not necessarily old version to some forward version. And now you kind of understand materially why that's important. The storage version, the thing that's actually stored in NSE needs to be stored in a way that is compatible both with this version of the API server and potentially the next one. And that's actually how we manage that. So these are the pieces of this blog post that I think really, really apply to API removals and deprecations, right? What we talked about so far is if we're going to remove an API from serving, then that means that the objects that you created with that old API server are still around. They're still hosted inside of the SED cluster. And they may be hosted in SED at some older version of storage version, right? But the thing that's important is that those objects don't just get dropped on the floor. They don't stop existing inside of your Kubernetes cluster. They're still there. They just need to be migrated to a new storage version because you've changed the version that is available via the API server. And that may have actually already taken place, in fact. So like the preferred version is likely forward for all of those objects that we talked about in the blog post, right? So whenever you're creating even the old versions right now, they're being stored as a new version. The conversion is already happening for you. But suffice to say that as you move from 121, the scenario of I had a cluster, it was operating at version 120. I created a bunch of objects against old APIs that were available to me in version 120. And then I upgraded my cluster to 1.21. Are those objects still there? Yes, they're still there. And they're still available. And you can still modify them and touch them and make use of them. But you can't create new objects at that API version anymore. If you tried, you would get an error back saying that that API version is not available. But all of those things that you did create are still present within the API server. If you want quick ways to understand what versions are available, right? So this is actually one of the tricks I wanted to share with you here. One of my favorite commands, I give so many shout outs to this command. It's like seriously a lifesaver, I think. Keep kettle, explain, deployment. But even if your CRD hasn't implemented the explain piece completely, this bit up here will already be made available inside of kubectl explain. So if you wanna know what the current preferred version of an object is, you can actually look in kubectl explain and it will tell you if you're gonna create a deployment, go ahead and create that as group apps version V1. And if you have CRDs defined inside of your cluster, you'll be able to see that as well, right? Because CRDs also extend the API server and they also have to define what their preferred version is. So if you do kubectl explain of some CRD that has been defined, you'll be able to see what the preferred version of that object is there as well. I'm not gonna get into ratcheting validation. There is in 120 plus, there's actually a storage version API. This is something new that I was unaware of. Apparently it's stuck behind a feature gate, which is pretty neat, but apparently it would perhaps show us information. Oh, that's cool. Okay, so with the storage version API, this is a relatively new thing. I don't think it's actually there by default. Internal API server kxio, so let's see. I'm gonna check to see if my cluster has this resource defined. So let's take a look and see if it is, right? So because I'm looking for a resource, the way that I'm gonna determine that resource is available or not, is I'm gonna do kubectl get, actually kubectl API resources. And I'm gonna look for, I'm just gonna grub for internal. And it looks like this particular cluster does not have that feature gate enabled. And the way that he enables that, I kinda wanna play with this one. So let's go ahead and turn this on. I think that looks really cool. Unfortunately, Evan gave us an example of what to do from the kind config to go ahead and enable this, right? So let's go ahead and do that. The kind delete cluster. And if you wanted to play with this stuff, you could play with this stuff in this way as well. I'm gonna make a directory that we're gonna work on here. We're gonna call this twikin5cdtwikin005 and then we're gonna do kind.yaml. And so I know that this is a kind.yaml because of the API version in this, right? So it says kind.jsio v1 alpha four. And then these are parameters I can pass to my kind cluster that configure the cluster in a specific way. And these parameters get passed to QMADM inside of the cluster on startup. So these feature gates, if you wanna enable a feature gate inside of kind, you can just use the feature gate stanza and then enable particular features. And then inside of the runtime config is also enabling this piece of the API server. It's the API piece. So this enables the API on the backend and you can think of this as enabling the API on the front end. Whether we surface that API or make it available, it's our kind of two different pieces here. But with this configuration, we can do kind create cluster. Dash-config kind.yaml. This will create our cluster. And as we saw before, we didn't see any internal version when I did a grep for that inside of API resources. But let's see what happens after this new cluster starts up. We'll see if we actually see it there now. I'd love to play with this API and see kind of like what it exposes. That's actually pretty neat. Again, huge shout out. You'll always hear me say this consistently every single time. Huge shout out to the kind community and the project for making stuff like this, not only possible, but relatively simple, right? Like, I can't imagine being able to do this kind of work without having some way of locally creating a Kubernetes cluster and playing with all the knobs and dials and learning about these environments in a place that is like safe to break and not in your production clusters and also something that's just very quick to start up. Now I do see that API. Okay. So I'm gonna go ahead and do, come back over here real quick and break my cube proxy and start it up again. We're gonna go explore that API just like we did before. We'll do a refresh and internal API server kh.io. So I would think that this would be our path. So our group is, there it is. Found it. I was like, why don't I see that in here? But I see it now. All right. So there's our group and our version, right? So our group is API's internal. Let's go ahead and go to the group first and just take a look at the content on that page. Slash. Here is the content that is made available on the group. So we see the group version is kind of meta. So I can get the group version of group version or storage version. Then we can see that the only version that is available right now is v1, alpha one and the preferred version is also v1, alpha one. No shocker there. It's only the only one available. And if we go and look at the API that is exposed there and we're gonna look at the kind of, oh, I messed up something there. All right, about that. v1, alpha one. These are the resources that have been defined. One of the resources is the kind of storage version.  Both of them are exposed. Likely these are differentiated by the permissions that are associated or the way that you can update them, right? And so the object itself can be created, deleted, delete, collection, get, list, patch, update, watch. And then there's a different permission if you want to apply get, patch or update to the storage version object status. If you think about our back, that makes a little bit more sense. But so if I go one step further and actually see the API that is extended, right? Like what objects, what's the actual framework for that API looks like? So this is the spec, status, the storage versions, things that I can learn about it, right? So this is the encoding version. This is the decodeable version for the admission registration object. Really neat stuff. The relatively new API, it's still behind a feature gate, but it really gives us like some interesting view into how storage version is being managed, right? So in this case, let's see. Apps V1, there are three decodeable versions. Apps V1, Apps V1 beta two, and Apps V1 beta one in my cluster. The encoding version is Apps V1. So if I were to look into SCD for anything stored at the group, Apps V1, then I would understand that it would be stored in this particular version and this particular group. And that if I were to do a get for the one of these things, I could get at any one of these other decodeable versions, right? So for example, let's just play with that real quick and I could show you exactly what that means. So if I do kubectl get deployments, well, kubectl create deployment twicken image equals nginx replicas to port equals 8080 deployment twicken dash OEML, this is the object that was created for me, right? It said deployment is stored in sd1 beta one, but can I get the older versions of that, right? So if I do to understand what versions might be available, one of the other commands you can do is API versions. These are all the versions that are available. So that might not work, but let's try it anyway. So kubectl get deployment. Yeah, I want to create, let's see, kubectl API resources. There we go. So ingress exists in both. Kind of like the example that Evan pointed out before, right? Ingress exists in extensions V1 beta one and exists in networking V1. So if I do this, do kubectl create ingress tests. I actually tried creating an ingress this way. Expose deployment test twicken, and then I can do kubectl ingress twicken.com default equals in suite. All right. So kubectl get ingress twicken. OEML. We can see that right now it's giving me the result as this new version, networking.ksio v1, but this is the fun part. So I'm using kubectl bash completion, right? So if I do kubectl get, I can see the things that I can get. I want to look at ingress, the ingress object. And if I look here, there's a few different versions that are available to me, which is really pretty neat. So what this means is these are the decodable versions available within the ingress object. I can actually get the old version. This is .extensions dash OEML. And it will on the fly convert this object according to the API that was presented when this API was available, right? So I can actually get, I can convert this object in this way. I can do kubectl get ingress.extensions, which tells it that I want to see extensions v1, beta one for that particular group, this particular ingress object. And I can also look at the other one, right? So I had ingresses and I also had, oh, there it is. So there's only the two storage versions right now. Ingresses.networking and that's the v1 and ingresses.extensions and that's the older one. And then in a relatively recent change, I can't remember exactly when this was introduced, but I think it was in like 1.20 or maybe 119. If you start interacting with an API that has been deprecated, even if it's just a get, you're going to get a warning. It tells you, hey, heads up. This is a thing you probably want to know about. You created this object using extensions v1, beta one, that's a deprecated API. So we're doing as much of you can in the community to make this possible that you won't get caught out by this. All right, I think we've covered quite a lot and I want to go back to our notes here and see where we are. Well, that's a pretty new API. The cube storage version migrator can handle migrating changes. So we can just take a look at how this works. So like I said, if you're going to do like an upgrade from version 116 and you're aiming for 120, how do you get there? One way to do it is by leveraging tools like this, right? Where we see the old version is still stored at the old version, but we want to change that. So we have this tool that gives us the ability to modify the stored version. And it goes through basically and touches everything that needs to be migrated. Actually, it touches everything. It doesn't care whether it needs to be migrated. We can see what the migrator did. It touched a bunch of different stuff. If you think about it, this is like database migration. This is almost effectively what it's doing. Once that's all done, we should see the ingress object be changed. And there it is. All right, CRDs define new APIs, new resources, just like we talked about. All APIs are cluster-scoped, even-scoped namespace CRDs, meaning that the API itself is expressed as a cluster operation. Now, this is an interesting point. I'm not gonna harp on it too long, but when we think about multi-tenancy, this becomes a very important point because when you define a new API, you make that available, you make that API, you define that API at the cluster level, not at the namespace level. And so it means that you have to think about the permissions piece of it a little bit. A CRD has a stored version, just like any other API. This field indicates every version that has been a stored version, all of them. You can't remove a version from a CRD until it's been removed from status. Oh, that's neat. So this is more details about how CRDs themselves are managed. Anyway, so that's what I wanted to cover today. I do recommend reading this blog post. There's some really great stuff, and it really gets into the detail. And I think it really highlights some ways of understanding and playing with how these things could work and how you can explore this stuff, leveraging kind and explore this stuff in a non-prod environment where you can learn all about the way that storage versions and all of these pieces are actually managed over time. And you can take away, you can work in a really safe environment without having to worry about learning this stuff on the fly in production environments, which I also really highly recommend. I hope you enjoyed this episode. I enjoyed doing it. I'll be back in two weeks, and I will see you then. And if you have any questions or if you ever want to chat about anything or anything else like that, you can reach out to me on Twitter or out on Kubernetes Slack. I am at MauiLion everywhere. Thank you so much for tuning in. And I'll see you next time. Thank you. And then graphics.


So apparently there's something new for all the kids who wanna work out these days. Maybe it's just me, but. So I was at the park where I often run and do like pull-ups and push-ups and things like that. There's this guy, he's wearing these really nice jeans, nice belt, no shirt in his 20s, playing music and in between his push-up sets, he's like dancing and pointing to people, goes to his truck, he gets out a quart of orange juice, he's drinking the orange juice. I'm like, okay, maybe this is a new thing. And then the piece de resistance, he sits down on the table that's like this park table and busts out his crack pipe. Sounds like a good Thursday afternoon. Juice!  Crack it, so what's for fitness? You can follow us on Instagram, Twitter for more juicy content, thank you so much. Hey, Patreon, I'd like those pictures of my mom. Gross. Even he hasn't seen him. Today we got a best ever food review show. Yay! You like him a lot? Yes, we do. A exotic Indian food tour in Jodhaipur. No one should have to eat this. Girlfriend shot. Girlfriend shot challenge? A girlfriend shot challenge. Oh, that means. I don't either, but no one should have to eat this. Let's see what he's talking about. I'm excited. Here we go. Great thumbnail. Oh no! Why? That was like a cinnamon challenge. That was really hot though. Welcome to Jodhpur, the second largest city in the Indian state of Rajasthan. Well, hi. The population of over one million. The big Indian city would be 10 million. A small Indian city is usually like a million people. The people of Jodhpur are inviting and more. We are arriving during a religious festival. Spirits are high, the colors are bright, and there's something new to experience around each corner. When it comes to cuisine, Jodhpur has a mystery in its air. You can find countless fascinating delicacies that take the cuisine to the next level. The masala is rising up with the steam. Oh, that's so good. With so much to be discovered, I can't wait to experience the flavors of Jodhpur. The style is spicy. Yeah, no one for them, but Hania to eat it is really good and really spicy. Rajasthani food is. You need some rasi to kind of cool it down. Today, I'm on a mission to find the most unique, exotic air this city has to offer. From food that requires mad skills to create, it is the valet of the hands. To a savory gulab jamun that includes four types of dairy. If you like rich food, you'll absolutely love this. And also street food that's more of a test than it is a snack. Once as their boyfriend's love for them, if they can handle the spice of this chat. So grab your elephant and hold on to your trunk because it's about to get exotic in Jodhpur, India. Welcome to India. Today we are in Jodhpur. I am with Hashi Dhan. What is going on? As you drive around, it is a nonstop stimulation There's constant, orange, beautiful colors everywhere. I saw a goat wandering around. I don't know if someone lost that goat. I didn't see a name, hey, got it. That's a pet goat. Not forgetting. Someday. Oh, okay. We headed to Mirchi Vada. That's a breakfast item.  Perfect. A giant chili. For four generations, this family has been perfecting one of Jodhpur's favorite breakfasts. And today I'll get a firsthand look Breakfast? Right now we're headed into the home of the owner. This is where they start making the stuffing for this dish. And here we are. This is the home kitchen, right? Right? Yeah. Okay, great. Grandpa, the one who perfected the Mirchi Vada's unique masala spice blend is still the only one who knows its ingredients. With a recipe securely locked away- Because one of the ingredients is math. Not even- I don't trust you. It's a secret Vada recipe. You really have to earn that. The basic ingredients are just to find in any home kitchen. But the secret is the combination. It makes it very special about this dish. Wow. Mirchi Vada starts with boiled potatoes. After they're mashed up, they add in the super secret masala. We don't know the portions, but my foodie friend can suss out what's inside. Okay, so with your food skills, do you think you can deduce what is it? Reduction? Yeah. They basically do a tempering of human seeds in oil. And they add dry chili, chili powder, garam masala, salt, asafoetida, turmeric. Also, they add the dry lemon powder for timeliness. Yeah, it's rising up with the steam. Oh, that's so good. Next up, stuffing the potato mixture inside the chilies. The most important part of this, if you slit it and the batter is thin, chilies get fried as well, it's quite cool. After stuffing, it's dunked in their batter made from lentil flour, then deep-fried. Here's the ballet of the hands. Sir, how are you able to concentrate doing two different things with two different hands while I'm talking to you? How do you do that? That's fine. This is where we're eating...   Oh, that's a high. Hey, how long have they had this elephant? He's 30 years old. Whoa, this is still a young guy right here.  So that's why they put this well over here for good vibrations, because of that.  Wow, this thing is beautiful. I've been not expecting to see an elephant this morning. It's time for a fresh new batch of merchivada. You can have it with bread to cut some of the heaviness or saltiness or just take on the chilies. There's no way I could eat that. Here we have our merchitada. Just a chili? I think they serve it with no chutney. Does that require anything? The masala has turkey inside. It's like it's all in there. No additions required. It just still looks like a little chili, right? The stem is here. The whole thing is coated. Did we bite into it? Cheers. It's real hot. Oh, that is spicy. The tanginess is on another level. I can see why... Would you eat that? Absolutely. It's such an intense dish, but I love it. How many of these is he selling a day? 2000 pieces of these.  It's majorly a snack item, but people have it in breakfast as well. It's quite filling, right? Yeah, ma'am. I love Indian breakfast, man. They're always savory and hot. And we are done. One or the other, or both. Our next exotic food is completely foreign to me, but it's even crazy to most people here in India. It starts with what's usually... Gulab Jamun? Gulab Jamun is basically a sweet dish that you get throughout India, but in Jodhpur, it's turned into a savory dish. We won't get this vegetable anywhere else. We're headed to Gypsy to learn more. This place has a Tali buffet. Patrons sit down with a platter as their server. Gulab Jamun is a savory dish. Braves, breads, and more. People come here because they don't think about what they have to order. In a Tali, they get everything. That's brilliant. But Harshik and I are just here to learn about one food. Gulab Jamun ki sabzi. This dish uses four kinds of dairy. Here's how it works. First, grate some milk salt made from dehydrated milk. I'm dying just to know what it tastes like on its own. Not cheese. Let's try it out. Oh, that is intense milk. And you don't have time to drink two glasses of milk a day. You can just eat milk on the go. Combine the grated milk solid with refined flour and cumin seeds. Roll them into balls and fry them in ghee, also known as clarified butter. The more typical Gulab Jamun would then be soaked in syrup and served as a dessert. But here at Gypsy, we're taking a turn for the savory. Once the color turns golden brown, we take it out, we put the Gulab Jamun into the gravy. That becomes the Gulab Jamun ki sabzi. That sounds really nice. His gravy includes mixed red chili powder, turmeric powder, coriander powder, garam masala, and a whole boatload of other seasonings. That all gets added to some more sizzling hot ghee. Once the sauce is ready, then it's in with the balls. That is unique. Very. What's so special and kind of crazy about this for you is that they converted what's usually a dessert into a savory dish. Then he calls a vegetable. Yeah. Where are the vegetables? Because I'd say every big collard vegetable is unique for you. But for me, there's four different dairy products in here. They're unbelievably rich. Did you have your vegetables today? Yeah. We could eat it at the same time though. Whoa.  I have no idea what that would taste like. Somehow it's just kind of delicate at the same time. Yeah, it's very delicate, yes. With the bread you would just dip in there. Show me your ways. Make a scoop out of it. Make a scoop out of it. Take the bite, scoop it out. All right, cheers. Oh, that is good. It's super soft. Insanely creamy. A ton of spice coming through. Very present basalas that all work together in a gentle way. I like that. Yeah, me too. If you like rich food, if you love rich food. I would try that. Absolutely. I have no idea what that would taste like though. Bob Jumman is delicious if it's fresh. We are on the cruel side. Huge difference. When it's fresh. This is the perfect place for an evening snack or even like six evening snacks. I tried panipuri for the first time in Mumbai and found them to be slightly addictive. Seriously, I've had like eight or nine already. Last one, last one. But Mumbai only had one flavor. This place has like eight or maybe nine flavors. Whoa. Eight, eight, eight. Hold on, one, two, three, four, five, six, seven. Oh god. It's actually, it's eight. The panipuri must start with a hollow fried crisp. It's like a nacho chip bomb. Remember our panipuri eating contest? So different places have different stuff. Once again, they probably have fresh, yeah. That tore the roof of my mouth though. Then you get to choose one of eight different sauces. One flavor is this? This one, it has sugar and black salt and white salt. This one has mint. Mint? Yeah. That's the dangerous one. The garlic one. Garlic, yes. Green, man, no. We all love the flavor of green. What do you recommend? The dates one. Dates. Bam. We gotta go for it right away before it leaks out. Ready? It's good, but it's like spicy water. No. No. No. Okay, just tell me my surprise. Very delicious. The crisp is like corn chip to me. And then this, I don't know. It tastes like spicy water to me. What are you saying? It's a sweet one. What else do you want to try? Try the spicy one. We haven't tried the spicy one yet. Spicy. Okay, here we go. The spicy panipuri. That is very spicy. I couldn't go like eight of those in a row, but to mix it up, that's a fun one. Which do you think we should end with? Garlic. All right, we're going a hard, the garlic one. Nice. I'd try the garlic one. Green in color. Garlic panipuri. Whoa, that is so garlicy. That is like garlic juice. To me, all these are fairly spicy. They could just fill one of these up with pink baptismal. That would be perfect for me. Tuck, tuck, tuck, tuck, tuck, tuck, tuck, tuck, tuck, tuck, it is our final food on this inside convention. What is the name of this food? It's called Girlfriend's Chow. Okay, here's the one. Why did you name this Girlfriend's Chow? Like I said, girls like spicy food, so that's why he gave the name of the Girlfriend's Chow. Do you guys like spicy food? Ladko ka masu. No, no. He also told me earlier that girls taste their boyfriend's love for them if they can handle the spice of this chat. Oh, so this isn't a street food. This is a test. A challenge. But how spicy is this? I think you'll find it out. Okay, the Girlfriend's chat starts with baked wheat cups. Then green chilies, Kashmiri red chili powder, Rajasthani red chili powder, some chaat masala, garam masala. But this is like a cupcake of seasoning. Then perhaps to reduce the intensity, it gets some gram flour shavings, tamarind chutney, gram flour fried balls of different colors, pressed fried chickpeas, tomatoes and coriander. Please note, this should not be considered real food.  Can you just look at it from the top? It looks innocent enough. It's like two little cupcakes. Oh God. All right, we're going to end strong on this one, okay? We have like the cupcake of death right here, crispy outside, and then it is just a little bit of masala and spice. Chilies to three different kinds of red pepper powders. Why? Oh no, that was like a cinnamon challenge. Oh, and then the spice. That wasn't even the worst part. It was really hot now.  It is so spicy.  I need to find a lessie. Can we go find a lessie? I'm done. I'm done. Poor to me is something out of a dream. Over stimulating sounds, smells, and sights. A child dancing at a festival, kids wearing eye makeup, locals partying in unison, and flavors completely different from anything I've experienced in India before. Guys, thank you so much for watching. This video was made possible by OneTrip Vietnam. OneTrip is the highest rated tour company in Vietnam. Oh man, he always liked his videos. Another great one obviously, the spiciest food I had on our India trip was in Rajasthan. When we went to J-Pore. Yeah, I just remember. And there's one video that was just never released because the footage just, because when we went to this place, it was a cool place.  Then we had food at the end, but it was at night, and so all the footage was just not good. The audio was terrible. I didn't have a camera at the time. So I just used my phone.  Yeah, I remember repeatedly, and Drani and I were checking on you. Like, you okay? And we were exhausted by that time. Yep. Like it was. Yeah. We were running on fumes. There were probably three nights in a row around that time, where I was talking to Drani at the end of the day and saying, Corbin's doing amazing. That was hot stuff today, man. That was hot for me. Probably the most delicious food in Rajasthan was at the palace. Yeah. That was every little bite of that. Exquisite. Was exquisite. Most of the food we had was very, very good. That, as well as when we were in New Delhi, that chef who made us those things at the restaurant, that was pretty stellar stuff too. That was amazing. It's such good food. Yeah. I would love to be able to... Rajasthan probably had the coolest architecture. Yeah. The architecture there, because we went to obviously some palaces and just otherworldly. Yeah, really amazing. It's absolutely incredible. So glad we got to go to so many different places. Me too. Me too. Can't wait to go back. Anyways, let us know what other food videos we can react to. And if you've eaten any of this food, down below. Josh!


Dancing is something very special to me and I'm spiritual for me and I wanted to know How do you see dance fitting in this spiritual space? Well, I I'm not I'm sorry. I'm not really seeing you dancing, but people told me you're a great dancer I Heard that tomorrow you're releasing some new album or something. I will check that out the connection Uh-huh. Yeah, and um, it is it is a really really cool One of my biggest and let me say that I know you shouldn't say that yourself. I will watch it and tell you how cool it is Okay, well, yeah recently got to travel to Michael Jackson's estate here in In the valley they call it the Havenhurst Jackson estate and I thought it was So amazing because the first concert that I ever went to as a child was a Michael Jackson concert And I also got to Shake his hand and he is influenced my life You're supposed to shake a leg with him, huh? Not a hand You know what he was shooting the video I didn't get the chance to shake a leg But um, it was it was a very profound experience for me because I remembered um, I remembered uh How much he's influenced my purpose in being a musician a lot of a lot of people make music to sell records And I've done that, you know, I started at the precious age of 15. That's when I put out my first album I'll be 36 in Actually like two weeks so my my purpose and my meaning for sharing and connecting with people all over the planet has continued to grow and change and It was so cool for my album the connection to be able to go to the Havenhurst estate and You know speak about his influence. So the album is um a really meaningful album and um That's what I want to continue to do as an artist is just continue to give people the tool to Be positive and feel good with music and um one of the questions that I have was um a lot of composing music singing dancing and performing on stage um Is a spiritual thing How can I tap into a deeper dimensions of that in my life and um, how can I you piled up two questions one on top of another man? Okay, the one I'm sorry I got excited So yeah, see Michael Jackson Had the power to inspire a whole generation of people what he did was so unique and eye catching Anywhere in the world. I'm telling you in the remotest part of India in some village where there is no nothing They know Michael Jackson. I mean that's a level of inspiration. He brought in a whole lot of youth dancing like him in India In imitation of him so that was a powerful possibility, but I'm sorry if I say something that hurts anybody But I feel his life could have been, you know, he could have Made this into really transforming the world in some way, but unfortunately I feel No proper guidance around him. No proper advisors And all commercial people I think it went waste because just five days ago. I was at the Elvis priestly Museum in Memphis as a part of the journey when we were coming back. We stopped at Memphis Also went to martin luther king's, you know that place where the assassination happened So when I was at Elvis's place, also, that's what I thought he also for two decades He dominated the music scene inspired a whole generation of people But his own life became a total mess because There is no proper Guidance, there is no balance. There is genius, but there is no balance So This is a sad story that's happened to a whole lot of great talent in the world, particularly in America Because people grow up on all the wrong things at a very early age when I say wrong things There is chemical influence. There is alcohol. There is all kinds of distractions That we must understand this always that Our intelligence, our competence, our genius, all these things are important But for all of them to find expression in the world, the most fundamental thing is balance Now you're a dancer. You ask what is the significance of dance You can be as good a dancer as you are, but if there is no stable platform, you're not going to dance, isn't it? This is all life is Whatever our dance is, whether we're going to dance on stage or in the office or wherever in whichever form our dance finds expression in the world The most important thing is a stable platform within ourselves a balance If this balance is missing all talent, all competence, all genius, all You know, whatever exuberant things you have will go waste So both this personality is one that you mentioned Michael Jackson and also Elvis Presley are a classic example of this lack of balance When there is such a massive talent that you are capable of influencing an entire generation not in one country across the world When you have that this could have been used to really Create a new generation of people new sort of planet. Actually, you can you can do this You had the power to do that. It is just the lack of balance So I'm saying oh, Mary and you better dance your way as well as you can Whatever it is, maybe it is not like everybody can be a Michael Jackson or we don't have to imitate that But a new possibility in that sphere because music and dance has such a Significant influence on the youth But bringing balance to yourself and bringing balance to everybody who gets in touch with you This millions of youth who get in touch with you I think is a fundamental responsibility because when you have the power to influence other people's lives You must ensure that the influence is very positive and it is towards their well-being never against their well-being So having said that about you asked about dance. What's the significance of dance? Whoo, come on now Oh, come on now. That's that's the truth. I'm sorry, but I had to just Balance this is so important. This is so important So in terms of dance, see I must tell you this Uh We come from a culture where all our gods dance Everywhere else. God is a serious guy Very serious guy But in India if a man if he doesn't dance, we won't treat him as a god All Indian gods male and female always dance because if they can't dance he can't be good Because dance essentially means That life is happening at seed Dancing in a particular style is a different matter If you do not know this, uh My my daughter is a full, you know, full-time classical dancer Indian classical dancer I took her off education and put her into this because it meant so much to us that You know, what is dance is not just entertainment So if you're doing a particular style of dance, it's one thing Otherwise even a child will get up and dance when it feels exuberant It's the exuberance of life Finding expression is dance When body cannot keep quiet It wants to do something exuberant it will dance Well, you guys have might have evolved your own kind of Styles and I don't know you slide on the Do you do all that like Michael Jackson Was doing Those are all, uh, you know, but uh, what do you say many methods of doing it But essentially it's the exuberance of the physical self That the best way the body can express itself is dance The highest way the body can express itself is dance actually because that is the highest level of exuberance it finds So for this also, you know balance is most important If you want to dance well, the most important thing is balance, but most people dance only when they're drunk Ha ha ha We have a night long festival once a year nearly a million people last time 870,000 people attended this event And uh, it is telecast nationwide in over 100 channels in India and all this But uh full night from evening six to morning six No alcohol no drug none of these things are you know, anywhere come near us But entire night the whole crowd will sing and dance. Okay Wow I have to see you must come and see I'll ask them to send you some videos Yes, I have to I have the experience as well. That's that's amazing See without the exuberance of uh, when your emotion becomes exuberant, maybe you will sing I am not saying a practice singer or a professional singer When emotion becomes exuberant people want to sing even those who don't know anything to sing They will also do something. Hmm. Hmm. You know, at least they will hum When the physical self becomes very exuberant. It naturally dances. That's how it should be Well styles of dancing and other things are a different matter. There are culture in it. There is various aspects to it But uh, essentially dance is an outpouring of once life's exuberance life energy is exuberant So we don't consider somebody Divine or godly unless their life energies are exuberant and overflowing


In the last decade, significant developments have happened in the financial industry due to the use of mathematics and statistics to create new financial instruments. The course on financial mathematics, which is offered jointly by the Department of Mathematics and the Department of Statistics at stage 3 or postgraduate level, provides a thorough overview of mathematical and statistical foundations of modern portfolio theory, derivative pricing and the theory of interest rates. Modern jobs in the financial services industry make heavy use of complex mathematical and statistical models. For instance, on the buy side in hedge funds, people make use of statistical arbitrage and machine learning in order to try and gain that competitive advantage. On the sell side, there is more focus on pricing of derivatives and also on hedging risk. And for this, people make heavy use of stochastic partial differential equations. Above all, the most important skill is the problem solving skill. There is no better way to learn this than to study mathematics. The other aspect of the course that students find particularly beneficial are guest lectures from the financial industry. In my day to day work, every transaction we undertake is underpinned by some form of mathematical model. To be able to understand the situation and interpret that is critical for the success of any transaction. When looking for employees, we're looking for someone who stands out. A person with financial mathematics background offers that point of difference. Students will find the course a useful stepping stone if they wish to pursue international postgraduate financial mathematics programs. The course will also feature the programming language R. One thing that's good about the course is that we use R. This is open source software that's been developed here actually at the University of Auckland and is now spread throughout the world. There is enormous scope for data analysis, portfolios, option pricing. It's the main vehicle used throughout the financial services industry because it's open source and it's incredibly comprehensive. Of course we can't predict the future but we can perhaps arrange to gain advantage from it when it occurs so we can make decisions today about how to set up a portfolio or other investments so as to structure the risks that we're taking and advantages that we might gain from them. Students come from the Business School of Mathematics because it will give them significant opportunities in the workplace. I think that ability to look at how a formula works to piece it together bit by bit is something that I learnt in 370 and I know that I use that ability every day to help supplement what my team is doing. For students who want to pursue a career in the financial industry this course will help you to stand out. It will equip you with the tools necessary to work in the high-end sector of the financial industry. It will also allow you a competitive advantage if you want to study at a post-graduate level but secretly at internationally recognized master's programs specializing in mathematical finance.


Here we go All right, it's the next day and this looks perfect I'm still doing a bit of prep till I get the resin Pour it and actually the pouring the resin is actually very very fast It's all the prep work and making sure that it works well that really takes the time So there's a tiny bit of an overhang on this side. So I'm just gonna Turn it over and trim that with a knife.  So just take this board and just nice slow patient cuts sharp new blade Makes everything easier Beautiful the next step is we're gonna put tape on the bottom of this actual piece of wood Because when the resin drips over the side It's gonna create little drips all the way along because we'll have it raised up But if we have tape just painters tape it'll just peel right off the next day And you'll have a nice clean tidy edge if you don't put the tape That's okay. You can file them off Sometimes you can even stick a chisel underneath the drops and just kind of pick the drips off Just do it right do it once and use painter tape on the edge. You'll be happy you did So it has a cure time of 24 hours meaning that after 24 hours It'll feel hard to the touch But if you pushed with a fingernail you could actually put a little dent into the surface It'll self heal so you won't stay there But the actual full cure time is three days till it's really hard like stay until you want to start tap dancing on it, right? Which I recommend you go to the belly. Nope. Do you ever dress up in tights? No comment Again, how long that take two minutes one minute, but it's worth it Just do the prep work, and you'll save yourself more time later. That's the truth for anything in life, isn't it Jeff? I actually wrote a book about it. It's called do the work prep work, and it'll make everything easier Okay So the next step is I'm going to raise this up just a bit off the surface again so that this doesn't get stuck to your actual table you're working on and Then I want to make sure it's perfectly level. These are painters triangles As you can get at a hardware store in the painting section usually now. They're great, but they are ridiculously expensive So even though I have those just look around your house and steal things from your baby These work fine. All you're doing is lifting it up You don't have to go and buy these but you can if you want to and then the last thing that we want is These are called wood shims There's basically a piece of wood. That's a really long thin triangle. So if I need to raise up a corner I Have my stand and then I have a wood shim underneath it if I need to raise up this corner You just slide the wood shim in further, right? Fantastic little trick. So everyone's got a smartphone and there are tons of level apps This is a good level because it has both directions at once this one's just called dual DUAL level What this level is telling me is I could raise the side for this to wave me up a bit and This corner has to go up the highest So I'm gonna raise this corner the highest and then I'm going to also raise this corner quite high To bring up that whole side. Look at that Next step once you're done ret putting resin on you have to let it sit overnight. So I Always start by knowing what I'm gonna cover it with When it's actually done and there's hair dust Bugs all those things can go and land in your art resin and Rune it basically right because we resin a lot of stuff We just have this room here full of different shapes and sizes of boxes, right? You know like lids to like paper boxes and stuff just save all that kind of stuff because it becomes very useful Great. So we're level. It's raised. We taped up the bottom Next I'm gonna put gloves on because I'm gonna start using resin these ones are called Nitro gloves and they also have some Chemical resistant properties you can wash art resin off of your hands, but it's like a glue So this is mostly so that you don't get all sticky and dirty, right? There's no VOC isn't it so I don't have to wear a mask and it's very low odor, which is fantastic So I'm gonna need a mixing cup my resin and a stir stick Yeah, you ever so organized that you lose things. This is a resin spreader. It's great because see these notches See that so imagine if let's go down to the painting You'll see how this goes you're dragging the thick resin across the surface But these notches allow for some of the resin to stay, right? So that's spreader and then I pulled out one of these stir sticks and these are great too Because you can also peel the resin right off of them and reuse them. So we buy our mixing cups At the dollar store. Let me show you something. So this is a juice jug What's cool about this is I finished resigning something then just put it upside down on your even on a scrap piece of paper A piece of cardboard But watch this. Hopefully I'll peel up Look at that. The whole thing just peeled right out and it's a brand new Macing container. Of course, you have to do some measuring So having something with lines on it sure does help the good thing about our resin is it's an equal one-to-one mixing ratio So you can just visualize how much you need that being said how much resin do we need to cover this? That's a great question, David. I'm gonna do some math. So on art resin calm at the very bottom There's what's called the resin usage calculator and you just type in your dimensions and then you will see how much resin you'll need and What size bottle you would be best for you to purchase so you're not wasting too much Of course like anything the more you buy that better the value and once you do this You're gonna want a lot of resin because you're gonna love it And this is the strongest glue in the universe resin hardener Mix the two together you have 45 minutes to work with it