Okay, hey everyone, thanks for coming to our talk today on rapidly scaling for breaking news with Carpenter and Kata My name is Mel Cone. I use any pronouns. I'm a senior software engineer at the New York Times in Delivery Engineering, which is a platform team that is building the shared platform in New York Times and I'm specifically on the team that manages and Maintains and adds features to our shared Kubernetes clusters in AWS Hi everyone, my name is Deepak. I'm also my pronouns are he him. I'm also on the same team That better. I'm also on the same team as Mel working on the same shared platform Get right into it. So what's a breaking news alert? So a breaking news alert is a push notification that we send our users And I'm sure you've gotten some of these or might even get one get one during the talk But as you expect this leads to a sudden influx of traffic of people to the website and the mobile app And this leads to them interesting traffic patterns we see sudden spiky traffic and this is an example of an average traffic spike usually caused by a push notification We regularly see traffic spike to two to three X and this happens within a minute in The past we've over provisioned to deal with this, but that costs a lot of money I mean want to minimize amount of infrastructure that we're using so you want to need a way to scale down and scale up quickly This is another example of a traffic pattern that we see This is a result of a daily release of a game And this also leads to about a three X increase in traffic and a very short time span This traffic is regular so we can we know when to expect this and we know how long it lasts So we can we can also scale particularly scale for this So now Mel will dive deeper into our architecture and what we're scaling Okay, so why do we see this traffic spike the New York Times has a unified HDB ingress This is a relatively recent thing that handles some of our internal and external traffic It's the goal is to have everything eventually pass through the ingress But the ingress controller needs to scale to handle the traffic spikes from the B&A's and games so we often see the three X increase in terms of three X number of replicas of the unified HTTP ingress and This is running on the Kubernetes clusters So when a user opens the New York Times website or app the request is routed through our HTTP ingress The ingress sends the request to the corresponding upstream for the appropriate services That said these services often need to make calls to other New York Times services behind our ingress For example, the front page service will make a call to our auth service to check if the user is logged in and has a subscription or a call to our personalization service which Customizes the organization of stories presented to a user and then those services need to make calls back to the front page Service this means that there's a spike in user traffic But there's also a spike in internal traffic, which makes the traffic spike for ingress even bigger So in this diagram, this is a portion of our shared platform specifically the shared Kubernetes clusters in AWS We run a multi-tenant Kubernetes runtime with clusters in multiple regions and environments including a sandbox cluster we use for testing changes and teams are Provided with a tenant cloud account along with access to a team namespace in the cluster in the dev stage and prod clusters So whenever we need to scale we need to scale across multiple clusters obviously prod is a bigger scale, but it's all of them So now we'll talk about how we're scaling with carpenter and how we're scaling our nodes So why wouldn't we just use autoscaler? So first Carpenter is a native Kubernetes autoscaler So it's installed as a CRD and we're already using a Kubernetes-based workflow for installing our other parts of our cluster So we're able to configure carpenter via YAML as part of this workflow With autoscaler we had a lot of terraform that have to be run separately to be installed on each of our environments So and that had to be maintained separately now. I'm going to hand it back to Mel to talk further about carpenter Yeah, so Carpenter is able to scale at the pod level by finding the optimal way to pack pods into nodes using a bin packing algorithm Well, the cluster autoscaler is more naive in its approach It scales at the node level based on overall resource demand So for example if you have resource limits on a pod and you need to increase the number number of replicas It looks at the resource limits for that pods and sees where it can schedule it There's a bit of an algorithm there, but it's more naive and then Carpenter also takes into consideration all instance types that meet the requirements you specify So we're going to show an example in a minute But you can give multiple instance types as options Which is what really gets a lot of the power of carpenter because carpenter has multiple algorithms that are like optimizing from a cost Perspective which thing to spin up so if you look at the diagram and Carpenter if you have pending Kubernetes pods if they can be fit on an existing node They go ahead and get scheduled by the scheduler if they're not then carpenter will spin up a new node That said Carpenter will do node consolidation and so if it realizes okay I can get a bigger node that's cheaper than the total cost of the other nodes then it will spin up that node and move things over there So Carpenter uses groupless auto scaling so the migration was a little complicated We went from managed node groups to this groupless auto scaling that leverages the security groups and Security groups and the subnets it adds a tag so that it knows where to spin stuff up But it's groupless because of the different instance types and It can't be used with the cluster all out of scalar. They explicitly say not to So Provisioners which now as a very recently are with carpenter graduating to beta like a week and a half ago are being renamed to node pools which that's gonna be fun when we upgrade And they're the equivalent of a node group in EKS So this example right here is one of the provisioners that we use in our cluster specifically for tenants So we have a bit more flexibility here than say the provisioner for our unified HTTP ingress and you can see you have the different instance types and you have the The part for selium specifically like the startup taint and then you can we also have our nodes expire every 14 days Part of the reason is spot instances Can't be replaced and like changed to a different size They're just deleted and so we want to cycle the nodes and then it's also like fun, you know chaos Engineering to have our node cycle every 14 days, and it's at max 14 days. Sometimes they go away sooner than that and then we have consolidation enabled which you don't have to do and I'll go into a bit more, but we don't do that for our unified HTTP ingress So Consolidation so this is kind of like the big way that we save money So carpenter has three ways to consolidate nodes It reduces the cost by either deleting or replacing the nodes Either if they're empty if their workloads can run on other nodes or if they can be replaced with cheaper nodes and by cheaper I it could possibly mean one node replacing multiple nodes or one node replacing one now So in order it goes through and it sees can it delete any empty nodes? Or it tries to delete multiple nodes and launch a cheaper single node Or it deletes a single node and then tries and replace with a cheaper node on the right hand side This is something taken from the carpenter docs. I took a portion of it. This is actually the node pool which has all these new pieces in the spec and everything too But it allows you to define how disruption works So you have this consolidation policy either you say when under utilized or when empty and then you say how long it will wait before it makes a decision and then you can also Set it to expire after a certain amount of time, which is the same thing that you saw on the previous slide with the provisioner that we had the 14 days Okay, so this is code taken from the carpenter get hub repository. I'm not going to walk through like every line Just the high level. So this is the cool part with spot instances So part of the reason we migrated a carpenter it was a lot easier to do spot instances there than it was to do with our managed node groups and We really wanted to use spot instances And so this filter unwanted spot what it does is it filters out spot types that are more expensive than the cheapest on-demand type that We could launch so if the on-demand type is Going to be just as the same price or like cheaper then doesn't make sense to do a spot instance We also For anyone who doesn't know a lot about spot instances you essentially can like bid on them And so people could like outbid you we don't bid and so it's kind of like an I even how we do it So it's either it's cheaper. It's not the end and It also is going to find like the cheapest offering for the spot instances From the options that you give it So if you don't give it as many options, you're not going to have as big of a like bang for your buck because it doesn't have as many options To choose from and that also makes consolidation harder if you don't have as many options to choose from Cool, so now we have a quick demo and on carpenter works It should autoplay and I can't see it on my laptop, so I'll explain it from here But on the left side just have a quick HTTP deployment that I'll be scaling up and on the right side Just tailing some logs from the carpenter controller so we can see what carpenter is doing So right now you can see that carpenter has might be a little small But the carpenter has some spot instance pricing that it's aware of it's aware of what instances it can pick from when scaling and on the left side Just scaling up say we have a breaking news event something big Scaling up to 500 Replicas and we're going to watch that scale up On the right side once once the pods scale up and carpenters aware You should be able to see that carpenter will decide first what note which go through the algorithm see Which note pick an instance type and see which it can provision And we should see on the right side Yeah, so now we can see that carpenter has decided to provision a new node it picked a specific instance type And on the left side we can see now that the node is there And we can also see And we can see that that you provisioning a second note also to meet the demand to schedule all the pods and Then if you look at the tags on the on the node We can see that carpenter picked a certain instance type and it's tagged by carpenter And now the important part is that we want to be able to also scale down if we scale up, so I'm gonna try scaling down on the left And scaling back down to one Yeah, now we're scaling down. Yeah, this part I was trying to see if there was pod the node that command didn't work, but yep Waiting for the node to be ready, but yeah, I'm just waiting for the Waiting for me to scale it down Sorry, we didn't do a live demo. We didn't want to test the demo gods, but this was recorded yesterday Close enough to live Yeah, now we can see that this is me showing that the pods have been scheduled on that node that was provisioned And part of what you're seeing is it takes a little bit for it to decide to disrupt the node and get rid of it It needs to like move around the pods and then also get rid of all the demon set pods Yeah, and now we're scaling it down to one and then we can see that the deployment has now as one replica So so we should see that carpenter is not gonna go through that consolidation algorithm where it's gonna either Delete the node or try to replace with the single node But in this case it's likely just gonna delete both nodes and we've scaled down from 500 pods To one pod so on the right side soon We'll see the carpenters and decide to delete each of those know that it's created and we'll be back to our original state Very quickly Part of the other reason we decided to use carpenter was also it's faster than the cluster autoscaler so we did like some benchmarking and I want to say cluster autoscaler maybe takes like two minutes and Carpenter is like 90 seconds sometimes even 60 seconds a little longer to schedule something on it And that sounds like not a lot But when you have the traffic spikes we have that is a lot and also is still not really enough Yeah Cool That's the demo now. We'll talk about more of Thank you And now we'll talk more about how we're scaling with Kata So just to give an overview of Kata Kata as an event based scaler and and it's extends the HPA So it configures the HPA for your deployment instead of it doesn't replace the HPA So scaling is set up via scaled object and the scaled object defines which deployment is the target and then it also Define what the trigger is so triggered could be a bunch of event sources that Kata provides out of the box So let's revisit some of those traffic spikes. So this is the this is the release traffic spike the predictable traffic spike So we needed a way to make sure we can scale up in time and we know when these spikes are so we're able to use Kata's Cron event type to scale our ingress controller So essentially we specify specify time for to scale up to a certain replica amount and then Kata's Patches the HPA to scale it up for this time period and this sort of been very doable with the Kubernetes Cron job also But Kata allows us to define our scaling logic in one YAML and it also modifies the replicas for us So this is just a diagram combining carpenter and Kata and how we scale together So when I want to when I drop happens or at least happens We can see that Kata first adjusts our ingress controller replicas so it modifies the HPA which killed up to deployment and then carpenter in response has to be able to schedule all of these pods that have been That have been created So carpenter adjusts the node pool that manages the ingress controller and then optimizes for Kata picking the best in in sense And once the Kata Cron event ends we know that Kato scale down the deployment and in response carpenter will either delete the nodes or replace the node cheaper nodes and We've also explored other ways to scale. So this is the revisiting the breaking news alert So we know when we're gonna send a breaking news alerts We need a way to like scale in response this event, but these aren't predictable We don't know like it's gonna happen at the same time every day So these spikes are more random and more frequent so we can't use the Cron solution So Kata supports an external push trigger. So we essentially set up the server ourself to Send alert to Kata So as you see in this diagram we have We first have like the breaking news alert going out after the article is published No, the webhook server that that the team that sends out the notification sends an alert to and then in order to set up the External server we have to set up. We there needs to be a gRPC server that Kato sits to so that's where the external push goes So the the request gets sent to the server webhook server Which gets sent to the gRPC server and then once it's a gRPC server Kata knows to scale up and it would be a similar type of Solution whereas Kato would scale up to an X amount once it receives that alert that a breaking news alert is going out And now we're going to discuss some lessons learned from this process So cost savings We were looking to save money by not over provisioning as much So this is really hard given the way that the breaking news alerts work And what Deepak was talking about with like getting a heads up the heads up is like a couple seconds And when your spike happens in like 20s to 40 seconds and lasts for only two and a half minutes And is so huge and it basically means that like by the time you scale up you have to scale down So, you know over-provisioning was kind of what we were able to do and to be honest We still do it somewhat but a lot less now And a lot of that is to do with the cron job that he mentioned but also the things in carpenter So there's a no consolidation So that means less over provisioning right because it's going to pick like the cheaper nodes It's important that we have like a variety of nodes that it can pick from otherwise like if you say, okay You can only do this specific instance type then it can't do anything It's either like okay It's spot or it's on demand the end and we do have some provisioners that are only on demand because they're like more critical workloads Yeah, and so With the cron job, we don't need to over provision things as much and can adjust the scaled object max replicas based on observed traffic patterns So for example Recently with some of the releases of the newer games there has been like a record number of downloads of games And so it was like, huh, I wonder what's going. Oh, okay, cool So because we scale up ahead of time We also have fewer alerts and things breaking in the cluster since it's not attempting to scale on the fly And we also have more instance types to choose from so with the managed node group You have like an instance type with carpenter. You can give a set of instance types So further improvements. Oh, and I forgot to mention we saw a 25 decrease 25% decrease in cost, which is exciting Further improvements. So our unified HEP ingress Because of how important it is and because of like the networking and compute means has one instance type that it can do And also we only do on demand So obviously we can't really leverage carpenter for that and also we don't do node consolidation for it So we're already looking into ways that we can either like have more options for nodes or maybe like switch from like a network Optimized instance type to compute optimized instance type and just like really do more investigation into that and reminder that this unified HEP ingress is like maybe a year old year and a half. So like yeah and then We also the external BNA scalar. We're trying to figure out like how To do that because of the little bit of heads up that does it's not really worth it to like implement a whole scalar when it's not Even a little less Overprovisioning would be good, but it's not worth it given the like little bit of heads up and the way that the workflow works is It's kind of like unexpected There's a whole process of getting an article approved and I think it's like in slack They post like hey, please give us a breaking news alert now And it's like there are other teams that are doing different things and so by the time it gets to us it's like And then we also want to have potentially Kata as a service for tenants So right now we have Kata rolled out to the tenants in our clusters But we don't really have many who are using it And I think part of the reason is because they don't know necessarily how or why I know deep luck said that We they have different metrics that you can scale on so there's a lot of stuff built in so datadog All the different clouds like CPU pretty much anything you can imagine and like we said you can write your own So ideally we'd have something like in the shared platform where it's like Hey, do you want to scale her for this deployment and like Give us a bit of the configuration and then we'll do it for you so with Kata and Carpenter together we were able to save significant money and Our lives are a lot easier to get paged way less often Okay, so the other big thing is the shared platform. So shared platform is Centralization and standardization, right? So Before the shared platform people were running and managing their own infrastructure Understandably there's going to be differences in that infrastructure not only because of the needs of the team But also because of the expertise and what the team prefers But Now that we have A lot of our services on the shared platform is to shift really the shared cluster We're able to establish a baseline And more easily compare across services and then detect outliers So for example, we can ask questions like why does this one service have significantly higher rps Sometimes the answer is the scale is larger and that's fine Um, but other times it may be because there's like not an optimal traffic flow and something needs to be changed Like maybe it's making too many too many requests.  So I guess I'm just gonna wing this. Um So I already said some tenants make significantly more requests I talked about earlier that's kind of like you have to make calls to other services Within the new york times and then they make calls back and how that contributes to the spike But there are some that are making more requests than others and it's we haven't like fully figured out what's going on But it's a lot easier to tell and also with there being Us scaling the architecture rather than the tenants needing to And like I said this like standardization We have a much better idea like okay, this is potentially something with the cluster versus this is something going on with the tenants Like this is expected behavior All of that. Um And then it also means they don't have to worry about it. So in the past a lot of different teams Scaled up for these breaking news alerts or for the games release because like for example like off or uh I there I gotta be careful off or say like personalization or algorithms stuff like that Uh, and now they don't need to do that anymore I think we mentioned early on that not all of our services are Even behind the ingress which you don't have to be running your service in our shared cluster to be behind the ingress You can be elsewhere. Um So we're hoping that the more people that move to the platform That that means it'll be easier to see some of these outliers and then it'll be good for everyone They can go back we can go back to the team We can advise them as sneeze and say like hey, we're seeing this Can you tell us why this might be happening? Can we help you improve your service? Okay, um, so with that, thank you very much. Um We had a couple talks earlier this week from other of our co-workers They already happened, but when I argocon and went at multi-tenancy con I encourage y'all to check it out if you get the chance Thank you everyone Very very good talk one of the best talk of the whole coup con seriously It's the last day the last talk And like it's full Appreciate it. It's also both of our first toxic cute con So still I have a question So you you're using hita it's scaling the number of it's using the hp to scale let's say 500 pods Some of them are going to be scheduled because you already have the nodes But then carpenter is going to create new nodes It can take some time So do you have any way to kind of or how do you manage to have some resources? Like when carpenter scale down the number of nodes you still have some Balloon you can you can schedule pods directly Say that last part again The last part of it you were like, how do you manage to blah? How do you manage to The the the number of nodes or the compute power you still have when carpenter moves down So you can accommodate at least some of the spot some of the pods were when a new scaling happened Okay So first of all part of the like benefit of the crown job is rather than doing like node by node It can start up like a bunch of nodes Um, and you're specifically asking like when it's scaling down how we still have new power So the answer is it doesn't so it's it's basically like they're kind of working separately So carpenter is working like on the node level and then kata is working on the like pod hba level So there's not going to be nodes just sitting there unless there are nodes that are underutilized Um in general because of the node consolidation and everything We don't really have many nodes that are like at a high under utilization unless it's the hba engross So What that means is as it's increasing the replicas with kata. It's like, okay I need to do schedule this many more replicas Carpenter sees it's not interacting with kata. It just sees that it needs to schedule more pods And then it will spin up the nodes which is part of the reason why this is like difficult to do with a bna because Like like we would have to either like purposely create them So let me rephrase Yeah Can can you tell carpenter to keep that number of nodes when it's scaled down? Yeah, yeah, yeah, you can so you can say so you can either say like uh You could say keep them around for longer Um, you would have to have like a like different configuration for the provisioner So potentially like have a different provisioner specifically for This event which you can't overlap the provisioner So you can't have like a provisioner that would apply to where a pod could be like Scheduled based on one or the other and so We just And I just lost my train of thought But yeah, so it's basically like we don't have extra ahead of time It's just like They're there and then they're gone and we would have to do extra work to keep them around And in general, we don't really want to keep them around It does it's pretty smart about like it's you saw it took a little bit of time for it to scale back down Um, but yeah We have seen some errors on scale down because you can imagine like it's going like this and then going like that And so it's a lot Go ahead. Thank you. Hello guys I have a question like is it okay to run cluster after scaling and carpenter at the same time? No, you have to run only one tool correct Um, so usually what you do is you scale down the cluster autoscaler to zero And you do that after you have carpenter set up so you can't have the two together because They are both scaling the same thing in different ways and so they don't they explicitly says in the carpet documentation to not have it Okay, thank you. Thank you um, have you had to Make any changes to Cube scheduler or have you observed any weird interactions between carpenter and cube scheduler? Because I can I with carpenter has been packing and cube scheduler doing whatever the heck cube scheduler does Uh, I can imagine that those are going to fight Yeah, yeah, so we haven't made any changes to cube scheduler um Some of the stuff that we've seen that's kind of weird is we had to like implement these interruption cues specifically because there was not like graceful termination of stuff, especially with like the demon sets and so It was essentially like sometimes weird things would happen We're like it would struggle to like drain and delete the node or for some reason like something wasn't scheduled Wasn't at the right number of replicas and so that's how we've gotten around it. But yeah, you're totally right. Um, so I think it's kind of like Wonky stuff happens, but it's like the way it is right now It's like okay the carpenter can kind of like figure it out and fix it as in like Say the cube scheduler like naively schedule something on a node and carpenter's like I don't know. I want one node for all of that instead Then it's able to like delete the node and then schedule things on that node And you saw in the logs that it will show you like how many pods it's scheduling on the node. Thank you. Yep Fantastic talk. Thank you. Are you concerned about capacity limitations in the cloud? So you're moving from free scaling from having an over provisioned by the sounds of it And you're now scaling on the arm based on breaking news as we enter the holiday periods the clouds get busier Are you concerned about running out of capacity when breaking news occurs and kuda carpenter can't scale anymore? I'm going to answer this very carefully Yes, um I think in general. Well, there are a couple things to consider. So first of all, like we are currently migrating more people to the clusters Um, and the ingress which means more scaling needed I would say holidays are not as big of a thing. It's really the election And I would say the big one first big one that's coming up is the iowa caucus, which I think is in january um, so in general what we do is we have A lot of stress testing and load testing and in particular this like a separate thing, but we have an in-house Load testing setup specifically using k6 Which allows people to like configure their load test and they can run stuff themselves But there's a whole team that does elections readiness to make sure that we're like in a good situation for this But yeah, you're totally right That's brilliant. I would say that's a fantastic talk for the next q-com. Thank you. Thank you Hi, uh with mixing up different instance sizes. How are you managing? Demon set scaling like setting their resources is just like statically to the largest instance size you run or Is there like any dynamic scaling there? So I think that So specifically what it'll do is it it doesn't consider like Aside from knowing how much capacity it needs for the demon sets it doesn't consider that when it's spinning up new nodes So it's just like what it knows. Okay. I need like maybe there are like seven demon sets So I need seven pods that are like this resource constraints that need to go on that node But it will spin up the new Node based on the other stuff that needs to get scheduled. Yeah, I'm asking like um For larger nodes, you may need more demons more demon sets resources than smaller nodes You just provision like for the larger sizes at all. Yeah, so we This is not ideal, but currently what we do is we've been shifting it. So basically like Carpenter doesn't do stuff exactly the same every time But it seems to kind of pick similar instances And so what we've had to do over time as more stuff is migrated as the like node sizes increases We've have to change those resource limits and We've actually recently removed the cpu limit for the datadog demon set specifically because of the way that like cpu starvation fails It's like a silent fail compared to memory starvation Um, and I think ideally and I know we're doing this in the shared cluster we have in gk8 We want to just like completely remove cpu this is me speaking. I would like to completely remove cpu limits Um for the deployments all the deployments tenants and everything because There's been some gnarly stuff because of the cpu starvation Thanks Hi, I was wondering um as you started migrating to this like really fast cluster auto scaling How much did your application start of time? Uh play into that like if you had a few applications that were really slow to start up that were really like blocking it Would you see something where all of your applications would like, you know, be high cpu usage and then it would scale even more Yeah, so that's a great question. Um, so part of the difficulty is like we want to live in a world with the shared platform Where people don't have to worry about the infrastructure That's not the world we live in and honestly like if you have a shared platform like that congratulations because that Um, and so essentially what we've had to do is be really clear in our documentation And then also when we see like outliers and advice people and basically be like, hey So here naively is what kubernetes is doing like pods can come and go all of that You don't know where they're going to get scheduled. So you need to be more resilient So we've had stuff happen in the past where tenants are like having to change things But usually because it's in one place we look at it we go help them We talk to them we get an understanding and we've also done some of these migrations ourselves like with the team Um, but yeah, it's a great question. It's it's going to shift Cool. Yeah, thank you I have a couple of questions. So the first one is uh about the consolidation So I see you're no poor configuration. You set up the consulate dead after 30 seconds So have you ever seen um the no that got the Consolidation replaced through the small no and in the next five minutes it terminated send no and spin up the Big sizer And that created a lot of trigger a lot of the part get the rescheduling Yeah, um, so That specific thing maybe not but like something similar as in we've definitely seen situations where like especially because of these like really fast Short-lived spikes where nodes will start get started up and then it'll be like, oh shit. Wait. I need like I have now like five nodes. It could be one node. I'm going to consolidate them I would say it doesn't happen that fast And there's also the fact that like we're not doing node consolidation for the hdb engras And so we don't have to worry about that happening with it And that is the main thing that has to scale just because of the scale with everything going through it Uh, yeah, I guess because we since we have a continuously delivery um every Minute so we have like a 10k user running our platform So that deployment is actually triggering the consolidations very frequently and users try actually complain for that Um, the other question is about your spark instance. So how do you dealing with the spark instance get recycled from the aws? As in like when they're like recalled you mean or when they're in cycle to recycle to ask you Your spark instance get back to the aws pool. Yeah, um, so The answer is we we just the The interruption queue has been important the interruption queue. I mentioned mentioned before but in general I think we would have to do like significant more work to Deal with that gracefully because we're not doing like the bidding and all of that So in general what will happen is it like gets recalled or whatever it's called That node goes away and then it like gracefully terminates and then it's going to like spit up new nodes to schedule this stuff And to my knowledge, it's not going to like de-schedule the pods before it like spins up, you know, but i'm not 100% on that So um regarding to the k dot component This for for the predictable use case for the breaking news. What about the Unpredictable news like a earth grace gunshot Is that also possible to use the k dot to scale that Yeah, so currently we're mainly using it for the predictable use case, but like that's the custom external push Thing trigger I was showing that's what we're prototyping and that would account for like, okay A team tells us they're about to send out like a news notification They know they're we're probably gonna get more traffic that would we would account for that And then we and other than that we used to use k dot for like the normal cpu and metrics that you can do with hp So you rely on the some external matrix to trigger that Oh, not currently we are prototype. That's like what we're working on, but we will Just to like walk back to what I said during the presentation So the reason we're not doing it right now is because it's so little time It's just not worth it Like I think it's literally a couple seconds and then also like we're one of the downstream services And so it's like other stuff is doing things Um, I know that like the team that sends out the push alerts have an has an idea of audience that they're sending it Out to and that's really helpful But it's it's basically like ed has been worth it There are other things to work on but we definitely want to do it And honestly ideally we would probably talk to the folks sending out the push notifications Or the ones requesting them and be like we really need a heads up because I think historically maybe like years ago We got like 30 minutes now. They're just like whatever a couple seconds. There you go So sorry last question. No, these are great questions about the A carpenter controller. So when I first deployed that I actually Mace up a lot of configuration. It's leading me to the terminate all of my Cup no, yes, I remember when we migrated Have you had any Alert or how do you prevent that? Yeah, um, so we definitely have alerting. Didn't you set up some of the learning for us? Yeah, we have we have alerting and we also have multiple node groups We have or node pool sorry a note a node pool for the actual ingress controller, but we don't use spot instances because that's more That's more resilient and then we have like other node groups to manage tenant workloads So I think that's also we separated. Yeah, they're based on the carpenter log or you based on the carpenter controller matrix The controller metrics matrix, okay And we also I don't remember the specifics of it But there are like certain anomalies that we look for that we page on or at least post to like a slack channel that we can Look at so it's like ah, you know this this node hasn't gone away like what's going on stuff like that But yeah, it's not it's not perfect. It's not ideal. Cool. Thanks. Thank you Carpenter is considering the node types in order to do the auto scaling. Is it also considering the region or the zone? You can do that. I was actually something I didn't talk about. This is a great question So in general, I'm pretty sure it balances stuff There's a bit of weirdness in that especially with these scaling events and that like say we have you know US east 1a b and c right the availability zones It will balance them across and I think this is something like in the configuration that we did The problem is is that if it's like, okay, we have 20 and one 40 and one and 20 and the other It's like well, let's spin up 20 and the other two But then if it like takes them down then it's like wait We have to get rid of the other ones and then start up new ones And then it's like this constant loop of like wait, we don't have the right number That was one of the weird behaviors we saw So I'm trying to remember what we did for it. I think it just got into like a gnarly state at some point and we just had to fix it But I think part of that was for like the really the like regular predictable spike And so we were able to kind of be like, okay, wait I think there was some setting like with rebalancing so that it wasn't as naive about how it did it that we were able to use Cool. Thank you. Thank you So with the really short discreet Traffic Spikes that you get have you And I'm not really sure how like Amazon exactly like bills like when it starts when it ends Have you all thought about integrating like fargate? Which might have a faster like startup and shut down I think initially we did consider fargate, but we never actually used it in production I think we might have done a poc So I'm not sure I think initially we might have if you remember If you if we've used fargate To also solve the similar type of problem A great talk by the way, it was one of the most valuable ones. I think so far that I've I've been in Because we use carpenter. We we're just starting our journey with that. Yeah Have you lived through any availability zone gray failures and how does carpenter handle that when you say gray failures? Can you be specific about what you mean? Yeah? Oh, I mean like an availability zone goes down or something launch and um I don't think we've had Availability zone gray failures or at least not to our knowledge and now I'm like we should set up an alert for that Because I would not be surprised if that's not happening. So maybe I should go do that now The fun part of that with amazon is you don't get an alert you find out about it later I would set up an alert to like figure out their shenanigans is what I'm saying Like I'd have to look at it and see what's going on But that's like a great point like I I should go look at that so I can figure What I would think you'd probably have to do is is Tell carpenter to not deal with the availability zone and do it manually which is a problem But I was just wondering if you've seen that before yeah, and for what it's worth We we do have like multi region. So we have clusters in multiple regions and as of recently we have a cross region service mesh and The unified ingress is set up to fail over So that's not exactly because if there's like one availability zone that's having problems But if there's something like really wonky happening it will just fail over to the other region And so maybe then but like yeah, it's still not great. Got you. Thank you Thank you for the great talk Does carpenter take into account compute savings plan if you have someone That's a great question that I actually thought about today when I was uh doing the talk and I was like I don't know the answer to that and I don't have the time to look it up I don't think it does especially as someone who is like Weighted around a lot and aws cost explorer which is confusing very confusing when you do savings plans and spot instances and all of that I don't think it does um and then also part of the problem And I know for sure it doesn't in the sense of if we tell it hey Do these like you can pick from these node types It's not smart enough to be like you have a savings plan for this specific node type So i'm going to make sure I use that node type. It's a little more naive, but that's that's a great question Can we maybe tune the weights on the node types to say that this is more preferable? I don't know. I want to say this is something that came up before and either like we couldn't do it or it was not Like we couldn't figure out exactly how to do it because like mind you carpenter just graduated to beta Um, so i'm not sure Thank you I have one more question I know you guys will mention about scale up the by parts right or by nodes I'm wondering what you guys to do with other resources like a db or Like a messaging system. How do you guys scale like database or yeah like other resources not only parts or nodes As in like how do we scale like say a database that's not running in the cluster that like an application is communicating with Like I was thinking like uh, when you guys get a break in use, right? Like not only it requires a pot Level of the resources, right? Yeah, maybe other resource dependent resources that you guys may need How you guys scale those like So There is like the like built into the hpa, right? There's like the cpu and memory But in terms of like the database all that i'm not sure But I do know that for stuff that's outside of the cluster Which is usually like the more common thing is that people are running like their main app in the cluster And maybe they're like communicating out to a database or something that a lot of times they have to scale up ahead of time There are like multiple other teams are dealing with some more problem. Okay. All right. Thank you. Thank you Sorry to keep everyone here So in your demonstration you kind of mentioned that use the cron job part of key to however I believe it does require that you give it like an actual Number to scale up to so like how did you derive that number to use hysterical data? Um, I'm guessing this is for the regularly occurring spikes. So Yeah, we use historical data and it's like obviously like like slightly over provision So it's like historical data plus like 20% But yeah, it's a set number of replicas that we scale our ingress controller up And to be clear the number in the slide was not the number we do because I'm not going to put that in the slide Kind of on the flip side. So when you have those sudden spikes, what do you actually use to determine the number of pods to scale up to? Is it a cpu memory like resource thing? Or is there an external metric because you were saying what previous you got like a 30 minute heads up and now you get a few seconds Um, that's assuming that you have some sort of coordination between the breaking news Alert versus like if there's an earthquake setting, you know, everyone's going to check their phone and google You know, is there an earthquake near me? So how do you manage the sudden spikes rather than the regular spikes? So the answer is um, just hope for the I mean, okay. That's a little bit of a person location But we are like continuously trying to decrease the amount of time it takes to scale So I think at this point we've we've gotten like most of what we're going to get out of it But we're hoping the stuff improves over time and I do really think that the main way that's going to like Help with this problem is either getting a heads up from the team that requests the push alert or uh Doing this external scaler in kata. That's like the custom scaler that uh subscribes to the web hook Because they're really unpredictable and also you don't necessarily know what's going to go out I think the only thing that we have is like a slack channel and then maybe you could get some idea But in general like if an election is coming up, we're like, oh, we know we're going to get more traffic So we're probably going to over provision some because like I'm sure for when the debate was happening Right, we probably saw a really big spike. I wasn't paying attention because I was on call and I was here, but yeah And also the the expected traffic we were showing that scale that spike is much quicker Like that's like a few seconds versus like 30 seconds. I meant another one. So it's like we're able to scale for that Yeah, just curious if like cpu or memory was an adequate signal to scale up in that rapid time period because you know You have pod startup and scheduling and all that stuff. So just wondering. Thank you. Thank you


So we will be talking about the standard enthalpy of combustion which is denoted by this delta H C naught and there are some clues to what this means here itself. The delta H is for enthalpy, C is for combustion and the naught or the circle here is the standard part and we know that when we talk of combustion we are referring to a reaction with oxygen. So what the standard enthalpy of combustion means is that we are calculating the enthalpy per mole or per unit amount when the reactant reacts with oxygen and all the reactants and the products are in their standard states. So let's take an example to understand this better. So here we have one mole of butane which is C4H10 undergoing a combustion that is reaction with oxygen giving CO2 and water. This is a reaction with oxygen and also the reactants and the products are in their standard states and because this reaction is balanced we know that the enthalpy here will be for one mole of butane. So I can say that the standard enthalpy of combustion for this reaction is minus 2658 kilojoules per mole. Now a couple of more things to take note here. The first thing is that here we have one mole of butane that is undergoing combustion and the standard enthalpy of combustion that is given here is given for this one mole. So let's say if we multiply the entire reaction by 2 we get 2 moles of butane, 13 moles of oxygen, 8 moles of CO2 and 10 moles of water and the enthalpy also is multiplied by 2 and it becomes negative 5316 kilojoules per mole. But this enthalpy is not equal to the standard enthalpy of combustion because the standard enthalpy of combustion is defined for one mole. So that's one thing. Now the other thing to note is that this value is negative. Now when a substance undergoes a combustion reaction we know that energy is released like when we are burning something and because in combustion energy is released it is an exothermic reaction and so by our sign convention we have the enthalpy to be negative and usually this value is negative but sometimes like in the case of nitrogen it can be positive because nitrogen does not easily undergo combustion and it only does so at very high temperatures. If we have the data for standard enthalpies of combustion of different substances one very useful thing we can find out from this data is we can compare different fuels and find out which is better. So let's say I look up the values for methane and hydrogen. Now I want to know which of these is a better fuel or in other words what I want to find out is for the same amount let's say one gram of methane and one gram of hydrogen which of these will release higher energy on combustion. So for the same amount of fuel the one that releases more energy is a better fuel and from these values we know that the energy released in combustion is expressed as kilo joule per mole. So if I divide this whole thing by molar mass I am going to write it as let's say m is the molar mass. This m has units of gram per mole and so I can get rid of the mole and on dividing by molar mass I am turning these units of kilo joule per mole into kilo joule per gram and by using this number after division by molar mass I can compare the two fuels. So let's do that here and see what happens. So the molar mass of methane is 16 so I am going to divide this by 16 and for hydrogen the molar mass is 2 so I am going to divide it by 2. So if I calculate this value I get minus 55.6 kilo joule per gram and for hydrogen it is minus 142.9 kilo joule per gram and this value that we get by dividing the standard enthalpy of combustion by the molar mass is called calorific value and so by comparing the calorific value we can say that hydrogen gives more energy per gram than methane and so it is a better fuel. Apart from the calculation of calorific value there is one more way in which these standard enthalpy of combustion values are useful. Let's see what that is. Let's say we want to calculate the value of standard enthalpy of formation of methane and we know that methane is formed by this reaction. So to calculate this instead of actually calculating this value we can use our standard enthalpies of combustion and calculate the enthalpy of formation. So for each of these that is graphite, hydrogen and methane we know what will be the values of standard enthalpies of combustion so if we look at the combustion reactions of these three we can see that here carbon, hydrogen and methane undergo combustion to give products which are carbon dioxide and water. So we know that these are the combustion reactions and these are their corresponding enthalpies. So what we want to know is can we rearrange these reactions to get this reaction here? You can pause the video here and give us a try and we will continue in a bit. One way to think about how to rearrange these reactions to get this reaction is to look at the left hand side and the right hand side individually. So let's take these two reactions first. Now what if I multiply the second reaction by 2 and add it to the first one? So I will get something like C plus 2H2 here which is the left hand side of this reaction. And here on the right hand side we have this CH4. So if we reverse this reaction and add it to these two above we can get this CH4 on the right hand side. So let's try this out. Let's first multiply this reaction throughout by 2 so this will become 2H2 this half multiplied by 2 will become 1 here and there will be a 2H2O here. And now let's just add this reaction and this reaction and now that we can see that the products are the same in both of these reactions. Let's reverse this combustion reaction for methane and write it down here so we have and now if we were to add both of these reactions so we can strike off whatever occurs on both sides and what we get by addition is the same reaction for which we wanted to find the enthalpy of formation. So now because of Hess's law we know that the enthalpy change for this reaction will be equal to the summation of enthalpies of the different steps that we took. So we can write this as enthalpy of formation is equal to 2 times the standard enthalpy of combustion for hydrogen because we first multiplied by 2 here then we added it with this reaction. So we are going to add the standard enthalpy of combustion of graphite and then because we reverse this reaction and then added it to this sum we are going to subtract the standard enthalpy of combustion of methane from this. So because these three values are given to us to find the enthalpy of formation of methane all we have to do is plug these in so we have 2 times this value plus this value minus the enthalpy of combustion for methane which is this value. So if we calculate this keeping in mind the negative sign here we get the enthalpy of formation of methane to be minus 74.8 kilojoule per mole.


So, if you just tuned the VDA channel, let me give you a review of the previous chapters. So, what's VDA? VDA is virtual distributed internet and it's a networking standard since 2004. So, we are quite grown up. We support virtual distributed internet is supported by KAMU, KVM, virtual box users and many others. And in the last two years, three years, we have completely rebuilt all the code. We have already coded everything from scratch. And VDA4, the new version, has many interesting features. But instead of listing the features from the slide, I think that we can see it in a more pictorial way. I've depicted all the items, all the entities as jigsaw puzzle ties. And each kind of boundary is a different API or protocol. So, the new idea is this, libvdplug4, which is a library that is compatible, backwards compatible with all the tools, virtual machines that are already supporting VDA. But on the same time, this libvdplug4 has plugins. And so you can use VDA to create virtual networks using very different technologies. I, in my mind, this kind of libvdplug4 library is something like SFP, the small form factor plug-in socket. I mean, in your physical switches, you have this kind of ports that can have some kind of transceivers that adapt this general-purpose port to your needs, either copper or fibers or whatsoever. Translate this concept from the physical world to the logical world to the virtual world. And what you get? So, you get this, and then we return back. You can have a plug-in that can be connected to a virtual switch. You can have a plug-in which is a hub. You can have another plug-in which is a switch. Or you can have a plug-in which is null, like DevNull. It's very useful, like DevNull, because you can use this plug-in to change the behavior of different tools. And you can have a plug-in which is a top plug-in, so you can connect the virtual world to the real world, to the running operating systems by creating a virtual top interface. And as you can see from this slide, let's have a look back. The new idea is that you can load and use each kind of plug-in using a kind of descriptor, locator, which is similar to the URL for the web. You have the technology and a description, parameters, which are specific for each technology. For example, this is the null, the VD2 compatible form, the top, the new VXVD, Slip, and so on. One interesting tool is the tool named the VD plug, which actually is just a tool which can have a VD4 client and a plug-in and converts all the communication to a standard stream, and an input standard output, so you can use a tool like the double pipe to connect to plugs. Or you can have the plug having two slots for the VD plug entities to create a cable. But I think it's nicer to see examples because an example works much more than a thousand words. Okay, look at this architecture. I wanted to connect two KBM machines to a single topper. So I create a topper and I assign this top to a user so that this will be the single, the only command needing root access. Then I connect the topper to a plug-in, which is a switch. And now I have a switch which is represented by this path. And then I can run as many KBM KMO virtual box machines connected to that switch just by naming the name of the switch, the part of the switch. And this is the architecture you gain. Okay, so let's go on. We have much more plug-ins. You have a plug-in that gives the input to a program. We have a plug-in that runs Slip. Slip is an emulator of network. So using Slip, you are connected to the internet, but all the connection, all the network traffic coming from the virtual world appears to the world as it were generated by this process. So you can connect your network to the internet as clients without any root access. We have UDP, we have Unisockets. For example, I want to run a KBM machine on my Linux box here. And I want this KBM machine to be connected to the world using the networking of a remote machine where I have just user access. I have no root access. And I can do this just by a single command because KBM and the UVDL, sorry, the locator is command This type of command, I have the command to run the remote VD plug. VXVD is a very nice tool because it creates a lot of clouds. I think that the best way to introduce you to the concept of VXVD is this sentence. Apps of the same multicast address flock together.  And as soon as you give the same multicast address to some of them, they are connected to a virtual local network. No infrastructure to their configuration. And as I showed last year, it is as performant as VXLan. So it's fast. But let us move on. This is history and I have spent 10 minutes of my 30 minutes for history. Move on. We have VXVD, which is VXVD. But with access control, we have designed nested with the plug. So plug where you can insert further plug, for example, encryption plugs. So you can use the whole infrastructure. You can decide whether to use VXVD or Slip or whatsoever. You can add cryptography, you can play with the virtual lines. But, okay. I think that the most important new feature it is. VDNS and a twerking space. So instead of running an entire virtual machine or inside of filing against a lot of comments, there is a very simple way to create a networking space where you can run your application. So you have the same kind of insulation from the networking point of view you had with a virtual machine. But it's, I mean just networking insulation. But it's very light. You can have this VDNS with just one socket. And for example, you can create a networking environment to connect your application to a virtual machine of yours using a point-to-point communication. But what is more interesting is, okay, that you can use, merging the two concepts, the local outer cloud and the VDNS space. You can create a network of night spaces. So you can have processes into networking night spaces running all over your local outer network and they are connected together on virtual networks. You can have, if you are a data center, you can have processes of your customers running where you need them, where your load balancing policy says that it's better to run them and they are connected together in a transparent way. But as you can have VDNS with one socket, you can have VDNS with multiple sockets. So this can be used to connect virtual networks together. So using standard tools, IP tables to masquerade the IP address or to packet filtering, shaping DSP servers. And you can run all this stuff into a night space and in that way this kind of night space became a virtual network function. So you connect virtual networks, you add networking management, networking processing functions, but the new point is that you don't need the new tools, you don't need new software because you are using all the stuff which is already in your Linux kernel. This is a kind of an example. If you want to connect your stuff to the internet, on one point you have to connect your entire infrastructure to the internet. One point, maybe more points. In this example, I've connected this box using Tupper to a VDNS multi which has two different sockets. One is connected to the Tupper, the other is connected to a local earth cloud and this is connected to another router, shape, whatsoever, and this creates a second layer of networking to virtual machines other namespaces or what you want. So I've told this, you can run all this stuff. Let me start from this, then we'll turn back to the final demo. There is a physical network. You have firewall, routers, switches, and machines. So there is a big deal of network virtualization. Then the second step was to have virtual machines. Virtual machines, you need to connect virtual machines, what is in yellow is virtual. So yeah, virtual machines are connected to virtual switches. Our old VDNS switch or VStack or whatsoever are virtual switches. Into a Linux box I put Tux to remember the operating system. And the remaining part of the architecture was physical. So network function virtualization, you have a function like routers, that can be processed instead of physical devices. And this is a step towards generalization, maintainability, because you can update the firmware, like I say in this way, of this router in a very simple way. It's a general solution. But the step, the final step is to completely decouple the physical structure from the logical structure. All the switches, there are switches in the physical world, which are completely related to the switches in the virtual world. You don't have any virtual switch at all, because they are distributed. VXVD creates the local outer cloud. And so you have the firewall, which is a virtual network function, running on an A space, a router, which is another virtual network function in another namespace. And you have namespaces and virtual machines spread, scattered all over your data center. This is the point. So I have the slides. I could try to do it live. So I get the risk of demos. So, okay, oops. Okay, first of all, I have... How many? 10. Okay, problems. Okay, first of all, I have to set up the gateway to the internet. To set up the gateway to the internet, I have to create the tent up first time it needs. So I create the tent up now alongside all my real interfaces on my laptop, there is TAPR. So I did a standard way to create a networking infrastructure in an old way. So I create a bridge. So I bring all the interfaces of the bridges up. I create a... I give an IP address to my bridge. And this is the old way to create the connection to the internet. From now on, I'm working on only using BLE. And you can notice that because I will not use SUDO anymore. The first thing I do, I start this with DNS. Okay. Let me show you the command before typing return. Okay. Okay, this is the command. Within a space, multiple interfaces. So please look at simple as the syntax. First, UVDL, TAPTOPER. So the first interface is connected here. The second is connected to this VXVD network. VXVD with connected. I could add other locators. And I'll just have a namespace with having as many interfaces as you need. Okay. There is a quite long prompt because I'm a user. But I've entered a username space. And inside the username space as a user, I have ambient capabilities. So now I have the ambient capability to bind service so to create sockets and imports less than 123. 1,023, sorry. I can create broadcasts. But the more important, I am net admin. So as a user, I can create, I can give IP addresses, define routing and so on. And it is what I'm going to do just now. So as you can see inside, inside and in space, I have just three VDE networks. They are connected to TAPTOP. They are connected to this network. They are connected to that network. Now using standard configuration comments as a user, I can bring up the networking. And okay. It is working. I can ping my bridge. But now I can use the standard stuff. And this is the simplest example I have in mind. So create nothing. Creating a masquerading router. Using the standard command I used on a real Linux box. So I have enabled IP forward. And I have, using IP tables, I have added a rule saying that all the traffic routed towards BD0 must be nut, masqueraded. Okay. So I can create another. This window actually is on my laptop. But this window could be the window running, a shell running on any other host of the local network. Okay. I forgot I have to give an address Let us say that the network 10, 10, 10 we are in Brussels, so 10, 10 is quite famous. So 10, 10, 10 is the network connected to the virtual network, the VXVD network. So we can create a BD in space on the other machine. Connected to that virtual network, bring up the interface, give another address but now I use a static address but if you have a DSP server on the virtual router you can have your address provided in a dynamic way. Create this. And now I am connected up here. Okay. If I have just one more minute let me see if... Actually, maybe I am not connected with the entire machine to the internet. So if down there... I am connecting my laptop to the internet. I hope there is Wi-Fi available here. Okay. Let me see. It is not working but there is some trouble somewhere. Anyway, it should work because I wanted to... There is something... For sure I forgot something but the idea... I was on an internal machine. I had to cross this and enter the network. I can assure you it works and in this way you can have the gateway to the internet. This is the only part in which you need to be root. All the other stuff can be implemented, can be logically implemented as a user. Okay. That's all. Are there questions? So are there any questions? Please. Okay. So I am going to repeat the question because I am hardly... So the gentleman was asking if the slides will be available afterwards and if there will be a way to repeat the demo afterwards I am interested. Two times yes, the slides are already available on the page of this presentation at Forstum. So you can have the slides quite quickly now. If you have time, we can have a demo side-by-side using a table around Forstum. Okay. Super. Everybody is interested. I can give a public demo. So it's time for one more question. Why 10? So... 10 is a famous character of cartoon which is famous.  Okay. Okay. Thank you very much.


आप देख रहे हैं, हमारा बहुत ही खास कारेक्रम, इतिहास के पनने मेरी नजर से। मैं अकसर कहा करता हूँ, क्रिकित हमारे देश का एक आँसा धर्म या रिलिजन है, जो अंदिया को उनाइत करता है। इस में कोई शक नहीं, क्रिकित is the most popular sport in India, जिस तरे से राजनिती पर, हर नाग्रिक को एसा लकता है, कि उनको विप्पनि करने का जनम सित अदिकार है, कि उसी तरे से क्रिकित पर, हिन्दूस्तान में, हिलाडी कम और कोमेंटेटर्स या फिर एकसपर्स जादा है, अब इंगलन्द के विरुद, जो आने या एवाली शिंखला है, उसी के बारे में देखी, तीम एनाूंस होने के पहल है, लोगोने ताए कर लिया, तेस तीम का कैक्टें के तोर पर आजिंके रहाने को बने रहना चाहिये, या फिर विराथ खोली की वापसी होनी चाहिये, आप कभी शाएद इस पे एक अपीनियन है, और अभी आप जातो सिलेक्तर से खूश होंगे, या भेहद नारास. मेरे को हिंदूस्तान के क्रिकेटिं तीमस, और उंके फोट्छीूंस को ट्रक कते हुए, लग्बभक पचाज साल होगे है. उन दिनों, शुर्वाद के दिनों में, अभी विविविवार्ती से लेके, क्रिकेट कोमेंट्री, और समाचार, सब कुछ एक बड़े से रेडियो सेट में ही, सुना जा सकता था. इन्द्या की ब्रिस्बेन, टेस, जीत, जिसके चलते हुन लोग, सीरीज भी जीत गय, इस सरटन्ट्री वोज्ट लोग, टेस, जीत, जिसके चलते हुन लोग, सीरीज भी जीत गय, is certainly one of the most glorious moments of इन्द्दिन ख्रिकेटीं history.  तल कटा के एक जमिन्दार ने उस वकत के गवन्डण जन्रल को ये लैंद गिफ्ट की। विगित ख़िए बरहेंगे तो दोमें अगध ब्रिटिश आप नहीं अंतिन 1848 तो पार्सी कमनिती ये भोमभेग फोंड the oriental cricket club, it was the first cricket club established by Indians. In 1877, the British invited the Parsis to play a match for the first time. From 1912, Parsis 6 Hindus and even Muslims of Bombay began being invited to play a quadrangular tournament with the British. Earlier in 1888, Ranjit Singh Ji, Vibhaji Jadeja, popularly known as Ranjit and the Prince of Navanagar began playing cricket after taking admission in England to study law. He excelled, he did well and was eventually picked up by the British to play for the English test team. He made his debut against Australia at Manchester in July 1896 and played a total of 15 tests till 1902 when he stopped playing test cricket. On his test debut, he scored 62 and 154 not out.  K.S. Dhulip Singh, Ranjit Singh's nephew and he also played cricket for England. On his test debut, in June 1929, he retired from South Africa. He played his last test match in 1931 in New Zealand. He played 12 test matches and scored 995 runs with a total of 995 runs with a total of 5 runs. His average was also exceedingly good at 58.52. In 1926, the Imperial Cricket Council invited India to play a test match against England. In June 1932, he played this match on the London Lords' field. He lost the match in 158 runs in India. But, as a face bowling star, he scored 5 wickets in England's first innings. Out of which, he scored 3 and 2. In Poshiarpur, Mr. Nisar played 6 tests in his career and scored 25 wickets. He scored more than 5 wickets in one innings. Mr. Nisar was the first star of the Indian Cricket. He was known for sheer pace and could swing and cut the ball with extreme vigor. His partnership with fellow pacer, Amar Singh, was as legendary as it was successful. Mohammad Nisar's birth was in India. His death was in Lahore, Pakistan.  In 1932, Hindustan played its first test match. But it was not until 1952 that we won our first ever test match. This was against England at Madras in February 1952. In this match, England took guard and was shot out for 266 runs. Spinn K. Prakhyaad Gholad, Vinu Mankad, scored 55 runs and scored 8 wickets. In response, India declared 457 for 9. Pankaj Roy and Pauli Umerighar scored centuries. In his second innings, England lost only 183 runs. That was how India won its first ever test match. In England's second innings, Vinu Mankad scored 4 wickets. In his match figures, 69.4 overs, 24 made 108 runs and 12 wickets. In 1952, India won Pakistan's first ever test series 2-1. India's next golden moment was in 1968. In New Zealand, the team won their first series on foreign soil. The scoreline of this three test series was 2-1 in India's favour. And the captain of the team was Mansur Ali Khan Bhattwadi. This victory was largely responsible due to the web of spin woven by the Indian spinners Bapu Nadkarni, Erapalli Prasanna and Bishan Singh Bedi. The next golden moments were the back-to-back series wins in 1971 with Ajith Vadikar as captain in the West Indies and in England. India and West Indies played five test matches out of which four were drawn while India won to clinch the series 1-0. The test which India won with 7 wickets in the court of Spain was by Dilip Sardesai. He put a great century in the first century. Sunil Gawaskar was a debut match and he scored 65-67. The series against England was a three test match affair and two of them were drawn. India's first victory in England came after almost 40 years of playing the first test match on British soil. India conceded a first-eaning lead of 71 runs in the final test but shot out the opponents for 101 runs in the second evenings. The chief-damager was India's weak-spinner Bhagwat Chandrasekhar. 7th of April 1976 May, the test match began again in the court of Spain and when the Indian team landed in the field the deficit of 0-1 was still in place. The West Indies scored 359 runs and the Vivian Richards scored 177 runs. In response, India's team scored 228 runs. To this deficit of 131 runs the West Indian team added another 271 for the loss of 6 before declaring their innings. In front of India, there was a very special target of 405 runs but India shocked the whole world and achieved that. India established the highest ever 4th innings chase record. This record lasted for 27 years till 2003 when the West Indies ultimately defeated Australia and won the test match with 418-7. Sunil Gawaskar scored 102 runs and won the game. Gundappa Vishwanath scored 112 runs and Mohindar Amanath scored 85-85 runs.  In England, the World Cup of 1983 was a major landmark of Indian cricket history. In 1983, the World Cup victory is still written a lot. 17-5 against Zimbabwe and then went to the finals and defeated the West Indies. By then, colour television was introduced in India. The network was also spread a lot. It was also present in many places. The delirious celebrations continued well past midnight that night. The World Cup victory was the first time that the Indian medium passers swung the game in India's way. Its low scoring final match point. India won 83 runs for a short out win. Now, only formalities are left. The West Indies will walk over. But this is not the case. Mohindar Amanath and Madanlal took 3-3 wickets. Balwinder Singh Sandhu scored 2 runs and Captain Kapil Dev and Raja Vinay scored 1-1 wickets The whole West Indies team all out on 1-40 and India won the match and won the World Cup in India. There was only one spinner in this team, Kirti Azad. And he scored only 3 runs. It is believed that after the win of the 1983 World Cup Kirti Azad has never looked back. Sponsors have come. New players have started to improve. Kirti Azad became role models and the game slowly was on way to becoming a fairly profitable profession. In the last 20 years, India's team has knocked down new watershed moments. In Kolkata, in 2001, VVS Lakshman and Rahul Dravidt were among the participants. Along with that, Harbhajan Singh's Patrick was in the same test match against Australia. This is also always remembered in the Cricketing International Annals. In 2002, after winning the Nat West Trophy against England, Saurav Ganguly's Lord's balcony and his shirt were taken as a flag. No one remembers this image. This victory was also the result of another rear guard action led by Mohammed Kaif, Yuvraj Singh and Harbhajan Singh. After that, in 2004, in Pakistan, the first series of victory was the 2020 World Cup. In a way, these victories and several others were in preparation for the 2011 World Cup and the 2013 Champions Trophy victories. After defeating Australia at this time of victory, we should also remember Indian Cricketers whom we believe are heartbreak moments of Indian Cricket. I will not talk about them yet. But always remember, no one should be afraid. In a few days, Indian Cricketers will face the new challenge as a team in England. We hope that this religion will continue uniting Indians irrespective of their personal identities. Namaskar.