Another game for the Wanderers, another goal, how is it feeling? It's a good experience, especially here at Shinji's former team. It's a big crowd that showed up today to watch him and especially after Wanderers. It got off to a good start for what he brought today. Disappointed that we didn't get a draw, but that's how football goes. And how has the tour been treating you so far? It's really taken a toll on us boys. It's really tiring and a lot of travelling. Not much time to rest up. We're making the most of it and giving it our all. So boys are improving a lot day in and day out. This is also preparation for us when the ACL comes just to get used to travelling a lot. And you're enjoying the countryside of Japan and off to China next? Yeah, we haven't seen much though. But along the way here to the stadium we have seen a little bit of the countryside. It's looking really nice and looking forward to trip to China.


Another major medical center has keaked against a bill to establish a federal nephrology and kidney research and treatment center, Hadejia. President of the NMC, Dr. Oji Ojima, suggested that preventive research should be the core focus instead of treatment, which is usually at a near-fatal stage. He said that these alongside are the medical professionals during a one-day public hearing at a Senate wing of the National Assembly. On the bill to provide the National Nephrology Center, we regrettably reject this bill in its entire take. As experts, we should not wait for intense headache to get to where it is decapitation that will be the solution. If there is preponderance of kidney disease in Hadejia, the solution is primary health care. Find out what is causing the problem and nip it in the board. You do not sit by and wait for people to get to terminal conditions before you offer them treatment. We have HIV-associated nephropathy, we have lupus nephritis, we have chronic glomerular nephritis. So our own opinion as an association is that we deserve to have a kidney institute. However, it should be in a center that is readily accessible to all of the Federation. We looked at Hadejia, yes, they have this condition, but HIV-associated nephropathy, should we site one in Benwe? If we are talking about lupus nephritis, should there be one only in Lagos? Let us have a central place that all of us can come to, accessible by air, accessible by road readily. We have the manpower, but Hadejia actually deserves attention. No, the medical, what do you call it, lab scientists also said. So we will really look at it. I think that there will be a debate. As I said, we do not have opinion. That is why I said the ministry would look when there is a disease. Sometimes, Mr. President, without making any comment, maybe there are some genetic challenges and so on which are not public. That is what they are trying to do. Of course, as the front says, we do not wait until CKD to need it, but so the ministry of health needs to be aware of this. Meanwhile, a bill to establish Federal Medical Center or Kiwi, which is past second reading, and the floor of the Senate has been amended to the establishment of Federal Autopedic Hospital, Ihite, Ihetoboma, local government area of Emo State. Senator Frank Ibezim, who sponsored the bill, disclosed that the amendment was necessitated because a similar one has been passed by the House of Representatives sponsored by Princess Miriam Onoha and has advanced to the concurrent stage at the Senate. We are meant to understand that my sister representing a Kiwi, not federal constituency. Honorable Princess Miriam Onoha has a similar bill that has been passed and that is coming for concurrence in the Senate. So that has blood in my heart because we are all from the same senatorial, same zone, and there's need for us to partner and make sure that this bill gets the required assent as soon as possible. And I've spoken with the leadership and I've agreed that in the next business day, that bill will come for concurrence. I've also written and asked that with past, first, and second reading and were in public hearing that my bill should be amended to become an autopedic hospital. The truth of the matter is we can hear about her dreams, but we can't have emphasized the importance of health and medical facilities. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


My name is Anisha Singh and I run a venture firm called Shee Capital. I'm the founder of Shee Capital. So, before Shee Capital, I used to run my Dala. So, I've been a founder. Founders are eternally optimistic. So, I'll tell you this. There is no better time. I started in 2009. This is a cycle we're seeing. I think it's a brilliant time to be raising. One thing is, if you can drive in India, you can drive anywhere in the world. I think if you start up in India, you can start up anywhere in the world. So, my advice is do it. Don't wait. It's a great reality check to have and hygiene check to have in general. You don't want anybody dipping into your bank account every now and then and just playing crazy and everything. So, I think it's good that we've got into correctional mode in general across and it shows a maturing of the market. We need to do this as a country because we have such a robust ecosystem. And to show other international investors that we are truly world class, we need to be doing it. I just think it's a nice correction that needed to happen and it's happening. With any maturing of any system, we would have seen this happen.


A month has passed since Bitcoin Miami 2021, which was called the largest crypto event in history. Over 12,000 attendees, 100 expert speakers, and some of the most well-known faces in crypto brought Bitcoin to the forefront of the world's attention. The sheer size of Bitcoin 2021 showed mainstream adoption is well underway and demonstrated the progress that Bitcoin has made since its inception. Bitcoin 2021 was held at Winwood Miami, a unique neighborhood known for its colorful murals, eclectic bars, trendy boutiques, and hipster coffee shops. Welcome to the Miami Winwood Art District. This area was founded in 2003 by a group of art dealers and has been absolutely revitalized in the years recent. Anything from graffiti on the side of buildings to the sidewalks to the streets, this area has come to life in its home to the 2021 Miami Bitcoin Conference. Unsurprisingly, a major theme at Bitcoin 2021 was non-fungible tokens, or NFTs. There was an entire NFT art gallery at the event which displayed Bitcoin-themed artwork from 35 different creators. Crypto artists like Taylor Good, also known as WarHotel, and Sergei Gordinko, the do-it-you-love artist, enlarged displays of their artwork featured within the gallery. We were informed by the organizers that almost 100 works of art were sold during the event. Art wasn't the only thing you could buy with Crypto at Bitcoin 2021. You can even buy your drinks here at the conference with Bitcoin. The bar also accepted Bitcoin as payment for drinks. While a fun concept, this feature was completely impractical because the payment transaction fees were higher than the prices of the drinks. All right, so we tried to pay with our Bitcoin, but the transaction fee cost too much. So unfortunately, we're using our card because convenience is king. In addition to Bitcoin 2021, the Shitcoin 2021 conference took place on June 3rd through the 4th down the street at the Man of Windwood. According to the Shitcoin 2021 website, the event was dedicated to all the companies that got turned away from participating at Bitcoin 2021 for not being BTC focused. Bitcoin 2021 was also the first Bitcoin event to take place in person since the start of the COVID-19 pandemic. But while the turnout was impressive, many participants ended up catching the virus as a result of the conference's loose health safety measures. COVID aside, Miami seemed to be the perfect place to host Bitcoin 2021. Miami Mayor Frances Suarez is taking a number of initiatives to ensure Miami becomes the world's next crypto hub, such as proposing legislation that would allow Miami's citizens to pay taxes with crypto. The event wrapped up with a special pre-recorded announcement from El Salvador president Naib Hukili. Bitcoin was adopted as legal tender in El Salvador. With this capstone, Bitcoin 2021 cemented itself as a milestone event in crypto history. In El Salvador, we are trying to rescue this idea and start the design of a country for the future, using the best ingredients that makes us who we are, while using sensibility to find the best examples of ideas from history and around the world. I believe Bitcoin could be one of these ideas. That is why next week, I will send to Congress a bill that will make Bitcoin a legal tender in El Salvador. Hit the subscribe button to get notified for our upcoming in-depth video analysis of El Salvador's Bitcoin law later this month. Hey, it's Rachel here at Bitcoin 2021 and we are wrapping up. And this has been a crazy, amazing event. Let me just tell you guys, after all these years of going to Bitcoin conferences, this has been one of the very best that I have ever attended. The speakers at this conference have been incredible. Very big names, very informative panels. We've got so many great interviews with speakers like Alex Maschinski, Yoni from Etoro, so it's just been so much fun and I can't wait to watch all the panels all over again. And yeah, thank you for joining us. Coin Telegraph editor at large John Rice also shared his thoughts after the event. So Bitcoin 2021 has been an interesting conference because I think people forget to a large extent that we're a small community in the global scheme of things and we sometimes feel kind of alone. We don't always know that there's a Bitcoiner next door or 10 Bitcoiners on our block. So I think when you get to an event like this and you see thousands of other people who are trying to do the same thing as we are, which is basically change a big thing with a small number of people, it makes you feel less alone. It makes you feel more like you're part of a community. So that extent, I think Bitcoin 21 has proved that we are here as a part of a celebration. It's not just a conference. It's a celebration of something we believe in, something we feel is important as a community and the kind of perennial struggle of a small group of people against an establishment. So when you look around here and you see hundreds of people, if you don't know them or if you aren't even learning from them because you already know a whole ton about Bitcoin, it doesn't really matter. Because what it's reinforcing to you is you are not alone. There are other people who believe what you do and there are other people who feel it's important.


All right. I think we can get started. So hi, everyone. Welcome to this talk. It's titled Fantastic Ordinals and How to Avoid Them, Autoscaling Challenges in a Cloud Database. My name is Manish Gill, and I work for a company called Clickhouse. So I'm an engineering manager at Clickhouse. I work in a team, which is the scale team. And the primary thing we do at Clickhouse is think about vertical and horizontal auto-scaling, idling, scaling to zero. And we also heavily contribute to the Clickhouse operator in order to facilitate the scaling mechanics. So this is the agenda for today. We're going to look at Clickhouse, the technology. We're going to talk about Clickhouse Cloud. We're going to talk about the auto-scaling architecture of Clickhouse. We're going to look at vertical scaling specifically as a problem that we faced in Clickhouse and then what the models of vertical scaling are, specifically what we call the break-first model of scaling. Then we're going to go into something we like to call make-before-break as an alternative approach. We're going to look at some nuances of how stateful sets work and the limitations of stateful set and how we dealt with the problems. OK, so Clickhouse. Clickhouse itself, probably some people are familiar with it. It's an open-source, column-oriented database. It's a distributed OLAP database specifically. So the workloads that Clickhouse typically runs are analytical in nature. It's got a reputation as being a really, really fast database. So most of the times you will see people running Clickhouse clusters with terabytes to petabytes of magnitude. And yeah, feel free to Google it. It's a pretty powerful tool. For the purpose of our talk, what's important is it's a distributed multi-master database and it's eventually consistent. And so whenever I'm talking about pods or replicas in Clickhouse, there is no such concept of a primary with a replica in turn. So all replicas can ingest traffic and all of the replicas can serve queries. So no one replica is special here. So Clickhouse Cloud is our serverless offering of Clickhouse. We try to give you a fully managed experience. You don't have to deploy your own shared nothing architectures. It's serverless. It's got idling and autoscaling. And that's what my team is responsible for. It's also got separation of compute and storage. So the storage layer in Clickhouse turns out to be like as is the trend these days, S3. So the actual data gets stored inside object storage. We are available in two major cloud providers right now, AWS and GCP. And we have an Azure offering coming soon as well. As you can see here, so we have these server pods running inside a Clickhouse instance. And these are the compute nodes. We also have PVCs attached to these nodes right now because we still have some metadata pointing to that S3 data on these PVCs. But we're slowly trying to get rid of these PVCs and eventually move to a fully stateless model. That'll probably happen sometime soon. Apart from Clickhouse Cloud, there's also a bring your own key kind of a model where for enterprise customers who probably don't want data to ever leave their VPN, they can just provision this entire data plane inside their own account and then get the benefit of all these serverless offering inside. So when you're managing a database on Kubernetes, the first thing you start out with is an operator. And that's kind of what we did as well. So here we can see a very simple specification of how an operator looks like. We have a custom resource inside our Kubernetes clusters. This is the Clickhouse Cloud cluster resource. That's being managed by the operator. This operator, in turn, is creating and managing a stateful set. And this stateful set has server pods 0, 1, and 2 with PVC attached to them. So this looks pretty familiar to most people, I hope. OK, so now we understand the backdrop of there's an operator, there are server pods. So let's talk a little bit about how autoscaling works. So there are specifically two components in any autoscaling infrastructure. And that's what we built ourselves as well. The first bit is the recommender. So autoscaling starts with the data. It starts with metrics. So in this case, we can see from the left, we have like Clickhouse itself exposes a lot of metrics that we collect. We also have Querylog itself, which is actually accounting for the memory that a particular query is going to take. And we are able to do this because, actually, Clickhouse keeps track of its own memory internally. We also collect Kublet metrics because that's important. That also gives us another view about CPU utilization, et cetera. And all of this data gets collected inside a metrics database, which is coincidentally also a Clickhouse cluster. And the second half of this equation is the recommender. So this is a controller, which is periodically running, looking at this data, and then making decisions about what should be the right size of any particular cluster. So this is actually the component generating recommendations. And this is where the algorithm lives. So you can start with heuristics, and you can make it as complicated and as fancy as you want it to be. So any smart and sophisticated autoscaling strategies that you want to do it, you probably want to do it in the recommender. So that's what we use as well. And so once you have this pipeline flowing, you start from the metrics, and then you go at the end. You have these recommendations being generated. How do you actually make use of them? How do you actually do autoscaling based on these recommendations? So this is where the actual autoscaling component comes in. So as you can see from the left, we have two inputs. One is the recommendations that get generated. The other is the user defined limits. So users probably want to have minimum and maximum limits to control their cost. And so those get accounted for inside these recommendations. So the recommender could probably say, hey, I would like this cluster. This cluster is just so memory intensive. I want this to be at 100 gigabytes or something. But the user has a max limit set. So we actually account for that as well. And this is where things get a little interesting. So once the autoscaler detects that, hey, I actually have a cluster which is not rightly sized, and I want to kind of scale it up or scale it down, it's going to start doing pod evictions. And this is how it works inside the Kubernetes VPA as well, which is the moment we detect that we need to do some right sizing. We evict a pod, and then we resize it. And this is how it looks like. So the autoscaler is going to trigger this pod eviction. You can see the pod on the right here. And what's going to happen is a controller is going to come in. And in this case, it's just a stateful set controller. And this controller is going to resubmit that pod, which is very natural. We know that. And this is where the magic happens. So as soon as the pod gets resubmitted, we have a mutating webhook. And this webhook is going to intercept that pod submission, and then it's going to mutate the resource request. So what happens here is that the controller will probably have its own idea of what the size should be. But the webhook, which is aware of the recommendation as well as the limits, is going to say, no, no, no. You actually need to size this pod according to this recommendation that I have. And so this is how you can actually size this pod. And we can see that in this case, the pod got sized to a bigger size. So we understand how the autoscaling infrastructure works in general now. And this is anyone who knows VPA kind of knows this. We just happen to be building these components ourselves in order to get some more fine-grained control over how to do evictions, et cetera. So we don't use those out-of-the-box components. Specifically, in vertical scaling, this is kind of what we want. So this is an example of you probably have three server pods running with 16 gigs each. And you want to scale them up to go to 32 gigs. And this is a fairly normal requirement for vertical scaling. And this is what we mean when we say vertical scaling is actually break first, right? So it's break first because you actually, like we just discussed, you have to evict a pod. And that eviction is like a restart. And that's causing a disruption to the customer's workload. So while we are actually trying to resize this customer, this pod is actually in the process of getting restarted. We have a PDB, which is like maximum unavailable. It is always going to be one. We don't do zero because that's kind of unreasonable. So maximum unavailable pods is just one. And so what happens there is that you resize these replicas one at a time, right? So you start from the last replica, you resize it. It takes its sweet time trying to come up. And then you do the same thing with the second one and then do the same thing again. And at the end, you have like a scale up operation that's complete. So this is pretty slow, right? So what happens is customers who want to kind of get the benefits of scaling, they complain that, hey, the scaling is just not reactive. And this is the reason auto scaling in Clickhouse Cloud is sometimes not that reactive. The other problem that vertical scaling can cause it, it can cause additional pressure on the remaining replicas, right? So in this case, we can see that while this third replica is getting restarted, it's actually putting more pressure on the existing replicas. So you have this kind of tension here where you actually wanted to scale up because you wanted more resources. But what you ended up doing was, in order to get those resources, you ended up taking away the resources you had for some time. And in the worst scenario, what this can lead to is these other replicas, they're just going to crumble under the load and they're just going to start getting throttled, maybe get umkilled or those kind of things. And customers start to complain about this. So vertical scaling, as it exists, is kind of a deal breaker. We want to move away from this model, right? Yeah, like I just said, it's slow, replaces one part at a time, it's disruptive, puts pressure, the remaining replicas get squeezed. And one last thing is we actually maintain some overhead to kind of combat this. So in our recommendations, we account for this fact that, hey, actually we want to resize this replica to some number, let's say 32 gigs, but we know that it's going to get squeezed, so let's kind of have some percentage as an overhead. And that also means kind of the customer is actually, the utilization was, the desired workload for the utilization was kind of lower. But just because of the nature of vertical scaling, we account for that in the overhead. That's kind of bad, right? We actually want to kind of reduce our overhead and the customer's utilization should be as close to the allocation as possible. So Kubernetes experts will know that there is this cap which allows for this interesting new feature, which is in place pod vertical scaling. This allows for resizing of pods in place. And this seems like a very convenient feature. There is actually no restart needed. And so why am I here talking about vertical scaling and this disruption when this feature already exists in Kubernetes? The last bullet point kind of gives it away, which is like, it's an alpha feature. There are some limitations to it. So we probably don't want to use it right now, but there is another important point, which is why we don't use it, and that's packing. My colleagues, Gianfee and Vinay actually did a talk yesterday about how we are using most allocated scheduling strategy in cyclic house cloud to pack the pods as efficiently as possible. But this is kind of just to give you like a glimpse of what's happening here. So in this scenario, you have like three sort of server pods running on a single node. And we can see there is some room for at least one pod to get resized in place without doing any sort of disruption, right? So let's see what happens. So yeah, there is enough room and we can just resize this replica in place. And we're happy, right? But this guy again, this picture gives away the problem here in this situation, which is that now the node is actually full. Now there is no more space left to kind of do any more further resizes or like kind of increase the memory or the CPU of this node as well. Yeah, no room left for the spot. So the idea here is that even though this API exists in Kubernetes, you can kind of take advantage of this. You can do vertical scaling in place. It's actually best effort.  It can also, you can just request to do vertical scaling in place. And this API can come in and it can just reject that request, right? And so the fundamental tension here is that packing efficiency, if you want to kind of optimize for cost and pack your pause as efficiently as possible, this is kind of like working against this idea of doing in-place resizing. So you can either have like in-place resizing with some buffer always, which means you're eating the cost or you can pack as efficiently as possible.  Additionally, we actually have a homogeneous fleet. So there are no like low priority co-tenant workloads like some sort of an API server or like different tiers of workloads running where we can kind of evict this low priority pod to make room for this other, like bigger to kind of do in-place scaling. That's also not something that we do because we always want to pack as efficiently as possible. So now we come to like this idea that in-place vertical scaling is not really working for us. Default vertical scaling is kind of disruptive. So maybe we can do something better. And this is the approach that we've been calling make before break. And the name is also like there's a hint in there like how it works. You just have to make new replicas before you break the existing replicas. So here we can see the example. We started with three different parts of 16 gigs each. You'd actually don't need to do anything. You just scale it out and kind of leverage the horizontal scaling to kind of get the additional capacity, right? So when you scale out, you get like a force replica. And because we have the mutating web book that we discussed early on, when this force replica gets submitted it's always going to come up with like a bigger size. And then you can just break any of the existing three replicas. And now you have like a single cycle where one part got resized without any kind of a disruption. You can also take this like one step further, like for maybe like some customers would like to have like instant or like almost instant scaling. So you can just immediately double the capacity, right? So you can just have, hey, I have 16 gigs replicas. I can just go from 16 to 32 gigs replicas. And then I can just slowly drain away my other older replicas. And now I just have exactly what I wanted with just three 32 gigs replicas. So what are the advantages of this approach, right? So it's fast. We kind of know that there is no more restarting that's happening like the PDB is kind of not getting in your way. It's non-disruptive. There is no more pressure on existing pods because the existing pods can still continue to serve queries as safely as possible until they're kind of ready to go away. And because we have this, we can now reduce that extra overhead that we have been recommending. So now we're gonna talk about like stateful sets and why am I talking about stateful sets specifically here? Because MBB as an approach, like it's great on paper, but it actually doesn't work with the practical realities of how stateful sets today exist in Kubernetes and I'm gonna explain how. So this is the scaling behavior of a single stateful set today in Kubernetes. It's always going to scale out from left to right. So, and this is why actually the title of this talk is ordinals because you're gonna start with ordinal zero and then you're always going to scale out like one, two, three, four, five. And as you can see, when you're doing like this three and four and five replicas, they're kind of like this make. So you can make arbitrary replicas inside the single stateful set. The problem is you cannot break the old replicas because if you follow the arrow inside the scale in, if you try to tell a stateful set, hey, I would like to have just three replicas, it's going to remove the three new ones you just created. And that's the opposite of what we want. We want to actually break replicas zero, one and two. We don't want to break three, four and five. So stateful sets are getting in our way. Like this is the default behavior. There is one slightly hacky way in which you can achieve this. And this is the feature in stateful sets called start ordinal feature. It was introduced in this cap. And so the idea here is again, simple. So you start with three replicas, you just change the replica count to four and then you get a make operation. That's fine. So you're happy with your make. And then you can just tell the stateful set, hey, I would like you to start your ordinals from position one instead of position zero. And what this does is now you can just, what it means is that part zero is kind of going to get deleted, right? So which is again, make before break. Why is this something like, you know, I said it's hacky, so why is that? It's hacky because you want the flexibility, right? You might have a backup running on replica zero and maybe replica one was the one you intended to break because hey, that one has enough, like it doesn't have enough queries running on it or maybe you can drain it safely. So you're not getting the flexibility that you need. So you're always going to be like breaking in this strict order. Think of it like a sliding window moving from left to right in which you're going to break this. So this feature kind of works, kind of gives you what you need in terms of make before break, but not really what you want to use. You want the flexibility. Yeah, so breaking arbitrary pods. So we've kind of defined the problem here, right? So vertical scaling, what are the problems there? We want to do make before break and make is fine, but we want to break these arbitrary pods. And we are not the first to face these problems.  Specifically, we looked at a few of them like advanced stateful sets by pink app and open cruise. And the key idea that all of these projects follow is this idea of delete slots. And I'm going to explain what that is. So as you can see here in advanced stateful sets, the way it works is you can just specify the replica count as well as the delete slots in an atomic operation. So when you do this, what happens is you're starting through replicas because the controller has to maintain the replica count, it will make a new pod, which is the third pod that's ordinal number three that's coming up. And because you also specify the delete slots, it's going to break the delete slot that you wanted it to do, right? So this does what you wanted to do. So what's the problem here? Like, why did we not use this particular solution to do make before break? And the answer is really like it was an experimental project. We did not want to take ownership of this code base. The default behavior is still falling back to ordinals. And my biggest gripe and my biggest complaint with this sort of solution is that I think ordinals are kind of just hard to keep track of when they have gaps in them. And I think this goes to the nature of ordinals, right? They are, or you kind of expect replicas to be in a continuous order. So when you're like an on-call engineer and you see replica 70, replica 72 and replica 73, you're going to ask the question, hey, what happened to 71? Is it in the delete slot? And you're going to have to do this mental math and look at the CR and see what's going on. And over time, especially with horizontal scaling inside the mix, this is just going to get ugly, right? So we kind of, I started to think, okay, maybe we want to move away from this idea of ordinals all together and look at a different solution. So OpenCruise, again, it's a separate project. It's got two different controllers. One is also coincidentally called advanced stateful sets. The other is called clone sets. So the OpenCruise advanced stateful set has the same downsides that I just mentioned about the pin cap controller. So we didn't really take a look at that very seriously. Clone sets were the interesting ones. They are actually supposed to be a replacement for deployments, but they support like volume claim templates. So you can do things like a stateful, you can run stateful workloads on top of them. But notice here that the pod names actually random. It's not ordered. That was kind of where I started to move towards that, hey, instead of ordering pods, maybe we just want randomness inside of our names and kind of treat everything equally. So there is no server zero is not special and there is no unintuitive behavior in terms of ordinals. This is what a clone set looks like. So you have like five pods running at the one I've highlighted in Boulder. Like you can just select and delete that. You can just add it to the spec and this will get deleted. And the key thing to remember here is that the operation of reducing the replica count as well as adding the break pod to the set, that has to be atomic because otherwise you will just get a new pod popping back up to maintain the original replica count. But yeah, so this kind of does what we want which is selective pod deletion.  The catch which clone sets was actually specifically not with their arbitrary pod deletion but with their pod update strategy. So clone set actually supports like recreate strategy as well as in place pod updates. And where we got stuck was just PVC deletion as well as PVC recreation. So when you recreate a PVC, when you update a pod, the clone set controller is always going to recreate the PVC and that's not what you want for a database workload. You actually, and so the only thing you're left to do is do in place updates. The catch there is if you do in place updates you actually can't modify your resources or even update your environments. So you get a choice. You either can resize your pod or you can retain your PVC. You can't do both. And that was kind of like a place where we got stuck and clone sets were just, we wanted to have the best of both worlds, right? So we can't have that with clone sets. Okay, so we've looked at a few solutions and we realized, okay, no popular active out of the box solution is going to work for us for various reasons that we discussed. And this is like a quick recap of our requirement. We want a stateful set or something like a stateful set. We would like to move away from ordinals. Delete slots are a good idea. So we want that. We want to retain our PVCs. We want to respect that disruption budget for things like upgrades. We still want to have that. And I'm going to talk about topology spread constraints later because they get pretty important once we go towards our solution. And so the idea was we can just use the stateful sets, just use them in a different way. So this is what we call multi STS. And it's very simple. The vanilla stateful set is pretty powerful. We can leverage it. The very interesting property that the stateful set gives us is that we can still have operator downtime. So the operator is the code that we are changing almost every day while we're developing it. And if we wanted to kind of directly manage our pods inside the operator, we would have to kind of risk the fact that we could introduce bugs inside our operator. And the cost of that will be pretty high. So imagine like you're writing code which is kind of doing something with a pod and the pod gets kind of accidentally deleted or something. That's not what you want to do. So the stateful set is a battle test controller. So we wanted to leverage that. And the other thing is it's relatively easy to go from like one thing to many things inside your code base, just in terms of refactoring, instead of introducing like a new custom resource like an advanced stateful set or a clone set and then doing a migration. Remember, we still had customers running on the single STS. So we had to migrate, which can be like a whole separate talk of its own from like single STS to this multi STS model. But yeah, so the key idea here is just use one stateful set per pod. And this is what it looks like. So this is the same image from the beginning. You have the operator creating a stateful set managing three pods, but now we can just do this. So we have like three different stateful sets. Each of them with a random suffix. So when you have these suffix and then you can kind of implement your own delete slots and then you will have the ability to kind of make new replicas as well as break any arbitrary replicas. And you can achieve make before break vertical scaling without any of the problems that we discussed before. Yeah, so this is just a quick recap of what it looks like. So you have three stateful sets managing three pods. You do a make, the fourth pod comes in. It's like a bigger one. We introduce something like a condemn step. And this is kind of an intermediate step where the pod is marked for deletion. And this is done mostly for database reasons where click house, it's not really trivial to immediately get rid of a replica and the database figures everything out automatically. You have to do some sort of synchronization operations. Not very many, but there are still a few steps you have to do. So inside the operator code path, you actually condemn the replica first, do the synchronization and then you break that replica. And so this is the way we achieve make before break. And again, all the benefits that I just described previously you can do instant scaling, no disruptions. You can do like your performance is great now. So the requirements for implementing multi STS were not very many, but still worth talking about. We needed stable identity. So this is something that the ordinals kind of give you always. So pod zero is always going to be pod zero. If you remove it, it's always going to come back with the same name. And this stable identity is important for your host name and the less PVC name, other reasons. The way we do it is we kind of start to manage the state inside the operator ourselves and kind of use the stable identity for the random suffixes that we are generating as well. You have to do horizontal scaling, which was fairly trivial to do in terms of scaling out. Scaling in is where some of the challenges came into place. I'm gonna talk about them. Rolling updates, you kind of get for free because the pod disruption budget actually works on top of label selectors. So it doesn't matter if one stateful set is managing the pod or many stateful sets. As long as you have the right labels, you're still going to get your PDB. And again, topology spread constraints. I'm gonna talk about them specifically because that was the one area where we ran into a lot of challenges. And the only new feature that you're really adding here is the delete slots. So topology spread constraints and what were the problems with it? Inside Clickhouse Cloud, just to set the context, if you have like a service with three different replicas running, all the three replicas are going to be placed in different zones. And of course, we do this for availability reasons. And what happens when you try to do like make before break with topology spread constraints. So here we have an example of we have like, initially we have two parts, A and B, and now we want to make two new parts. We can do that. And these two parts will go into zone C. So A, B and C here are availability zones. And A, B, C, C is actually just a valid topology. The problem comes now when we actually want to break those old replicas, right? So what happens when you break A and B? You actually have both of the remaining replicas inside zone C. And now your zone C becomes a single zone point of failure. And it's actually not a valid topology if you have your max Q set to one. Another problem we ran into with topology spread constraints in MBB was the fact that we actually, like I mentioned earlier, we actually have idling in cyclic house cloud as well. So when you idle a cluster, you actually have three stateful sets. The parts are not scheduled anymore, but the PVCs are still lying around. So when the stateful set controller sets the desired count to one again, these parts will wake up, they will get scheduled and then they will reattach to these PVCs, right? But you can have a situation where a customer wants to add a new replica before their cluster wakes up. So you can imagine, I have three replicas before, but before, and my cluster is idling, I can just dial like a setting in my UI and just say, please give me four replicas when you wake up. And this is where a race condition happens where the stateful set controller can just immediately schedule the new replica, but the old one can sometimes take its sweet time to come back up. And when that happens, you can have a race here because the stateful set controller only takes the scheduled parts into account when making the decision about zone placement, right? So you can see here, like you have, like the initial two parts are in zones A and B and the third, the new replica that came in, it also came in with zone A because the only consideration we had while we were making the decision about the zones was A and B. We actually never looked at this replica marked in red because we never took the PVCs into account. And that's something I think this is also true for the default stateful set. But this situation actually would not have happened with default stateful sets because it will, if you remember the arrow that we were going from right to left, if we were to kind of do like a replica count, reduce it to two, it will remove CC. So this situation is avoidable with stateful sets, not with MBB, but this one I think is still kind of something I might have to test that but I think this race condition also exists with a stateful set controller today. So there are all these problems that we ran into with topology spread constraints. And really the way to solve them was kind of do zone pinning which is kind of we started to decide which availability zone apart is going to get placed into. And this logic kind of is now living inside of our operator where the operator makes the decision about which zones to place the part into. And because like previously, like we mentioned, like it wasn't taking PVCs into account, the operator can kind of just do that. So we count the PVCs, figure out what's the zone balance and then decide which zone to place the part into. So yeah, long story short, we have like breaking arbitrary parts. It does not work naturally with topology spread constraints. So that's the only catch if you would like to implement make before break yourself. And the same reasoning also applies with horizontal scaling. So your scaling in operations have to be zone aware. So just keep that in mind. The scheduler, like I said, is not really aware of existing PVCs. It's only aware of scheduled pods. So we've covered all this. And so why am I talking about make before break multiple stateful sets? And like, you know, we ran into these problems but we are not really the first ones. Stateful services on Kubernetes is hard. I'm sure many of you here know this already. Cloud native PG, which is a very famous post-careers operator, also decided to write a custom controller to manage the pods themselves. Strimsy actually, the highlighted section here mentioned the exact reason that we have, which is, you know, the ordinals are a problem when you want to do any arbitrary pod deletion inside a stateful set. Strimsy for those of you who don't know, it's a Kafka operator for Kubernetes. So yeah, it's like not a new problem but the stateful set controller just doesn't give us the flexibility that we need. So just a recap of what we've covered so far. Vertical scaling for stateful services. It can be very slow. It can be disruptive. Database workloads are kind of important. You don't want to, you want to be very, very careful about this. In-place pod resizing is a tempting feature. Does not always work if you want to pack as efficiently as possible. The stateful sets are not really as flexible as we would like them to be. We can't do arbitrary pod deletion. And the solution to this is something like that we have been calling make before break. You can try to use it. Delete slots because making again is easy. Breaking is the hard bit here. And so we've been using delete slots to kind of facilitate breaking of arbitrary replicas. And you can use third-party controllers that we discussed such as clone sets or advanced stateful sets if you kind of don't want to go too much into writing your own code. We had that bandwidth I guess. So we were able to kind of implement this multi-STS approach. But of course you can always use the third-party controllers. Be mindful of the caveats that I discussed. And yeah, topology issues are the ones that you need to be very, very careful about when we're doing this kind of thing. That is what I have for you today. Thank you so much. We are hiring. So come talk to me if you are interested in these problems. Thank you. Yeah, so if anyone has questions. Hey Manish, very interesting story on how you went through all these different things with the scaling. One important thing, does your operator now, because we didn't see much about the limits and requests in this context, right? We are talking about just the scaling part. So in your case, for your stateful sets or for your databases, now how do you have your limits and requests? Do you have a fixed reservation? Yeah, so our limits are always equal to the request. We have like a quality of service guaranteed. Okay, so that makes your life easy but not all stateful sets would be like that. So we actually have forever click house keeper. So we didn't mention this, but click house uses its own like a zookeeper replacement called click house keeper. And inside the keeper stateful set, we actually have a recently set burstable CPU. So the limits are actually different. Can work, I think. So in that case, also now you go through all these things to schedule the next one? No, no, no, so this is, we do this only for the server pods. Okay, thanks. All right, thank you.


And have a ride between USA, 3.3 km. You're going very hard. You have to finish off Justin Gonzalez.


Hi YouTube family, welcome to today's video, which is going to be a little tour of what my makeup is right now I very recently moved across states from Washington to the state of Utah when I did that I Decided that I didn't want to pack all of my makeup and what I finally did was I got brutal You guys I got so brutal with my makeup collection and you see my vanity that is behind me this vanity I did get off of Craigslist about four years ago It is not an antique, but I love it and I decided that what I was going to do That was that everything needed to go in this vanity and if it didn't go in the vanity I do have a little bit of an overflow for skin care And then I have a separate place for my palettes But other than that all the makeup needed to go in this vanity for me to be able to move So that's what I did. I made sure that everything fit in there and I decided that if it couldn't fit It had to go. I got rid of two-thirds of my makeup collection I got rid of so much you guys, but it's good because a lot of it was stuff That isn't even on the market anymore that I was hanging on to because I did like it But I really wasn't using it because I couldn't show it too much here on camera and off camera I like to test what I'm showing you like I said I was absolutely Extreme in how I did this declutter and what I kept so not showing you the declutter But I'm going to show you what I kept and show you how I organize it a little bit into this dresser I'm just gonna point the camera towards the dresser and show you guys basically how it looks every day as I'm doing my makeup And what is in these little drawers? Let's do get into this video I hope that you guys enjoy seeing my makeup collection and seeing how I store it Okay, so what I decided to do is kind of just pan across and give you a little bit of a view This is a really hard angle, but basically that is my bottle collection up there I love purple bottles and I collect them perfume bottles whatever those are miniatures up there and then right here You're going to see that I have my brushes and my eyeliners and my mascaras all in this little turntable type thing that I have and Then I also always have my mirror I won't open that because it will blind you and then over here I have makeup that I love to keep out because it's new makeup that I'm testing This is some new stuff from the tart holiday collection and then folklore costume it folklore I hope I'm saying that right cosmetics Was kind enough to send me a couple of things I think what I'm gonna have to do is I'm gonna have to pull out each drawer and kind of show you what's in here So this is my lipstick drawer. I had hundreds of lipsticks But I decided that if they couldn't fit into this little drawer right here that there was no way I was gonna keep them so I was able to keep quite a few that I love There's a lot of Maybelline in here a lot of Mac I kept just about all of my Mac and I kept a lot of my Urban Decay Which I love their lipstick formula as well So I really tried to pair them down to just what I'm using and just what I love and then just to tell you a little bit about This turntable that I have or this round holder that I have that swivels I just decided to pair down my brushes. I had so many brushes you guys I had this thing absolutely crammed full of brushes for face and eyes And I just decided that I was gonna just pair them down to the ones I love and then I decided to only keep a few mascaras out at a time the rest of the mascaras I just don't want have that many open because I really unless I don't like it and I put it to the side I don't want to have that many open So I have one primer and then four mascaras and then this one is just kind of cramped full of the Eyeliners that I love and the lip liners. I love I'll tell you my favorite lip liner right now My lip liners are lippies right now are from by Beauty from the Sephora collection I have several of those those are in my purse and then from Colourpop And then my favorite high eyeliners right now are the high liners from Mark Jacobs And I have a lot of those and then the ones from Milani I absolutely love the ones the metallic ones from Milani They are such a good formula and they stay so well But I do have a few other urban decay and stuff from the Sephora 21 days of beauty in there And then the mirror that I have I'm not going to open it because it is a trifold mirror And I'm afraid that it would just really blind you I will link this mirror if you want me to it doesn't have any Magnification in it, but it does light up you can put a USB cord underneath here And you can use batteries on it for a backup if you want to okay I'm going to take these down now after I take these down you can tell that this is not an antique because it has This hole right here this hole is meant for electrical cords for anything you need whether it be your curling iron hair dryer Or the electrical cord that will go to my mirror So over here in this corner is where I keep all of my setting sprays my primer sprays and my refreshing sprays Love having all of those. I think my very favorite is that Scandinavia one Which is the tallest one in there So that is one corner and then I have six little doors right here have some q-tips in here It has a replacement for my Shiseido eyelash curler And then it has all of my eye shadow primers that I have been trying and some Lash glue which I heard they ever use lashes. I don't even know why they're in there So that is not very full which I'm glad this next door is very full this one is all of my Concealers always wanting to test out and bring you the best concealer that there is and some of these are really good Some of them aren't so good, but I will mix these together too and make what works for me too So I did keep a lot of those concealers because most of those are already are still on the market right now And then this drawer right here is a lot of my eyebrow products an extra eyelash curler Some eyebrow setting gel and a couple eyebrow products that I try now down here. This is kind of another overflow drawer I have perfume samples. I got this three piece from Smashbox it's during their holidays and it's got their photo finish primer their primerizer and their primer water Love that and some eyeliners that didn't fit into that turn thing with all the brushes. This is my most of my Primers and I keep these because these are the ones that I really like this one I just got during all those 21 days of beauty from derma blend not sure about that one yet But I have all these others if you'd like me to review some primers I can do that and tell you which ones are my favorite up here is my true overflow drawer I have things that I love that I know I needed to pick up like these two Lip liners from bite beauty bite beauty is Discontinuing them and I just have things that I haven't opened yet a couple of mascaras and extra eyebrow tweezers Best ones are from elf and they cost you two bucks. Gotta love it and then over here in the corner that is where I keep all of my foundations and I don't actually have as many as I used to I had three times that many foundations But I just decided that it was just such a waste So those are some that I really love my flower light illusion is one of my favorites My juvious place the Knicks born to glow all of those so in this little corner drawer over here I did the same thing that I did with my lipsticks I downgraded all of my lip glosses to that if they can't go in this drawer that I'm not gonna keep them The other thing is I decided that if I can't show it on camera Then I don't want to keep it and the reason I can't show it on camera is because they don't have it anymore I did keep a couple that I had like really fallen in love with that You can't get anymore like this one is from land comb But honestly you guys I just decided that I just wasn't gonna do that anymore okay, so this drawer right here is my blush highlighter drawer and This drawer was so full that I had stuff stacked across the top I couldn't get into it at all and I just decided that I was just going to keep the things that I felt like I could show you or That I absolutely truly loved and I'll show you one of the things that I'm talking about like this little rock blush I love this blush, but you can't get it anymore So this is really probably one of the very few things that I kept that you can't get anymore Everything else in this drawer. I can show you on camera and you guys won't be like well That's discontinued. So I just really liked pairing this down and having it very minimal It's quite empty compared to what it was. You can see these are laying down Everything was standing up and everything had its place and so yeah I hardly have anything in here compared to what I did have so and there are a few bronzers in there as well And then in my middle drawer right here, I kept like complexion products So I have my beauty sponges back here and then I have a couple of my primers right there my elf and my tacho Which I use almost every day and love and then I have all kinds of other powders in here Most of these standing up right here are powders that I absolutely love I do have this bronzer in here because it wouldn't fit in the other drawer this little thing right here Does have a few of my eyeshadow single eyeshadows in it But I had so many single eyeshadows and different things that I just wasn't using that I just decided that they were all going that had whole other huge Testive drawers that I had makeup in and everything went if it couldn't go in here It was going so these are some things that I'm I'm testing out But there are also a few things that I really love that I kept and I knew I would continue to use and There are some really good powders in here you guys.  This drawer is a mess. It always is But I love to make sure that I keep my everyday makeup in here With the exception of these products that are back in here. These are my wet and wild highlighters These are my two hour glass edit palettes, and then I have my mark Jacobs bronzer My loomi bronzer and my Laura Geller bronzer all back there Those are things that really couldn't fit into drawers And so I just kind of keep them in my drawer that I catch everything in So this is my everyday drawer This is the stuff that I reach for every day that I use every single day without fail And I'll put whatever foundations that I'm testing out at the moment. Whatever Concealers I'm testing out my bronzer that is in here is almost always this Sephora bronzer I think I've mentioned that recently my eyelash curler my eyeliner sharpeners my high brow My lip gloss that I wear to start every single day my setting sprays that I'm using at the moment And I'm really trying hard to use up this mark Jacobs Highlighter that's liquid because I've had it for too long, but I'm trying hard to use that one up My primer sprays or you know, whatever sprays illuminating sprays my nyx multi-tasking mixing medium I need to do a whole video on this you guys because this stuff is awesome And I think so many of you guys have heard that for me before things that I really just love using two-sided Tartist Felt and Coal liner in there my eyebrow product that I use my eye primer A new mascara that I haven't gotten into My mixing colors my mixing pigments and of course my fan for after I spray my face So that's my everyday makeup drawer. This is where the catch all gets right here But I know that if I open this drawer almost everything I need in here is going to be here for my makeup Look okay, what you're seeing right here is the corner to my nightstand And then that's a little vanity desk that I found at a yard sale And then the top of thing I found separately at a yard sale I'm getting ready to refurbish that in order for it to be white for my room and then to kind of match my vanity But I'm going to get closer to this so you guys can see what I have So this is where I store all of my eyeshadow palettes This is my laptop that I do all my editing on And then the rest is eyeshadow palettes or face palettes over here. I have all of my juvia's place I have my la girl that I talked to you guys recently about I do have a couple color pop in here This is a little music box that my mom gave me when I was sick. I love it Anyway, these are the gingerbread palettes from Too Faced There is the morphe jackland hill little palettes These are the hood of beauty obsession palettes in the gemstones that she had the urban decay born to run The saharan palette from alter ego and then over here. I have three from makeup revolution one from bh cosmetics This is the new so jaded palette the emily palette Three morphe palettes right here and then the jackland hill morphe This is when I made up myself of colors that I love that I just did single pans This is a melani palette that is from laura galler and then looking right over here is all of my Anastasia palettes with the alter ego temptress palette This is the dose of colors desi and katie palette friendication. I think is what that one is called I love that one. So i'm not going to get rid of it I do have the hood of beauty new nudes and then the two dupes That they had one from alter ego and then the other one from revolution pro I have a couple of tart palettes that are no longer available My flower beauty glow powder, which I absolutely love an elf palette And these are palettes that some of them are no longer available But like the jackland hill one I kept this specifically because I wanted to have those pressed shadows or not shadows Those pressed highlighters. So these up here are all blush palettes And then over there are my dominate cosmetics I do have a new one from wet and wild over there that i'm trying out and then I have the huge morphe purple palette Which I absolutely love. So that's everything for today's video I hope that you did enjoy seeing how I organized my makeup here I have a very small space and I have a very small space to film into but I love it You guys it's so much better than where we were before Please give this video a thumbs up if you do like seeing organizational videos I'm sure a few other organization videos will pop up if you do give a thumbs up So I appreciate you guys so much. Thank you for giving me a bit of your time today And watching what I kept from my brutal makeup declutter. Thank you guys so much for being here with me You're the very best part of my day. Love you so very much. Take care of yourselves And I'll see you in my very next video. Take care. Bye. Bye


Sam here with late in sports cards have been one box of 23 Bowman Chrome University football hobby for Michael B So good luck here Michael. See what we've got Hurry Chapel Caleb Williams early risers and Jason Marshall yellow Cameron kitchens to at 75 Devon leary Shawnee X Fractors Tavion Sanders and Dane Keith Jordan Travis early risers Harold Perkins Gabriel Josh Williams late Corum Jordan Travis Throwback What going else? Yeah, a couple of Jordan Travis's so far X Fractor Mason Taylor Michael Williams Jalen Wright first the fractor Drowne Dickie Rickard Reese first refractor Kobe Prentice Trey Benson Jaden Dolora first base auto for Arizona Nice one there Michael Brock Bowers. What's happening at Bowers? Yeah Purple this week. I know everyone Trey Benson to 399 for the Knowles Nate Wiggins Drake may Kool-Aid McKinsey refractor first Auto to 499 not first but refractor auto to 499 for Alabama Marvin Harrison rookie Mechagooka for Travis ours 55 insert Club Nick Kool-Aid Yeah, 299 or 399 Cooper Dejean White DB alert White cornerback alert Yes, there's around 3 399 Donovan Smith for He's actually Houston, huh Houston that's where he is now Okay, I got you Marvin Harrison good calling that would never know Victor Rosa first refractor roll skis jacquory Brooks first refractor and AJ Jefferson. I'll do it for your break there Michael Thanks again for the support buddy. Got him right out to my man