And we have a testimony of Angela come right up and she's going to share how she prayed for her friend's mom. So tell us your testimony. During the end of month of May, I found out my friend's mom had cancer. In the beginning of this month, she had to get surgery and for her cancer. The doctor said she had a very low chance of surviving. So like two weeks ago, I prayed as I was home alone. So I just prayed and I prayed that her mom would survive the cancer, survive the surgery, and she would be healed from cancer. Yeah, I prayed on April 2.  Amen. Praise God. Praise God. So the prayers, thank you Angela. So the prayers that we pray at night prayer, when we pray, I know we bring a lot of needs, you guys. Our prayers, they do not go in vain. In night prayer, we also pray for her mom's friends. And so she received her healing. But you know what? Angela has been going to her church for a while and she's seen so many testimonies of cancer that she knew that God could do it because of all the testimonies that we hear.


And River State Governor Simeon Eli Fuwara on Thursday said his predecessor and Minister of the Federal Capital Territory, Janssen Wike, remains his principal. The governor stated these in Port Harcourt River State at the rededication surveys of the 2023-2024 legal year of the River State Judiciary that was equally attended by Wike. The event was held at St. Cybrans Anglican Church Hospital Road in the state capital. He described the recent political crisis in the state as a thing of the past, calling again for peace in the oil-rich state. Fuwara noted that River State can, if the actors and their supporters are peaceful. Wike Wu is the FCT Minister arrived at the church service almost at the end and had a handshake with his successor before sitting on the same road with him while the service lasted. Hello, hope you enjoyed the news. Please do subscribe to our YouTube channel and don't forget to hit the notification button so you get notified about fresh news updates.


In this video, I'm going to talk about the properties of exponents. I'm actually going to do about five examples of the different properties of exponents just to give because with exponents There's a lot of different variability that we can have so I'm just going to go through a few examples This should be pretty quick This should be a pretty quick video But hopefully I'll be able to show you enough examples so we have a better understanding of what's going on Okay, so my first example something pretty basic What if I have three to the negative second okay now if the if I have a product if this is part of a problem I might be asked to simplify this I might be asked to evaluate this something to that effect Okay, so what I the first thing that I notice is that this is a negative exponent remember negative exponents make fractions Negative make exponents make fractions So what I'm going to do is I want to make a fraction out of this So what I'm going to do is I'm going to change this position and put it on the bottom of the fraction It's right now is on top of the fraction if you need to put a bar right there to see that it's on top Just take this and put it to the bottom when you do that we go from a negative two exponent to a positive two x point Okay, and then from there once I have changed the position once I have made the exponent positive I now need to evaluate this One on top three squared on bottom is nine. Okay, so there's a simple example Just with negative exponents. Okay, so let's do another quick example What if I have a fraction? Fractions can sometimes be a little bit a little bit more difficult There's a couple of different ways to look at this one. I will try to show you the simplest way So what I'm going to do since I see this negative exponent This negative exponent is being applied to both of these numbers So that's what I'm going to do I'm going to take this negative exponent and apply it to the two negative two and apply it to the three So it looks a little bit like this two to the negative two over three to the negative two Okay, now this problem is a little bit confusing because notice now we have a negative exponent in the bottom of the fraction So what I'm going to do is just like the top example I'm going to change its position so that We have positive exponents So on this first example up here on this first example up here It was on top and I brought it to the bottom which was fine But now here some are on top some are on bottom. So all I need to do now is just simply change its position So this is two to the negative second. It's on top. I'm going to take it to the bottom This three here three the negative second. It's on the bottom. I'm going to change its position and bring it to the top Okay, once I change its position the exponents actually become positive And once I change position the exponents become positive now after I have now that I have positive exponents now I can actually evaluate the Exponent this will be three squared is nine two squared is four. So that would be simply nine fourths Okay, that's one way to look at it another way to look at it if I can do this rather quickly You'd also think of it this way if I take the fraction and flip it initially if I take the fraction and just flip it initially That actually will change the exponent outside to be go from a negative two to a positive two This is another way to do this and then all you simply need to do is take this Exponent of two and apply it to the numbers inside. So three squared is nine and two squared is four That's another way to look at it. That's one of the really quickly go over that Okay, so those are two examples of Using exponents. Let's go through a couple more Here's another one. This is kind of long-winded one three z to the seventh times negative four Z to the second Okay, now I got numbers. I got variables. I got exponents. I got all sorts of stuff in here My my advice to you is just look at one thing at a time So the first thing I want to look at is I'm gonna look at the three and a negative four Those are numbers. These are constants. They're just numbers. So actually I'm gonna figure that out first So three times negative four is just going to be a negative 12 Okay, so I've evaluated the number portion of this first Okay Now what I have here is z to the seventh and z squared these z's are multiplying times one another so go look at your Property your power first you need your product property if I multiplying like bases I need to add the exponents multiplying light bases add the exponents So this is going to be z to the ninth power seven plus two gives me nine Okay, so that's what that would simplify to again when you look at something like this don't don't get Overwhelmed by everything that's going on just take one thing at a time All right, especially for this next one if I have y z to the third over Z to the fifth quantity third Okay, so this is a different type of example I get a lot of stuff going on But again take one thing at a time and you'll do just fine So as I look at this one thing that I noticed is that inside the parentheses I see a z on bottom and a z on top I'm going to evaluate that first and then I'm going to worry about the three exponent outside first are next Okay, so why is I only see one why so that doesn't is an effect and I see Z on top See how about I see three Z's on top five Z's on bottom So if I'm dividing like bases subtract the exponents, this is going to be y to the This is one way to explain now. I'm actually going to change this up here in just a second Okay, so if I take three minus five I get a negative two But there's kind of another way to explain this it might be a little bit easier I like to think of it this way. I have three Z's on top five Z's on bottom They're going to cancel so that I only have two left on the bottom Now notice this is z to the negative second if I have a negative exponent what I need to do with this is take this And put that on the bottom So notice z to the third z to the fifth three on top five on bottom They're going to cancel so I have two left on the bottom I think that's kind of a simpler way to look at it Yes, you can have Z to the negative second in there, but I would prefer that we have this way All right now that we've evaluated with the Z now let's take a look at the three I'm going to take that three and apply it to everything inside y to the third Z squared to the third Z squared to the third Squared to the third this is a power of a power Property that I'm going to use here if I'm taking the power of a power power of a power. I need to multiply The exponents so that two and three I actually multiply together to get six and that was a little bit tough there's a lot of properties that are intermixed into Into a problem like that. There's a lot of different ways to do it The only the only real good way of understanding these problems is to practice them over and over again Practice and a little bit of experience will get you to understand these a little bit better. Okay last example That I want to go with is having to use scientific notation 4.5 times 10 to the negative fifth over 1.5 times 10 to the sixth okay now in mathematics One place that we that is that this is that this is used in the real world It's an astronomy or even chemistry of where you use very very large numbers in astronomy You use very very large numbers light years Distance of the moon is from the earth. These are very very large numbers And chemistry you use very very small numbers the size of an atom Things like that. So what I'm going to show you just real quick using scientific notation using property of exponents We evaluate something like this Okay, so now this looks kind of confusing and you might try to plug this into a calculator But it's really very messy if you try to do that. So we're going to evaluate this using our property of exponents Now this right here 4.5 over 1.5 I can actually divide that and then these over here notice base of 10 base of 10 This is just going to be like the other problems that we've been working with okay But first I'm going to look at 4.5 divided by 1.5. Actually I'm going to look at that as 45 and 15 15 goes into 45 three times if I put a decimal here decimal here 1.5 goes into 45 three times Okay, so 4.5 divided by 1.5 is actually just three simply enough Okay, so now I'm going to look at this part here And I'm going to use the same rules For exponents if I have a base of 10 and a base of 10 if I'm dividing if I'm dividing like bases I need to subtract the exponents negative 5 Minus 6 so this is times 10 to be negative 11 negative 5 minus 6 got me that negative 11 Okay, so there's just an example one example using scientific notation It's not really that difficult to use scientific notation But it does help just to have an example so that you've seen one before and those are five about five examples For all the properties of exponents


Let's look at another factorization method that's called Pollard's P-1 factorization. And this is based on the Euler Fermat theorem. And this goes back to the idea, suppose I have two prime numbers and N is a product of those two primes. The Euler Fermat theorem guarantees that for any value A relatively prime to P, A to power P-1 is going to be congruent to one mod P. Now, it's worth pointing out that we don't actually know the value of P. On the other hand, it's a fairly standard strategy in mathematics to say, well, what happens if we just work with it and see what, and then maybe something interesting will happen? So let's start with this idea that I know this congruence holds, even if I don't know what the value of P is. I'll suppose P-1 is a factor of some number L. And then I know that L is P-1 times K. So what can I do? Well, A to power L is going to be congruent to A to power P-1 times K, because it's the same thing. This inside expression here, though, is congruent to one. So that means this entire expression also evaluates to one. And so that tells me P divides the difference A to the L minus one. And remember, P is a factor of N as well. So that tells me that the greatest common divisor of A to the L minus one and N will also include P. And that gives us a way of factoring N with one slight problem. How do we find L? Because if A to the L is not congruent to one mod P, this doesn't work. So I have to know what the right value of L is in order to make this work. Well, it wouldn't be an algorithm if we didn't have some way of finding it. So let's approach that. So this is Pollard's P minus one algorithm. And if I want to factor some number N, I'm going to choose a base that's relatively prime to N. I'm going to evaluate A to the K factorial for K equals 1, 2, 3 up to some practical limit, whatever we want to do as far as we want to take this computation. And we'll find the greatest common divisor of A to the K factorial minus 1 mod N and N itself. And you'll note that because we're finding A to the K factorial, there's a good chance in this product that we're going to include P minus one. And so we'll get A to the P minus one to something. And that may hopefully be congruent to one and our greatest common divisor, A to the K minus one and N will be one of the factors of N. Now, again, we might not have the actual value of P minus one in there. So it's possible that our greatest common divisor is going to be one. And it's also possible that we may have too much in there. And our greatest common divisor is going to be N itself. And so we look for any non-trivial greatest common divisor and whatever it is, that's going to be something that will factor N. For example, let's try to factor 1,403. So I'll take A equals, oh, I don't know, two. And I'll evaluate two to the K factorial mod 1403 for K equals 234. And so on as far as I care to work it. So let's figure that out. Two to the two factorial congruent to four. And I'll find the greatest common divisor of two to the two factorial minus one in 1403. And the GCD is one, which is a trivial factor. So we'll go on to the next. Now, one of the things that makes this computation only feasible is that when I evaluate these higher powers, remember the factorials are defined recursively. So three factorial is two factorial times three. So I can evaluate two to the three factorial as two to the two factorial raised to the third power. And that's going to be four to the third, 64. And again, I'm going to look at the greatest common divisor of that number minus one and 1403. Again, that's going to be one, so I'll continue and find the next term. So four factorial is the previous to the fourth power, 142. I'll find the greatest common divisor of the number minus one and the number of interest. Again, one, so I'll continue. The next greatest common divisor of the number minus one and 1403 is 61, which tells me that 61 divides both of these. I didn't actually care about dividing this, but 61 divides it into 1403, gives us the other factor. Oh, let's take another example. So I'm going to take two again and I'll evaluate a to the k factorial. So again, greatest common divisor of the number minus one and the number of interest is one, trivial factor, we'll go on to the next. Again, every factorial is the preceding factorial raised to a specific power. And again, we find the greatest common divisor, number minus one, number of interest, again one, so we move on to the next. The fourth greatest common divisor, again, one, move on to the next. Greatest common divisor of the number minus one is 41, and that gives us one factor.


Welcome everybody here. My name is Patrick Lee. I'm a software engineer running Cassandra here at Walmart. I've been working on Cassandra for the past eight years. I'm Andrew Weaver. So welcome. We're part of the Cassandra team here at Walmart. So Cassandra goes back a long way at Walmart. We started back in around 2011 in the 0.7 days. So today we're going to talk about multi-clown at Walmart. So this is just going to be kind of what we go through today. So I'm going to show you a little bit about kind of the current state where we are with Cassandra at Walmart, why we chose a multi-clown strategy, a little bit of detail on what that multi-clown strategy is. And then we'll try to get into some Cassandra specific stuff because I assume those that are joining this talk are more interested in the Cassandra side of things. And then some challenges we face and we want to leave some time at the end for Q&A. So just a little bit about our current footprint of Cassandra at Walmart. So we've got over 300 production clusters today, multi-pedabytes. Our cloud journey really started in about 2015 where we started migrating a lot of our bare metal clusters to the cloud. Since then we've had like 20 times growth of our Cassandra offering at Walmart. And every year it just continues to grow. Our entire footprint is 100 percent DMs. We've got almost every one of our clusters upgraded to Cassandra 4 from 311. We've got a couple clusters running 4-1 and we've actually got at least one that's running the 5-beta so we can get our early start on testing. Okay, so why multi-cloud? Multi-cloud is a little bit of a hot topic, maybe not quite as hot as AI, but so multi-cloud at Walmart, why do we want to have a multi-cloud strategy? So what that means for us is being able to use multiple cloud providers to run our workloads. So multiple public cloud providers as well as our own OpenStack private cloud. So we're a very, very large company and so that means we have a lot of different use cases. So yes, a lot of retail use cases, but there's also just a multitude of other use cases. Finance, HR, inventory management, all the things around retail and then compliance and just many, many different types of use cases in a company of our size. And so those use cases are going to have different needs. And so some of the reasons for going to a multi-cloud strategy, we do want to prevent vendor lock-ins. So obviously if you're tied to a single cloud provider, you know, you don't have a lot of options for moving or moving to another cloud provider that may suit your needs better, could be difficult. And along with that, we want to enable best of breed. And what we say, what we mean by best of breed is looking at the different public cloud providers and the feature sets that they offer, whether it's part of their IaaS offerings of compute and storage or looking at the platform as a service offerings, such as databases or other services. Like I said, we have a lot of different use cases. And so some of those products from one provider might work better than another provider for a specific use case. And so yes, while you're trying to prevent vendor lock-in, you do want to enable the use of some of these products. Yes, that's kind of an intention a little bit. But with the broad set of use cases that we have, being able to select the right tool for the job is very important. And then obviously, driving down costs. So if we can force our cloud providers to be competitive with each other and compete for our cloud spend, that helps us reduce costs. And so our multi-cloud, so the process that we kind of went through, first off, you just have to modernize your applications. Really the stateless applications, business applications, you have to get those ready to be able to be deployed in a portable fashion. And so part of that is Kubernetes and containerizing applications. And at Walmart, that's what we call the Walmart cloud-native platform. That's Kubernetes with a lot of automation. So developers are free to just write their code, just add a few lines of YAML and be able to deploy that application anywhere. I'll let you take the rest of this one. So that's the breed. So kind of what Andrew was talking about before. Obviously, different cloud providers are going to have different strengths. Our private cloud is going to be quite a bit cheaper. In public clouds, you've got your compute, you've got your storage, you've got the other pass offerings. So being able to pick and choose and being able to, for us, for having Cassandra to be able to deploy across any of them to meet the application needs and to be local to the application has been a real key factor. As far as pass products go, yes, some pass products are good. What we've learned is it's very important to read the fine print, do a lot of testing, and really get to understand those pass products. A lot of the limitations that you might run into for some of those might not be obvious at first. It might sound very great, but once you kind of get into the thick of it, that's when you start uncovering some of those. So it's best to spend that time up front and try to get a very good understanding of what those products are. Determining your data strategy placement. There could be different security considerations to be able to choose a particular cloud provider for certain data. So good idea to have a good placement strategy to figure out what data would we want to place in what provider cost could be a consideration for that as well. One of the very key things for us for Cassandra to be able to deploy across all these different cloud providers is having a central consistent observability stack. So metrics. If I deploy Cassandra, I need to be able to see the metrics. If I have a cluster that spans multiple providers, I want to be able to see a holistic view of that cluster to know how that cluster is behaving across those different providers. So our multi-cloud strategy started 2019. We've been running through 2020 up till now, and it's been very, very successful for us. We've been able to reduce cost and really accelerate the developer experience. So what is this thing? We keep talking about this multi-cloud. So for us in multi-cloud, we're using our very large private cloud, Azure and GCP. So what do we do is we get these different cloud providers together spread across the United States, and then we get those very close together so that there's milliseconds in between each. What this allows us to do is the application can be deployed in Azure, for example. Let's say they're using some of the Azure technologies, and that's the best fit for their application. So we're able to deploy the Cassandra clusters, have rings of Cassandra cluster in Azure, but then maybe they have some other use cases. Maybe they have some more data warehouse, ML type of workloads that we're going to run in our private cloud or in GCP. Maybe those are going to be more storage density, so we're going to deploy it in our private cloud to save cost. So I can have a Cassandra cluster that spans both our private cloud and our public cloud regions. So just have rings of Cassandra kind of all over the place to service different workloads. This allows us to be able to dynamically change and move in between those. It gives us the flexibility to take advantage of some features, let's say in private or public cloud, where I can more easily switch between let's say an 8-core machine to a 32-core machine and able to vertically scale to add more CPU for a particular event. Let's say they're going to have Black Friday event and we're going to have a surge of traffic. So instead of vertically scaling out a cluster, I can just horizontally scale out a cluster. I can vertically scale a lot faster. So given those choices in the regional triplet, we have multiple cloud providers. So how do I choose where I want to place my data? These four different factors, these are the main factors that go into making these choices. So obviously, you've got public versus private. And so in the public cloud, of course, cost is going to be much higher than if we run our own cloud. We can run our own hardware a lot cheaper than using somebody else's. The feature set, like we've talked about, you got everything in the kitchen sink in public cloud. The private cloud, it's a much smaller feature set. And of course, the scalability as well. In public cloud, if you have a bursty workload or you frequently are scaling up, you're more able to do that in the public cloud, whereas private cloud is more of a steady state type of deployment. And then performance. Performance, in order to keep costs low, our tech refreshes in a private cloud probably aren't as frequent. And again, it's not. You don't have the broad set of options that you typically have in a public cloud. So those are some of the factors that go into how I might choose to deploy a cluster, or maybe if I need to move a ring in a Cassandra cluster from one provider to another, we have that flexibility. And yeah. And so like, for instance, on performance, Azure has 11 different virtual machine series. They have five different managed disk offerings. Google has 12 different machine series, and you can customize them. Whereas in private cloud, it's a lot more, it's a much smaller set of choices. And so some Cassandra-specific things, considerations, if you're looking at a multi-cloud strategy. First, I'll just let you know, it was pretty easy. And this is a bit of a callback slide to our presentation in 2015 when we were talking about bringing Cassandra into Walmart, into a large enterprise. And we had a slide that just said, it was hard. It was like super hard to change people's minds and to build a case and get that in. But here we are. Eight years later, and we're able to say it was easy because for Cassandra, multi-cloud, multi-region, it just works. So as long as you've got that network connectivity between your cloud providers, deploying a cluster from one provider to another is just adding another ring to that cluster. So the native multi-DC replication is just business as usual. And obviously, you have to invest in automation to do this. So for us, we have a cloud deployment platform called OneOps that abstracts away all those differences between cloud providers. So we're able to leverage that. So you definitely have to have some automation in place. And benchmarking. Benchmarking is very important. We've done many, many different rounds of benchmarking, different configurations, and different providers, again, to just achieve that parity between providers. So because we want to offer a similar user experience at a similar cost, so particularly in the public cloud, you don't want one provider to be significantly more expensive if we can control that by being smart about what VM flavors you choose, the storage configuration, things like that. And so Cassandra, it just offers a better user experience in this type of situation than other data platforms. Like I said, it's business as usual. It's just another ring. It just happens to be in one cloud provider or another. And of course, we have to give our network engineers a lot of credit. So we have some really good cloud network engineers at Walmart that make this all work and make it look seamless from a network perspective. And then from a developer perspective, it does look a little bit more complex. I've got to try to co-locate my app with my data, and there's a lot more choices out there. So understanding what your environment is and where you're deploying and that you configure your driver to talk to the right endpoints and use the right local DC name and things like that. You just have to make sure you get that correct. And then some of the challenges that we faced. Like we said, these hardware differences between cloud providers do matter, VM flavors. So you really have to look at what you're getting in a particular VM flavor. Are you getting full cores? Are you getting hyper threads? Do you need to use a 32 core machine in one provider where in another you might use a 16 core? Discs also very different strategies if you look at Azure and Google and slightly different cost models and how those work. And so you really need to know where the throttles are. So most cloud services, including VMs and storage, they tend to have limits in different ways. And GCP and Azure do their storage. Those throttles are implemented very differently. So you have to understand that. And really it comes down to benchmarking it. So run those tests and try different configurations to understand what works and what doesn't and what gets you the most equivalent performance at a similar price between all of them. Because you really don't want any surprises when your customer goes and deploys to one cloud provider or another and then their cost shoots way up. So again, a similar user experience is very important. I'd say keeping up with the changes in those cloud providers too. So we have had to spend quite a bit of time up front obviously trying to evaluate the different VM SKUs, the different disk and everything to try to match that parity from where we started with private cloud. So we went to private cloud, we started moving into Azure. How do I get a similar experience, similar if not better experience as I move? And then again, when we started out in GCP. Okay, well then several years past, now there's what we were DSV2s at one point in Azure. And now they've got V5 hardware and everything's kind of changed. So over the past, let's say over this past year, we've been going through and reevaluating a lot of that stuff and then migrating to newer hardware. So again, we had to kind of go back to the drawing board, spent a lot of time up front trying to reevaluate everything that they have to offer. What's the right configurations? What's the right disk types? Is things the same as what we had or do we just need to switch to a different VM? Then when GCP comes in the mix, their offerings are very different as far as how the caching works and how the IO works. So how are we able to get the same for the same core account for the same storage? Or do we have to kind of get creative and try to add in some caching, some disk caching like some LVM cache is kind of one of the things that we kind of looked into to try to help better offset and get the performance on par with Azure. Okay. All right. Well, that is all we have as far as presentation. We can move into Q&A. If anybody has any questions, we do have a microphone up here so that people would be able to hear you on the stream. So we'll open it up for questions. Sorry to make you have to come all the way up here, but I don't know. Come on. You're first. Thank you. So my question is you mentioned that you have multiple cloud providers for the Cassandra ring. So for the same data center, do you use only one cloud provider or within a data center? Let's say you have three replicas. So you have one copy in Walmart. Another one is Azure. If that is the case, then how do you deal with even a smaller latency difference between the zones between different providers? Okay. So when you say data center, you mean Cassandra data center like the Cassandra. Okay. So there's different ways we can do this. So if we have a Cassandra data center, so one ring of a cluster, that is going to reside within one provider. So we're not doing anything. So when you see the different cloud provider logos in these regional triplets, so that would be one ring within one provider. So we're not doing anything like, we're not treating them as racks or anything like that. So that's been the best way to do it. I don't think you really want to spread a ring across multiple providers, but use multiple providers by having different rings for each provider. And the latency there, it's single digit millisecond latency is the goal between any provider within a triplet region. What do you use for observability? You mentioned that that's a key constraint you have to work. Observability. So we have an internal observability team. And a lot of that is Prometheus based and Grafana. So we're using the M3. The M3 is the data store, the exporter, the current exporter that we're using for Cassandra is the Instacluster metrics exporter. That seems to work the best for us. Yes. Thank you for interesting presentation. Simple question. How do you design the benchmark? Do you use a typical mix of YCSB or something like that? NoSQL Bench has been used a lot. Back in the early days, it was Cassandra Stress, which I still love, Cassandra Stress. It's a great tool. But NoSQL Bench offers a lot of flexibility. I think there's a talk today or tomorrow. Jonathan Schuch is giving that talk about NoSQL Bench. It's a very, very powerful tool. I would recommend setting aside some time to really look at it and understand it before just trying to run with it. But yeah, NoSQL Bench works really well. Okay. But I guess there is some gap between the real workload and SQL Benchmark. So how do you overcome? A gap between we are workload and SQL Benchmark. Yeah. So what we've got is what we start out with is some basic benchmarks, mostly around key value lookups. I mean, we try to benchmark different payload sizes, right? So that is not necessarily a real workload, but that gives us a baseline that we can use consistently across those cloud providers so that if I know, if I deploy this type of workload in private cloud, here's kind of the performance that I get. And then we use those same benchmarks every time we do a different cloud provider or do a different VM skew. So that way we have at least a common baseline. And then with that, so like when we were doing our Cassandra for migration and benchmarks, that was also evaluating some additional VM types. We ended up partnering with one of our application teams where we took one of their real use cases and deployed it alongside Cassandra 3, Cassandra 4 in Azure and private. And then we were able to have them run their applications while we were taking care of the backend database, they were running their application and we partnered together to put together kind of like a white, an internal white paper and stuff to figure out performance cost. And I did actually kind of match back. We saw similar gains, similar trends with our static benchmarks that we do. Okay, thank you. You guys mentioned about automation built to add new token rings, right? Do you have any automation built to scale token rings? Yes. So in the, like I mentioned, the tool that we use is called one ops. So that's a home grown tool. It's open source. But that is an automation platform. And part of it is it has automation for Cassandra. And so adding a ring is captured in that and then scaling up scaling out horizontally and scaling up vertically is all within that. We have a few little add-ons that we use in there to drive that automation. But we do use VNodes. So that ownership is a little bit easier to deal with. But we end up doing, we've gone as low as four, four VNodes. Thank you guys for the presentation. So the question I have, you guys talked a lot about the infrastructural side of things. So what about like application development side? So you said that Cassandra was easy. And I believe before Cassandra used a lot of like traditional relational databases, right? How was that like transition from relational model to non-relational model? Was it like easy? Like I'm trying to put every single app and you know, model to Cassandra now? Or are you still like debating between depends on the use case? So as far as data platform within Walmart, we have relational offerings and no SQL offerings. And so there's really just, if your problem is, can be solved with a relational database and maybe you have some needs for transactions. And you need to be acid compliant, which of course this is changing with 5.1. But going with the relational database is still, I mean, in my opinion, still a valid choice. We don't necessarily want to Cassandra all the things, because part of what we do is when we onboard an application team is we do ask a lot of questions upfront. And one of the major questions is going to be, can you tell me all of your select queries that you're going to run, right?  And if we can't help them design a proper data model, then it's not going to be good for either of us, right? Because things just are going to fall over. You're going to have issues that may not even be apparent upfront. But over time, you know, you're going to have a problem. And if it's data model related, it's really hard to change quickly. So yeah, there's definitely space for other databases out there. But we do have a lot of Cassandra use cases, and we do try to work with, as part of that onboarding process, really design the data model, make sure they understand what the consistency is like, eventual consistency that they can tolerate that, and that they're configuring their apps well. And really, you know, basic rule of thumb is use the Cassandra, the data stacks, typically it's data stacks Java driver, version four and up, specify the local data center, and then take all the other defaults. And you will be successful 99% of the time. So that's, that's kind of what we work with our application teams on. How could that answer your question? Yeah, partially. So my another question. So how do you like what's your approach to maintain data consistency? Let's say you have like multiple tables. Are you trying to spread data like across multiple tables? So it's trying to keep data, let's say in a single table, create like secondary indexes or, you know, something like that. Okay, okay. So yeah, if you have, so you got a denormalized model or sorry, or whether or not it's used, if you use them, no, no, we do not use materialized views or secondary indexes.  We do have some clusters out there that have secondary indexes and apparently it's working fine because we don't, you know, we don't get any alerts or anything. But yeah, materialized views are disabled everywhere. Bad experiences really probably not a fully baked feature at this point. I actually had it crash a cluster. Yeah, just because they decided to create it on top of a fully, a full table. And it just wreaked havoc for a while. So, so yeah, there's still those, those sticky points of if I've got to have update two tables at the same time, right? Do I do an atomic batch, right? And we really try to refactor those use cases out. Item potency is huge. If you get into this read, modify, write cycle, that may work for a while, but it doesn't scale as well. We get people trying to do a lot of, I just want to store a JSON blob, right? And that forces you into that read, modify, write cycle. And it forces you to read the entire payload every time. So we talked about benchmarking different sized payloads. That's very important because you'll run into different, different limits if I'm doing a one K read every time versus I'm trying to read 16 kilobytes of data, right? That's a very different use case. And it's not going to scale as high. So, yeah, there's a lot of data model and use case things that we really try to work out up front because you can run into trouble pretty quickly if you don't put thought into that up front versus third normal form and a relational database, you know, it's, it may not give you some millisecond latency, but it's comprehensible and it's probably going to work for you.  Thank you. It should be it. Yeah, I think, I think we're at time. I don't know. Let's take one. I'll make the call. We can take one more question. Thank you. If you can, could you elaborate on the use cases that maybe you've seen Cassandra like shine and then maybe some where you thought it might have been a good idea and then it turned out to be not so, not so high. Let's see. It's easier to name the bad ones. So anything with really large partitions? Unless it's like time series and use time window.  Okay. So we're working, I'll tell you right now, we're working through one. We've got the table is a couple hundred megs and like 19 megabyte max partition size. And we're getting queries of boarded due to tombstones. Yeah. So really tiny table, really tiny partition, but you can't read it because there's just so many tombstones. And it's just the way the app works. And we got bit because they did have, it was a zero GC grace, I think it was a zero. It was really low GC grace value and we got zombie data. So this is a really tough one to solve without just upping the tombstone limit. But yeah, I mean, like I mentioned, the JSON blob use cases and a lot of people complain about their latency and go to dig into it. It's like, well, let's do some optimization here. And those that have optimized and either modeled out as columns or really tried to shrink the size of that payload. They've seen big payoff in that reduction in latency. It's a lot more consistent latency as well. Treating it as a file system. There's ways that can work, particularly if you chunk up those files and, you know, spread them across partitions versus just dumping a large blob into the database. But, you know, I would argue object storage is going to be a lot easier to deal with and a lot cheaper. I don't know, other use cases, good ones, bad ones. I think that's been good.  We've had one of those that I wasn't seeing, but they never read it. So yeah, I don't understand. Heavy usage of like check and set and bad consistency levels, you know, trying to do each quorum all the time with no retry or anything. Yeah, if you have any questions, hit us up. After the fact, we'll be glad to sit down and chat. Yeah. All right. Thanks, everybody.