Hi, this boy Bellamy better not lay a finger on cricket. I will throw hands. Ladies and gentlemen, welcome back to the channel. My name is Jack the Buster. I, Shinobi, with episodes 150 and 151 of our One Piece Watch today. Our crew has been slowly trying to find their way to the Sky Island, making an acquaintance with Cricket and these two monkey brothers. Very kind folk, and since they have to find this bird before tomorrow to lead them to the only place they can get to Sky Island from, it appears to be very bad timing because Bellamy has made land, and he is coming for gold. If you're new to the channel, drop us up and hit that big boy bell and let's hop right into the episodes. Oh my God. Wait, did that look... Did that small bug do that? Well, I thought Zoro stepped in. It's a praying mantis. Dude, this bird is evil. The bugs control tower. I think they have their hands a little too full in here. What are those? Meatballs? I think they're bugs. Dung beetles? Oh, giant ladybugs. Excellent. Wait, this bird actually does control just everything. It's crush! He's trying to teach them a lesson. Yeah, I feel like Robin could just reach out and grab. I guess she doesn't know where it is. They're actually throwing salt. Oh my God. You gonna fight bugs? Thank God Chopper is here. Oh, they're pretty, and they're gigantic.  And why didn't Cricket not warn them more about the absurd things going on here? Saruyama Alliance. They said that name last episode, I think. Oh, he handles him. He handles him easy. Peasy. That's why I just see Cricket at a boy. He hasn't used his giant blade yet. Hell yeah, boys. What I'm talking about. It's kind of a weenie approach. This needs one opening. God dang, dude, Cricket smoothed with it. Let's just put him down. The disrespect, too. Okay. Okay. What is this soundtrack? Oh my God. Wait, his legs turn into springs. Wait, that's actually badass, though. Oh, no. Oh, wait, he missed. Wait, this is more power. No! This is more power for him to launch with. Oh, damn. Oh my gosh, Mary and Cricket, I guess. Who are you laughing for, bro? You got stomped. Oh, no. Oh, no. Oh, this is so sad. I didn't take long at all. And Bellamy was literally the only one that could take you guys down. Oh, no. The three... Oh, man. Like you said, to him goal that people struggle to find or get tastes better to him. It's more flavor. Oh, no. Oh. No shot is possible to die from that. Shoujo, stay down, dude! Where did man get the mic from? And where are the speakers? Oh, he doesn't need speakers, we've seen. Oh, my God. Damn, actually ruptured the earth. Does that not affect him? That's such a cool power, though. Boy, he's not even remotely affected by it. Damn, both monkey brothers are down. It's all up to Cricket. Surely Luffy and them were here. The only ones that also thought it did. He's not one to just let them leave whether he's hunted for all this time. Oh, Bellamy looks so crazed all the time. Tell me he's not dead, bro. If he is, and both monkey brothers are dead, I think it's a waste. They're too cool. Also, I just like their dynamic with Cricket, though. Oh, my God. Robin's on the page. Guys, you have to get back. Please, please, dude. Zilch. Okay, you didn't catch one either. Like I said, I think Robin is really holding out on us, honestly. It's right there. Luffy, stretch right now. Swipe it, and you just go. You don't want to kill it. Bellamy's already gone. Damn, they did already leave. And going Mary. Okay, they're not dead. They're definitely not dead. It's destroyed, bro. Boy, he's apologizing. Damn, even though everything was taken, he's still trying to get them on their mission. A real one. Cricket? He's like, we got it. Dude barely even seems concerned with what happened. Whole house destroyed. He doesn't want them to have to worry about it because he wants them to get to where they're going. And they know that symbol. Well, no, Luffy's pissed. One health. Damn. Zoro's just right on board with him. He doesn't need three hours. Oh my God. Giga Chad Luffy's like, I'll be back. Nothing else is going to stop him right now. Oh, this should be good. Is this the, is that the sniper? Oh no. He's part of Bellamy's crew. He's going to run into Luffy. News bird. Oh my God, wait. Is it going to be Luffy's new bounty? Oh.  Oh no. Oh no, they're going to know. No, it's the update.  He's like, oh, that's crazy. No, surely it doesn't actually say this. He's reading it over and over again. Do they know they're not messing around?  From how seriously people take it? You don't, you have to earn that bounty. 100 mil. That's much larger than Bellamy's. Everything that's pushed him is pushing Luffy now. It feels so good to see Luffy go all out against Bellamy while like, what were you just holding back before? He's coming. Okay, you can say you and Bellamy. Yeah, 55 mil. That'd be crazy if someone worth more stepped in. Oh no. I wonder if it's even going to faze Bellamy. Or he's going to realize that Luffy was just holding back.  Get out of the way. What a random nuisance. Oh my god, they're going to follow him. Done. Done. Like I said, I wonder if Bellamy will even be fazed by it or not. I don't think he, I think he's kind of mad that he's even being like compared to them. Beat the bugs up? He's laughing. Like I said, I'm not shocked that he's not, not going to be just, he's just like, oh yeah, like maybe they're not worth the bounty that they actually have. Oh man, he lacks very critical information. You think these are fakes? That's a good strat, honestly. What is the guy with the spiral? What the, what is that? That actually does make sense that people would fake bounties just to try and make a name for themselves. Although like, since we haven't seen any newspaper posts or it, you know, like any articles or what? But I mean, didn't someone say they heard about a warlord falling? But they just didn't know who to. I'm pretty sure that was like published news. Cause what, uh, Masura, one of the, one of the apes was trying to take that place or something. And Bellamy's got enough of an ego to take him on, especially after putting him down so easily the first time. Oh my god. He's like Batman. Or what? Hey, Luffy's got the dark shadows under his eyes. You know, he means business. Also, I love the concept of Luffy just standing on rooftops looking for him. And then eventually he just got so fed up. He said, hey, where are you? 151, let's get it. I knew he was just kind of trying to spring up right away. He's, he's matching his height. Okay, this could actually be a pretty sick fight. The thing on how strong Bellamy is. Kind of right. He's kind of right. Yeah, that's kind of right. Yeah. He's also a pirate. He can steal it if he wants to. Oh, that's sick. That's a cool shot. Give me a fight. I honestly don't want Luffy to put him down like right away. I hope it takes a little bit more. No way is he asking him this. I know he thinks his bounty is fake, but still. He knows in his heart. He's like, this, this, this is not fake, bro. How to be real if it was delivered by a bird? Like an official one. He's not planning on it. It's such a cool concept though. I think Luffy is going to put him in his place very quickly. Because he doesn't even know what Luffy does yet. He doesn't know any stretches. And as soon as he realizes Luffy has an ounce of the power that he has, I think he's going to get terrified. This one guy with the poster is still terrified.  Like just the collapse of an ego. God damn, he's fast though. Luffy is quiet. You best be scared. Oh, he actually got a good hit. He actually got a good hit in. That's not even going to hurt Luffy like falling and getting hit that hard. He just needs to throw one punch. He doesn't even throw in a punch yet. Maybe he wants to hide his hand. He's doing it on purpose too. Oh, he's perfectly fine. He's survived much more. I feel like Bellamy is also holding back though. Like maybe his arms can turn into it. That's so cool, dude. Oh my God. He's so fast. Oh my God. Natural born morons. Oh, he is actually making a real man. Imagine it takes one single hit. Just snipes him. Oh my God. You're fine. Bro just killed him. Like actually. You're strong, right? Why don't you fight? He's realizing that it's a real bounty and why it's real. Bro is trembling in his boots. Oh no, no, no. Anybody else wanted to fight Luffy? My God. It's almost like Bellamy hurt himself because he sprung towards him with such speed that he almost just ran into Luffy's fists in a sort of way. He just had to punch down. Oh, look at them at work. Are they fine? They feel immortal because they have a mission at hand. A dream of sorts. Luffy's kind of taught Zora to put his own ego aside. They were like flies to them. I just want Luffy to come walking back, bro. Oh, there he goes. Why didn't have to lay a finger on anyone else? Yup, just Bellamy. You have to be joking. You have to give him props. Where am I going? Yeah, up there. He's so straightforward, but so intimidating to them since they don't know who he is. Does he even have to try? We're checking it again? Yes, I can a fanboy. I think he's more intrigued by yeah, Zoro growing stronger. He told him to. That feels like a lifetime ago. Also, Zoro is a lot stronger now. The whole fight with Mr. One, I think was Zoro's biggest spike up until this point. Learning to cut steel. They're gonna take it on a warlord. Yeah, he's like proud of them. And being a warlord, you have sort of a connection to the government, but like it's not like the warlords like to be in cahoots with them or anything. They just, it's more in their benefit. The government doesn't like to be in cahoots with pirates, but it's part of their benefit too. Holy land of Marie Joy. Oh, the holy land was where we flashed to before. Red hair. I have elder stars. Red hair is shanks. Oh, they got a replaced crocodile. Regret powers.  They've called every warlord? We don't know the majority of them. Just god dang mihawk. Oh, no. Bro, Luffy has made way too much of a name for himself, bro. He's five elder stars. Don Quixote. Don Quixote. Wait. These are warlords? Don Quixote? Bartholomew?  Oh my god, we're meeting more warlords. What is happening? It's somebody else. Suzu, great tactician. Doflam? Doflamingo. There is no way that's this man's name. Why he look like that? Why he look like Zoro, dude? Why he look like a flashy Zoro? He can make people do whatever they want? Saint Sengoku and his lamb? Oh my gosh. Scum of the sea.  That's Bartholomew? The big guy? Only two showed up? I'm not super shocked that mihawk wouldn't come. Doesn't seem to care. Sheer boredom. Oh wait. Is that mihawk? I don't like his coat. Yeah, he did show up. Wow. Most unexpected man. Why the lamb eating on the table? Why is Doflamingo on the table? Is that how you say it? Doflamingo? Oh. Is this another warlord? He tap dances.  Yeah, they did. Oh, are you here to apply? Yeah, yeah, yeah. We do have an opening. Left feet? Oh, a peace officer. Oh, sounds like he wasn't a great peace officer. Oh, he's come with a recommendation. Not an application. My god, what is happening dog? Who is this? My god, that guy's a unit. Red-haired pirates. Oh, oh, these are the red-haired pirates. He had 94 mil. That's me. Oh, okay. Was that his application? Oh, the letter. Oh. Oh, it's a white-beard ship. That was a letter from Shang? What is happening? Shang sent a letter to white-beard about ace and black-beard. What is happening?  And then we got Shang? I'm so exhausted. So you report, now he's reporting back to Shang. Shang seems to have a lot more importance than I previously thought, especially being close with Mihawk. Life is more important than your pride. Oh my. What is Shang's mission here? He has like his own goal here. It's like he's trying to align people. Or figure out alliances. But ace is after black-beard. Oh, he's from black-beard's crew. Black-beard's crew is super small, too, if I remember correctly, after he left white-beard. He was like four or five people. Oh, it's the guy, the champion. Oh, it's the guy, the champion. Oh my god.  The Reaper. Dot-Q. What is this episode? Why is it doing this to me?  And why is white-beard five stories tall? I'm gonna be completely honest. I forgot about everything that had to do with Bellamy in the last episode. We just had a white-beard revealed to us. Interesting things going on with Shang's and Mihawk and ace. Like weird lines connecting. I don't understand it. Like I said, Shang's like it with his own sort of plan here. Shang's writing a letter to white-beard. And it makes me think again, if white-beard is considered the world's strongest man, and he never became a warlord, it's almost like I feel like he sees himself and he probably is above warlords. Like he's in his own level, his own league. And my best guess right now, since Shang's reached out to white-beard, is Shang's trying his best to form some sort of alliance with white-beard to take on either the government, the warlords, or to become their own sort of power entity. But at the same time, the whole lines with ace trying to find black-beard, black-beard killing one of white-beard's men and leaving, that kind of confuses everything. Those two confuse everything. But he's literally a giant. Like he's a unit. What a mammoth of a man. This episode delivered everything to me on this beautiful platter. And I just ate all that up. We had the reveal of two more warlords. What were their names? Don Quixote, Do Flamingo, and Bartholomew, Kuma. So Kuma and Do Flamingo. And then a bunch of name drops, especially in terms of like the upper echelon of the government too. I think little tidbits here and there are people that we will need to know in the future.  I think this is an enormous world-building episode trying to set up for these overarching things, these overarching people that we're supposed to be like keep checking in on. And I think this is a good sign that we're going to keep checking in on them, especially since they've all been revealed now. But guys, that has been it for episodes 150 and 151 of our One Piece Watchthrough Absolute Bombshell episode. Do drop us up on the channel if you are new to not miss any of the rest of the journey, and I'll catch you guys in the next one. Peace, peace.


Hi! Learning data science requires practice and today I'll show you how you can use ClickUp AI to generate projects ideas for you. So you open up a new document and click up. Pick right with AI and write down your prompt. I'm going to go with Create 10 interesting coding projects ideas with datasets from Kaggle, provide links and detailed instructions. Keep Generate and ClickUp AI creates them. Now you can use them to practice your coding skills. Write it out and follow for more.


It's going to be very important for you to equally socialize with him though. I'm going to do my best. Because he could fall in love with me real fast. Yeah, well you know I actually am working all day long like on the road doing stuff and you're here with the birds all day long so even Jersey fell in love with you and left me now. So not only true like my bird kind of thing, I mean it's rocky. You can't have rocky. Cody loves you. Cody loves me but he's, you know, it's different. Stretchy. The idea is maybe keep him away from his cage so that way he can actually. As I was thinking like I got to pack, I got to send merchandise, maybe I could bring him up there to work with me. Yeah, see and that's how she takes over the bird, okay. That's how that happens, you know. She takes the birds that are the new birds and puts them up with her and hangs out with them all day long and then she's like oh you need to socialize with them but how am I going to socialize with them if you're already doing it? That's why we have the weekend now. You have a game. That's why we have today. You have a game on the situation. Let's put it that way. I don't have that possibility. I can't compete with you. No, you can't compete. Right buddy? But Rocky will never come to you. Sorry. Rocky's mine. He's mine. Even Jersey. I mean, you know. I can make Rocky mine. I don't think so. I could. No. I could make Rocky mine. No, I don't think so. It's not possible. You have to say Rocky loves me. No, Rocky loves me. I can make him love me. He loves you. But he loves me. Yeah. He's like in love with me and he won't replace me. That's it. Sorry. This is why guys, I don't steal Rocky's love from George. You can't steal it. I can. No, you can't. I started working with Rocky and he started following me everywhere and he followed me up into my office. It was the cutest thing but then I was like, oh no, I can't have Rocky with all the other birds. So. Sure. Okay. Marlin, it's all about you now. What did he say? I don't know. He made it keep that nice. He did? Oh. You want to grab him? Yeah. Oh no. There goes my phone. All over the place. Okay. It's okay. I'll pay for it. Whatever. What are you nod now that you have to go to work and can't hang out with Marlin? Go on, Marlin. Come to your favorite person. No, he didn't. See with me, it's a stable relationship. When you see me, you get me. It's everyone that sees you get you. You don't know where you go with her because with her it means that she has to give all the other birds attention. With me, I only have to give two birds attention. You and Rocky. So what you're saying, you're promising him that it would just be him and Rocky? Just us. Yeah. Just me, you and Rocky. You can hang out and chill. Let me see. She don't know what you're talking about. You're not going to stay until breakfast. Shake it off, kitty baby. Find your bed now.


My name is David Polk. I'm a senior software engineer at Datadog, and I'm passionate about how Cassandra solves our big data problems. But recently, I've been curious about what Datadog can give back to Cassandra. So first, let me share a little bit about myself. I've always loved the water from competitor swimming to scuba diving, and when the water turns to snow, skiing. During a college internship, I learned that aerial images could be used to map underwater forests and clustering algorithms could be applied to quantify their degradation by human threats. Seeing how powerful the right pairing of data set and algorithm could be, I started my career developing machine learning platform at Amazon's Alexa. But one thing I noticed was that the iterations on the AI fundamentally depended on how efficient our platform was at processing the speech data set. So within a couple of years, I found myself fascinated with Cassandra because of how it enabled products to interact with huge data sets while providing high throughput. So how many of you here have experienced an outage and not immediately known the root cause? OK, a decent number of you. So you're not alone. The cause of a problem with Cassandra is not always immediately clear. Cassandra is complex. It's a distributed system that demands proper management of its resources, ranging from client connections across the cluster down to a node's heat memory. Understanding its resource management can be just as tricky. It requires knowing which cluster events are important, where are the logs related to those important events, how to monitor the metrics, and when to alarm in the monitors. There is so much data to process across the cluster, let alone in a single node. When you encounter a problem, you usually need to start from a high level symptom and drill down to the root cause. As Yuki mentioned in his open telemetry talk yesterday, this process can be time consuming, tracking down app teams, trying to uncover relevant query information. Datadog can assemble data from logs and other metrics to provide context that is helpful in minimizing incident response time. So for example, instead of manually gripping through the GC logs, for pause times, you could collect the metrics and extract their contents to Datadog. Or you can enable Datadog integration, such as JMX, for other kinds of information.  Let's say you're new to a team responsible for operating a Cassandra cluster. You don't know much about it yet, but you definitely want to make sure your cluster is healthy since it's on the critical path for your web app. Take a simple graph of request rate for the cluster. You can see that the cluster is servicing around 10,000 requests a second, has a cyclical traffic pattern. What kind of signal would you look for to determine if your cluster is not healthy? Well, here are three scenarios to walk you through what could happen. Let's say you're seeing a few sporadic timeout errors. And then one evening, you get an alert that a timeout's threshold has been breached. You acknowledge the page, get a cup of coffee, and start to troubleshoot. Since timeout errors can have different causes, how do you troubleshoot? Well, timeouts are triggered once latency is too high. So what does latency look like? You see that two days ago, latency was topping out around 60 milliseconds. And around the same time, the timeouts were pretty sparse. But once latency started to creep up above 100 milliseconds, that's when timeouts really started to explode. So why did latency start creeping up to begin with? You decide to look at one of the fundamental resources and node needs for answers. So high latency can occur when the node is strained for CPU resources. So what does CPU utilization look like? High CPU utilization. As the amount of CPU available decreased, the latency increased. So why the higher CPU utilization? A node naturally grows more strained for CPU resources as request traffic grows in the cluster. So what does request volume look like? Just as you suspected, more requests means fewer resources per each request. So let's provide more capacity to handle the extra requests. To provide more CPU headroom, you scale the cluster.  As you can see from the graphs, the new nodes come up on the top graph, the CPU saturation is alleviated, and the timeouts go away. What's your takeaway? CPU and timeouts are not linearly proportional. Once CPU is fully saturated, high latency is turned into timeouts, there are retries, and then you get a snowballing effect. So the rest of the week passes quietly, and you catch up on your sleep. But the next week, you get another alert that a timeout's threshold has been breached. You acknowledge the page, grab your cup of joe, and start again to troubleshoot. At first glance, you notice that the air has looked a little bit more abrupt than last time. You pocket the information. Once again, you start by checking the CPU. You start to roll out the easy problems. Last time you had an issue, it was CPU saturation. But here, the CPU across the cluster looks pretty healthy. Even though you see a slight surge in CPU when the timeouts occurred, the nodes that timed out did not experience a CPU deficit beyond the other nodes. You think back to the abruptness you originally saw. Hmm, what else can be abrupt? Garbage collection. So what is garbage collection? Memory management done by the JVM to remove unused objects from heap and free up memory. There are three types of garbage collection used by G1GC, a collector that we use for our Cassandra 3 clusters. First, there are minor collections. During these, the G1 operator will remove dead objects in young generations and move long surviving objects to old generations. These collections stop the job app briefly. Then second, there are mixed collections. During these, the G1 operator will find live objects in young regions while concurrently finding live objects in old regions, a search called concurrent marking. These can take a long time, but won't stop the Java heap. Third, the feared full collections. These occur because the job app allocates too many objects that can't be reclaimed quickly enough. They stop the Java heap to clear the entire heap, or they stop the Java app to clear the entire heap, both young and old generations. So what about GC? Why would GC be relevant? As you know, the collections can pause the app, in this case Cassandra. G1 tries to keep a pause target for collections. The limit is 200 milliseconds by default. But when a full collection is forced, the JVM must stop the application from running for a short time, during which Cassandra cannot service requests. So what do collection times look like for your cluster? So now that you see your garbage collection stats, you notice that before and after the timeout errors, the longest collection pause times were a lot shorter. But there were some times when the collection pause times went above 200 milliseconds, and as it turns out, these coincide with when you were seeing timeout errors. You don't have any direct signals here, but you're able to infer from the lack of CPU saturation and also the long tail GC latencies, the GC is not running smoothly. The timeouts correlated with the long pause times are enough of a signal for you to undertake GC log analysis. So when you look there, you see some detail about humongous objects. So what are those? Objects that take up more than half of a heap region are deemed humongous. The regions that consume these are automatically marked as old generation. You start to think about how these could affect collections. G1 can't collect a region without loudness information during mixed collections if concurrent marking that's supposed to attend to these old generations was not able to complete by the pause target, then these humongous objects remain in the old generation. And mixed collections require the concurrent mark to finish. So if G1 wasn't able to complete the mixed collections, could some of these old regions with humongous objects have been reclaimed again and again? And then you ask yourself, well, what would happen if many of your cluster's rights were deemed humongous objects? As the number of old regions grows, the number of free regions remaining on the heap would shrink. Pretty soon, there would be no more free regions left to utilize for the live objects in the young generations. And eventually, the compounded allocation of many humongous objects would only be alleviated by a full collection, which would get triggered. To avoid this, mixed collections should ensure that concurrent marking completes. Giving concurrent marking more time to complete would be one way to fix this. But that would really just be a band-aid on the humongous objects problem. Instead, you could find a way to decrease the number of humongous objects, and therefore, the allocation rate into the old generation. So you read Oracle's docs, and you see that you should be picking a heap region size where the ratio between that and the heap region is approximately 248, while also being a power of 2. You see that you have a significantly smaller heap region size confirming why you might have accumulated these humongous objects. So you increase the heap region size, deploy the update, wait for a few more collections to take place, and voila. You alleviated the G1 of instances where objects take over 50% of a region and turn freeing up the heap of accumulated old gen, as you can see in the top graph. Your high, long-tail GC-lanesies that trigger the timeouts have also disappeared, as you can see in the bottom graph. So now, but not least, a story about the graceful degradation of capability.  You tune its heap region size, so now the collection times are much more reasonable, and you're getting better at recognizing CPU overutilization before timeouts start occurring. One month later, you see that CPU is sitting at an acceptable level at most times, but was abnormally high for a few brief brief periods. So if you look here, you see over on the left that during a rule of restart, the load in the cluster was almost flatlining. In the CPU saturation scenario, there was no real mystery as to why the CPU utilization was high. In that scenario, the query rate was also growing. At the same time, there was also deficit of resources, so you follow protocol. First thought is to scale. So to add additional headroom, your two options are to either scale out and add more nodes or scale up the instance type. Vertically scaling is a relatively safe and normal operation, and you think you want to do that because you see that there is also around 5% disk utilization in the cluster, so vertically scaling would be a little less wasteful. So you run the vertical scale. The first node in the updated instance type comes up. It receives stream data from the other nodes. As it joins, it starts to serve CQL requests, gossip settles, all the other nodes view the node as fully up, and oof, surgeon errors. Like I said earlier, this cluster is on the critical path to your web application. During the operation, once the node joined, your web app started to experience an incredibly high rate. You hurry to cancel the workflow and stop the operation, and after a couple minutes, the array subsides. Why then? Why did we start failing queries when the node joined? So what happened? The new node joins and starts servicing requests, and all of a sudden, all the nodes in the cluster have their CPU completely saturated. So how could the CPU get more exhausted after providing the cluster with more CPU resources? Is crystal clear that something interesting is going on here? Well, you see a 4x increase in request rate when you look at the request graph. The cluster is normally sitting around 10 to 15,000 requests for reads per second, but after the node joined, you see a spike up to almost 30k reads per second. So you ask yourself, was this a retry storm? When the new node joined, the backlog of requests, it started to serve. Could quickly overload the node. If those requests were pushed to other nodes, it could have overloaded them as well, leading to a retry storm. But then you look at the retry graphs and see that that wasn't the case. But then you look at this dark green line over here on the bottom graph, which represents a second type of query. They caught your eyes since you started to see upwards of 10k of them. As soon as the incident began. So you ask yourself, what are these other queries? You look at the request type, range slices. But what are range slices? Range slices are cross-partition queries. Here are a couple of examples where one is if there is no partition key in the where clause, another would be if the in operator is used on a column, which is a non-partition key. And also, range slices are typically expensive. And as a result, are a common cause of highly unseeded timeouts. This aligns with what you just saw, a large spike in range slices along with an increased number of timeouts. So you wonder, where are the range slices coming from? You're tracking the queries with the app team's key spaces, and you see that there aren't any range slices there. So you wonder if the queries could be happening in the system level key spaces, which you hadn't been tracking since they are typically uneventful. In this case, you dig up some more fine gray metrics on requests. You end up finding that the driver was making a number of calls to the system tables without specifying a partition. So now you discovered the source of the range slice queries. But why would the driver need to make range slice queries? You look further and find that these range slices coincide with driver refresh periods. As it turns out, cluster level information, such as topology, schema, tokens, and status are held in the system level key spaces. And the client driver queries the system tables for changes to that information. Why? It helps the driver gather information about the current state of the cluster so that it can efficiently route queries.  And when you perform a topology change, all the nodes announced to all the CQL clients, the topology has changed. I'm going to redo all these queries. So you saw that you could get a spike when you do a rolling restart of the client nodes. But what you didn't know is that when you've started a single server, you could see a similar effect. So you took down things when you tried to vertically scale. You're probably thinking, the driver took down the cluster? What? Your company owns a number of Cassandra clusters and you haven't seen this behavior before. So why is it happening to this degree? And why haven't you seen it before? So now you turn to client connections graph. The total number of connections in this cluster is grown to be very large. The top graph here shows the client connections. And what you're seeing is that the number of these connections is grown to over 60,000 per node. So why didn't we see this before? This is a huge number. Over the course of the past six months, the number of connections in the cluster has been steadily rising, increasing from 20,000 connections early in the year to its peak of 60,000 connections. So it's been kind of a slow climb. So what's causing this increasing of connections? Well, like I mentioned earlier, the cluster is accessed by workers and then the number of workers is grown, so hence there are a number of connections to the cluster. So the driver is making these system level queries. When there are schema, status, or topology changes in the cluster, more specifically, range slice queries happen when the client is trying to connect to a host for the first time or reconnect to an existing host. So when the node went down, the cluster was perfectly fine. Each client received the down event and maybe tried to reconnect, but it stopped pretty quickly. However, when the new node joined and the cluster began to service TQL requests, every single client application began immediately performing range slice queries and interrogating the cluster for the most up-to-date information, which effectively VDOS the entire cluster and caused the outage. So in summary, range slice queries were triggered server side, but the number of range slice queries that needed to be sent to an increasingly large number of clients. So how do we solve this? There are two ways to solve this problem. First is you could entirely disable the metadata queries within the cluster. The effect of this would be the loss of token-aware routing. Cassandra knows which nodes are responsible for which token ranges, and so knows where to forward requests. In an application that values low latency queries, transmitting the information to the client is very valuable because it reduces the number of intranode requests to serve a single query. So maybe we don't want to disable it, but if we don't want to disable it, we need to try and control it. So there are a couple of refresh window settings, which defaults to zero, and that results in the immediate request of information on status changes. By increasing that refresh window from zero to, let's say, five minutes, each client will randomly be assigned some sleep period within that five minute window, and it'll wait for that amount of time before interrogating the cluster for updated information. So after applying a client side change to this five minute refresh window, you can see here that this green, this dark green line, which represents the range slice queries, has spread out because the number of client requests has spread out over that five minute window.  So what are our takeaways here with these three stories? Well, the first takeaway is the importance of practicality for discovering metrics. Over the past three scenarios, you saw how you could dive into a particular problem, but it extrapolates more broadly. You saw a few request graphs across the three scenarios. The left column graphs here is a slightly broader selection of request information, which is still only a subset of the request detail that we monitor. And then on the right side, you have another lower level signal that we use, disk usage. From these, we can answer important questions. Is a problem isolated to a single node, a replica set, a rack, a data center, or is a cluster wide? Is a recent trend problematic? Or is it only the peak of a cycle, such as compaction? If you were looking at the graph on disk, you might find that. And so many more. Having a breadth of information in your fingertips allows you to eliminate possibilities and quickly narrow the focus of what you have to investigate more deeply. And that allows you to move more quickly in a time-sensitive situation. OK, and here's a second key takeaway. When you have a lot of observability, it shows you a lot of interesting places that you can look deeper. Datadog provides you with the convenience of automatically strapping metrics. This enables you to navigate between correlated events, whether within a trace or across patterns that coincide. It also gives you more of a granular look at events. Otherwise, you might be limited to surface level details. SSH-ing into a node and browsing through current JMX request counts or a few GC log files would not necessarily give you the full picture of what's under the hood. So up here in the top graph, you could see how you could take some errors and then automatically find the correlated metrics, the containers which are associated with those errors, and then let's say you pick a container. Well, then from there you can see that container within the entire span of the containers or the nodes in your cluster. Looking at all sorts of tag details on them. So my second takeaway for you is the importance of navigability. Ease of navigation gives you the ability to grow your understanding of the relevant metrics of your cluster from a small subset to a broader one, as well as how to monitor those for outliers. And that's all. Does anyone have any questions? OK, I think I see two. Would you like to go first? Or either way? It's been a while I worked with Datadoc, but I remember you guys are very good with helpful widgets and everything. I'm wondering if you have any kind of tooling or what's not to monitor client side Cassandra matrix versus server side. Because there are scenarios where you can see a lot of timeouts and application side, but server looks healthy. That would be due to some conflicts unless you look at the throttling matrix. You wouldn't know that. So if you can tell more about it, that would be great. Sure, yeah, absolutely. And that's a good question. It's something that over the time of us running clusters, we've gotten better at. We didn't start there, but it's definitely an important piece of observability to have of client side and server side. And so where we've come from is a place where we've had silos and two different application teams and then us having our own dashboards. And we started to kind of bridge that difference by sharing dashboards around and kind of this. But then on top of this, there are traces, right? So APM traces. And you can start from a query on the client side. And then you can drill into how does that query break down? What are the different time? How long do the parts of that query take? And we're not quite there yet where we have a complete link from the client side to the server side for, let's say, a particular query we can drill down. But that's definitely the direction that we're looking at. And we're interested in going, so you can figure out if some query you thought should not take that long is actually maybe spanning multiple partitions or something like this. Let me get to it. You mentioned that you have all these correlated metrics. How do you scale that? Because there could be thousands of metrics. And correlation is computational, fairly expensive. So you have all these correlated metrics. And then you know which ones are most suspect. And then you can start looking at that. That may give you clues of what's actually the root cause. Whereas in your example, when you went through that, you said the engineer noticed this dark green spot himself. He didn't really use your correlation. Right. Can you talk a little bit about that? Yeah, sure. So I think this is kind of like where we were at versus where we're going. I picked some of these stories because they were interesting at the time for us. But then I guess looking back is 2020 hindsight. So as time has gone on, we've come to recognize all the tools which are readily available to us. And we follow this process called dog fooding to try and make our own observability better by what's already available and what's becoming available. So correlated metrics is a perfect example of that. You might see some trends that follow the same kind of shape as the signals that you're seeing when it's like an error. But a starting point where we were at and we're kind of building from is just having a large dashboard of all these metrics which you are going to think, OK, well, in general, as you grow, you're understanding the cluster. This is what we tend to see is like this is interesting and also this is interesting. So it's kind of more of a manual process at first. But then as time has gone on, we've recognized and started to take advantage of some of these more automated tools that as you're absolutely correct are important for making this robust. Oh, how did a note of focus on this specific? Yeah, so for example, you could look at this error, right? And you could see this is like a server side on the server side dashboard, the clusters dashboard. This is some sort of timeout error. And then from there, you have the ability to navigate towards other kind of information with the same tags. So maybe like the containers which are associated with that error event. And then from there, you can move on over to there and say, OK, well, this is the container. And this is kind of where the manual step comes in, right? I mean, you could then look at that container from the perspective of the cluster that's in and see, OK, is this error something which is limited? Is it something that it has to do with a resource limitation or something like this? And maybe a helpful visual way of doing that would be to look at the entire scope of the cluster and to see, OK, this is the node, this is the error. But in this particular case, this is kind of more just handcrafted together. So these are two disparate instances as opposed to one of the earlier scenarios where those actually were connected through step by step. Textual data? Yeah, well, I think with any sort of log or error or anything like this, you're going to be tagging it with all sorts of tags that kind of say that this event happened on this service or on this machine or during this event maybe if you wanted to tag it in a certain way. Right, exactly. And the labels kind of you can think of can support that correlation. They can kind of lift it up on stilts and say, OK, maybe there's some sort of trend. Maybe there's a trend that happened completely across the company in a different application. But just because the shape of it doesn't necessarily mean that it's caused by the same problem or something like this. So what you can use is tags to be like, OK, well, not only are these the same shape, but also these are closely related services. This is the client. This service is the target, and this is the client. And so you can leverage that information to kind of tighten the confidence in that correlation. Yeah, yeah, so I mean, this is all kind of in its own separate group within the company. But yes, there's all sorts of things related to ML forecasting of trends. Is this more of a seasonal or cyclical trend, or is this a permanent direction that we're going in? And then in this case, though, yeah, like figuring out the correlations of events statistically based. Yes, so in fact, if I were to, and I didn't have this on the slide, but if I were to click on view the correlated events, you would see a dropdown of a bunch of other events that maybe are errors on different services or something like this that occur around the same time, or maybe they have the same shape, or maybe they have a lot of the same tags. But these are the things. And then of course, they're ranked based off of some sort of similarity heuristic. Uh, huh? Maybe after. One more? So in Consentral, is there any metrics that we can monitor or see the network issues? Yeah, absolutely. So that's another panel that we have on our dashboards that I didn't show there. But absolutely, if you have some sort of network partition that is preventing some of your nodes from connecting with each other, that would show up on some of our dashboard graphs that are related to ingress, egress, all sorts of network data. Oh, OK, yeah. So yeah, everything comes through ultimately the Datadog agent, which is configured to just basically streamline metrics across different feeds, JMX being one of them. So it might listen in on JMX, but then some other things as well. Yeah, so these run as a container alongside, yeah. Well, I think I'm just about out of time. But thank you for attending. If you want to chat more, feel free to reach out to me. Yeah.


The American Egg Board is a promotional marketing board appointed by the US government, whose mission is to increase demand for egg and egg products on behalf of US egg producers. Now if an individual egg company wants to run an ad campaign, they can say whatever they want. But if an egg corporation wants to dip into the $10 million the American Egg Board sets aside for advertising, because the board is overseen by the federal government, corporations are not allowed to lie with those funds. What a concept. Which leads to quite revealing exchanges between egg corporations that want to use that money, and USDA and what egg companies can and cannot say about eggs. Thanks to the Freedom of Information Act, I was able to get my hands on some of those emails. Of course, there's a lot of what I got looked like this. Please note a number of items about our Salmonella Crisis Module. Any questions? Or even better, entire sheets of paper literally just said this. That was the whole sheet of paper. Our tax dollars hard at work. But check this out. This is some egg company trying to put out a brochure on healthy snacking for kids. But because of existing laws against falsely misleading advertising, the head of USDA's Poultry Research and Promotion Programs reminds them that you can't couch eggs or egg products as being healthy or nutritious. See, the words nutritious and healthy carry certain connotations. You know that food is actually good for you. But because eggs have the amount of cholesterol they do, plus all the saturated fat, the words healthy and nutritious are problematic when it comes to eggs. This is the USDA saying this. However, the USDA helpfully suggests you can say eggs are nutrient-dense. Wait a second. Why can you say eggs are nutrient-dense but not nutritious? Because there's no legal definition of nutrient-dense. You can say Twinkies and Coca-Cola are nutrient-dense, but legally you can't see something as nutritious unless it's actually nutritious. So, for example, the egg industry wanted to run this ad calling eggs a nutritional powerhouse that aids in weight loss. The USDA had to remind the industry you can't portray eggs as a diet food because of the fat and cholesterol content. In fact, they've nearly twice the calories of anything that can be called low-calorie. Nutritional powerhouse can't be used either. Fine, the industry said, they'll move to Plan B and headline the ad, exceptional nutrition.  So, the ad ended up fine true satisfaction, and instead of weight loss, they had to go with a diet that can reduce hunger. USDA congratulated them on their cleverness. Yes, a food that when eaten can reduce hunger. What a concept. You can't even call eggs a food relatively low in calories. You can't say eggs are low in saturated fat. They're not. You can't say they're relatively low in fat. You can't even call them a rich source of protein because they're not. It's illegal to advertise that eggs pack a nutritional wallop. Can't truthfully say that. Or have a high nutritional content. You can't say eggs are nutritious at all. Can't say nutritious. Cannot say eggs are nutritious. Sometimes you have to tell the industry a few times. Eggs have so much cholesterol, you can't even say they contribute nutritionally. Can't say eggs are healthful. Certainly can't say they're healthy. And how much cholesterol there are in those things? Can't say healthy. Can't even say eggs contribute healthful components. Since you can't say eggs are a healthy start to the day, the USDA suggests a satisfying start. Can't say, can't call eggs a healthful ingredient, but you can call eggs a recognizable ingredient. Can't truthfully say eggs are good for you.  The egg industry needs to steer clear of words like healthy or nutritious. For food to be labeled healthy under FDA rules, it has to be low in saturated fat. Eggs fail that test. And less than 90 milligrams of cholesterol per serving. Even half an egg fails that criteria. For the same reason you can't tout an ice cream for healthy bones, you can't say eggs are healthy because they exceed the limit for cholesterol. Egg corporations aren't even allowed to say things like eggs are an important part of a well-balanced healthy diet on an egg carton because it would be considered misleading, according to the USDA's National Egg Supervisor, since eggs contain significant amounts of fat and cholesterol, and therefore contribute to the leading kill in the United States heart disease. The industry can't afford to tell the truth about eggs or even the hens that lay them. The industry crams 5 to 10 birds in cages the size of a file cabinet their whole lives. But when providing footage to the media, the American Egg Board instructs, do not show multiple birds in cages. They look too crowded and open us up to activist criticism. In other words, do not show the truth. Not only is the industry barred from saying eggs are healthy, but they can't even refer to eggs as safe. All references to safety must be removed because more than 100,000 Americans or salmonella poisoned every year from eggs. The Egg Board response to this eggborne epidemic is that salmonella is a naturally occurring bacteria. The egg industry didn't think that should necessarily be the key message faring it may be counterproductive and no avoiding salmonella in eggs, aside from avoiding eggs altogether. That's why the American Egg Board can't even mention anything but eggs cooked hard and dry. No soft-boiled, no over-easy, no sunny-side-up because of salmonella. The American Egg Board's own research showed that the sunny-side-up cooking method should be considered unsafe. And because of avian influenza as well, not just salmonella, in light of bird flu viruses, eggs must be cooked firm. The VP of Marketing for the Egg Board complained to the USDA, saying they'd really like to not have to dictate that eggs, that yolks, are firm. What about some Washington Post article saying running yolks may be safe for everyone except pregnant women, infants, elderly, and those with chronic disease? Turns out that was a misquote. They can't be considered safe for anyone. Instead of safe, you can call eggs fresh. The USDA Marketing Service helpfully suggests. But you can't call eggs safe. You cannot say eggs are safe to eat. You can't say they're safe. You can't even mention safety. You can't say they're helpful. All references to healthfulness must be deleted as well. Wait a second. Eggs can't really be called healthy? Eggs can't even really be called safe? Says who? Says the United States Department of Agriculture.


I want to show you the footage of what I did the other day. So the other day I went to Podcast thing interview with another student who also does actual podcasty stuff and website stuff And I'm really cool stuff in the description down below Attractive what do we think that's taken me literally 20 minutes to do and I'm Hey guys, what's up welcome to my vlog I've got half my makeup on half no makeup and today I'm vlogging because I am going to Meet up with someone who found me through my channel who wants to do an interview talkie chap thing Because there are her student in said this like the center of London I city of London, uni not saying you need I go to but they Doing like podcast sort of thing And I got asked if I would have a chat and I said yes, and I'm super excited and decided to vlog today so let's get on with the day and That's how some fun Let's be who we want to be and I'm watching Disney Channel films because I've run out of things to put on Netflix I tried to match it. I think I did okay Like if they start smoking every time you straighten your hair They're still smoking If your hair, but if your hair is going to smoke every time you try and use them Get new ones. There's advice. Oh god. Oh god. I've been filming videos. I'm learning I've only just realized that my clone is unplugged. Oh gosh. We're gonna have crap audio for a few We go Yeah, no because like this is the thing, right as a journalist like it's so difficult to get interviews Like I literally have to message people on like every single platform to get like one response like I got literally everything. I know you're like Italy. I was like, oh, I love that. I answer pretty much like It's because I can't normally hi everyone. Hello. What's up? Welcome. We're chilling. Hi. So, um, I'm Romy I'm a journalism student at City University And yeah, I'm doing a little interview with Lydia today about her journey and her experiences with the mental health journey You can follow me on Instagram at this is Romy and I also have a website which is at thisisromy.net I'll link them all in the description. So Lydia When did your mental health like issues or problems start like when did things first get difficult for you? Um, six, seven years ago Like I was living with my mom And it was me and my mum who had kept just Up and down constantly and then we on that it was like lots of trauma then Things just crashed and I went in-patrick without really knowing what was going on Everything that's got like a chance of working is always worth the trouble So what do you think the most important thing is For other people and for yourself when you know, you're struggling and you're on a road to recovery What's like the most important thing for you? Reach out to people. Always. Even if you get a little bit on it, it's always worth like just phoning like a friend, phoning Helpline, phoning a charity. Charities are okay Stopped you. Yeah Or just like even like going to see the GP the day after or finding a way of releasing what you're thinking like You know talking to the voice know it's up what you think you can learn just Helps Going forward, you know, what's what are like your hopes and aspirations and dreams? I know you're studying film at university I don't really know at the moment. I want to get more into documentary production like pressioning But I just go with it now. I'm not trying to force myself to do things anymore. I'm just going with that Like doing the film degrees one I get to learn new stuff and two I'm just interested in like all the other kinds of film Why I don't want to do like like fiction films or anything like I can't I can't look like a sci-fi genre Like script or anything. I'm terrible at it. I could do like documentary stuff But I just got to have an experiment with it so it's a bit of fun and What's like the way that you would hope that things would change in our mental health system, you know going forward What needs to happen? I don't know. I would say they need to employ more people. They do try and employ more people. I think They just need to offer more lines of communication people Even it's just sending the letter saying how far like how long they gotta wait for or an estimation of higher Or it's like going once like a month. Oh, how are you doing? I'm not just missing people like Say, oh, are you just a borderline? Yeah Except for that before as well. Oh, you're just a borderline Which that they've noticed is so stigmatized, so it's like I don't just need to be less different I think Or more educated on the diagnosis rather than being like, oh well F1L says your attention see because I don't want to go with that Because I've had people say that as well that question all So it's just I think I just need more support of the people and keep them like humans rather than Hit marks And you know given the fact that you do have YouTube now I feel like you're more part of the community.  You know, do you feel like you're better now than you were? Definitely got better than I was definitely when I started YouTube originally. I was really depressed And then I got even worse. It's showing us in the video to one But like you can see that the progression through like when I was impatient to man and it's Like I watched up the I filmed a little video while I was impatient like the whole experience I had and then when I came I was like, oh, do I post this I was like I had it all like the little bits of that come online and be okay And I put that out and then like now I look back at it And I'm just like, how did I get to where I am now? For men, I don't know. I love it. I literally loved doing it. Amazing. How is it? And the turn my camera off We've nearly filmed for a whole 29 minutes. Wow