Hello everybody and welcome back to the regular expression series in Python. In this lesson, we're going to be taking a look at regex use cases. Now, like I said in a previous lesson, sometimes it can be confusing on how you're going to actually use regular expression and what you would actually write. But we're going to take a look at several real use cases that I've used it for in my job before and we'll see how we can do that. As you can see right here, we have this random text and the text says my name is Mr. Neo. My phone number is this phone number. My email is the chosen one at gmail.com. We have that for Neo, Morpheus and Trinity. And what we're going to do is we're going to use regular expression to kind of pinpoint some information that we want to pull out. Let's go ahead and run this and we're going to come right down here. And let's imagine the first thing that we want to do is specify the type of email accounts that these people have. Now, you can imagine there's thousands of these or millions of these, whatever you'd like to do. But our company wants to know are people using Gmail, Yahoo, Apple, AOL, etc. What are they using? And then we can run counts on these things, right? That would be good information to know. So let's go ahead and try to extract this information. Really, what we can first start to look at is just this right here. Yahoo, the Apple and the Gmail. We'll start really simple. Do re, find all. And of course, we're looking at the random underscore text. Now, the pattern that we're going to be looking for is we most likely are going to want to look for an at sign. You'll notice that nowhere else in here, at least in this example, do we have an at sign? So we should come right here and we should look for an at sign. Let's go ahead and take that. Then what we want after it is this text right here. So we're going to say we want lower case or uppercase letters. Sometimes, you know, who knows, it could be uppercase. Usually it's not them. But what we're going to do is a bracket, say a dash C. Now, we know it's all lowercase, so we'll just keep it that. But for example, if you wanted it uppercase, we can also do this. We'll just do a dash Z. And then at the very end, we'll say a plus. Now, why do we need the plus? We need to make sure that we're taking all of it or we're looking for all of it, not just one of them. And let's go ahead and run and just see what we get. So now we have at Gmail, at Yahoo and at Apple. Let's take away that plus really quickly. Now we just get at G, Y and A, which of course we want all the characters after that. Now we're taking all the A through Z. So when we run this, we aren't getting the dot com. And we'll also get that in just a little bit. But let's say we don't want these at signs and I probably wouldn't. I just want the string of the Yahoo and Apple and Gmail. We can do that by using something, a group called a group. We'll use parentheses for this. If we put parentheses around this, we're looking for this pattern, but the only thing that's going to be in our output is what's within our group right here. Let's go ahead and run this. So now we're specifying we only want what's within these parentheses in our output. So now we just have Gmail, Yahoo and Apple. Now if we wanted to get that dot com at the end, and we don't have to, but, you know, while we're here, why not? So we have this A through Z.  We're going to restart. So we have the at. Now we have to think through how we're going to specify that we want the Gmail.com. Well, we definitely should specify that there's a period right here. We should definitely need to be using that. So in somewhere in here, we'll use a backslash period to specify that there is a period that we're looking for. And we haven't used any character classes really yet. So let's toss one of those in there. And we could use backslash w. That stands for non-alpha numeric. So all of these letters right here would work perfect. So now what we can say is we can put this one in the bracket and we'll say, we're looking for a non-alpha numeric and then a period. And we need to do a plus sign again. Let's try running this. And now again, we get at gmail.com. And if we want to, again, we could wrap it in the parentheses and we'll have gmail.com, Yahoo.com and Apple.com. Again, we don't need the .coms. I was mostly just for just to see if we could. Now let's say we actually want the entire email, which is what I will say. Personally, this is the one that I've used the most. I need to pull out an actual email address. And this is a perfect start. We'll come right down here. We've already got the yahoo.com. And let's take away these right here because when we run this, we want the at gmail.com. We don't want the at not to be in there. So all we need to specify is that there's text before it. It could be a letter. It could be a word. It could be anything. And we should be able to get that using the backslash w as well. So we can almost copy this in a way to take this. Let's put this right here. We don't need this, of course, but we will need a plus because we're looking for more than just one. Then we're saying plus this right here. So let's go ahead and let's try this. And why did I put it right there? I'm going to put it down here. Do you get rid of that? There we go. And this at sign needs to be over here because we're looking for these letters. Then we're saying and the at plus all these so that at gmail.com that we're looking at before. Now let's go ahead and we'll run this. And now we get the actual full email addresses. Now, just to see if this is working, I'm going to put a one right here. Let's run this. And if we run this again, it still works because there are some people who put numbers.  If we did that, it wouldn't work because now it's not going to take that one. And that's what we don't want. So that's why we did it the way we did it instead of using the a dash, the a dash z. We wanted to make sure we encompassed different things that we may not have seen or thought about, you know, when someone was creating these email addresses like a number one in their email address. So we figured out how to get email addresses, which again, I do that. I've done that a lot. The next one we'll take a look at is this one right here, which is phone numbers. These are probably the most, the two that I've used it the most for. I've used it for a lot of other things, but these are the most common, which is why I'm using them as examples. It's coming right down here. We can see both go and let's copy this right down here. Get rid of all of this junk. Now, what we need to do is I want to pull in this entire phone number. I want to make sure that I bring in this entire phone number because we're creating a call list for one of our clients.  We need their phone numbers, but we have it in this, you know, in the list or whatever, um, we need to pull that information out. So how can we do that? So we'll start with something really simple, which is the backslash d. We'll run that. We get all of our numbers, which is great, but it doesn't really look that good, right? Looking at all of our numbers, all of our numbers, all of our numbers. Um, we don't want to like this. We need it. I would actually looks up here. Now, how can we do that? Well, we can start with looking at a pattern of three numbers. So now this already looks much better. We have one, two, three, four, five, six, seven, eight, nine, zero. So we don't have that last zero and we want to kind of combine these as well. So there's some other things that we need to do to make this really work. What we should do next to specify the next character. The next character is a dash. And then we can literally take this exact same thing. So we're looking for three numbers dash, three numbers, three numbers dash, three numbers, and then we're doing a dash instead of three numbers. We're looking for four numbers. Let's do a dash and now we'll change this to a four. Now we should be able to pull in this entire thing into one value. And that's really what we want. Go ahead and run this. So now we're getting the entire phone number. Now again, in the real world, it's much messier than this. Oftentimes you'll need to clean the data first, get it into the exact format that you want it, let's say this format, and then you'll run something like this. So this is already kind of pre-cleaned or it's pretty for us. So this is how we're able to get both email addresses and phone numbers. That's really all we're trying to look at. But I do want to talk a little bit more about how the data is actually stored. Typically the data is not coming in like this, one giant string. More often than not, it's in something like a list. That's the one that you'll see the most. So let's take something like this and let me copy that again. I don't think I actually got that actually copy from this side. It's not working the other side. I don't know why. So let's create a list. We'll say my underscore list and we'll say this right here. We'll save this. And let's say we want to run some of these other regular expression things that we were running before, except we want to run it on a list. We can definitely do that. It's not crazy difficult to do, but it is going to look a little bit different, of course. And I will say for email in my underscore list. And then we'll do the regular expression. So we can come back up and copy something like this. But now we have to do it on the iterable or the variable or the temp variable that we're looking at and we'll go ahead and run this. If we run this, not going to get an output, of course. We have to do a print from over here. Let's say print and now we get what we were looking at up here, which is gmail.com, yahoo.com and apple.com. Now, this isn't probably how I would actually write it, right?  You're applying your regular expression to it. But now we're getting all these values in separate lists, which genuinely makes no sense. So and that's because reasons find all. So what we'd probably do is use something like list comprehension. So I could say a domain underscore list is equal to and then I'll create a bracket, which will be our list. And now we're going to apply this in the same way that we did the for loop. And now we're going to say for email in my underscore list. And we'll come right down here and we'll say, we'll print out our domain underscore list. Let's try this. It's not going to work, but let's try it. The issue is, is we're creating a list within several lists. And that's because we're using find all. But because this is a list, we can also use indexing on this. So all we have to do is come right here and we'll put a zero. So when we run this now, you'll notice that it's all within one list. Now, you can apply this same logic, maybe not with the list comprehension, but you can apply the same logic to something like a dictionary or a set or a tuple or really any of those things. And you can apply regular expression to it. Again, typically, you're not going to find data that's just sitting here in this open string, unless you're working with like a large quote or a book or, you know, some website or something like that. Typically, you're going to break it up into you'll have it in list. It'll be broken up into its own individual values. So those are some real use cases that I've used regular expression for many, many, many times in my real job. I wanted to show you kind of walk you through how that actually looks and works. You're not going to probably apply this exact regular expression pattern in your work, but you can adapt it. You just identify what are you looking for in your string? Identify what's there, how you need to write it to specify that pattern. And then you can return those exact values. I really hope this was helpful. Regular expression was really difficult for me when I was first starting out. It was like just crazy complex, but I hope I've kind of walked through it in a way that you can understand it and start using it because genuinely regular expression is really, really powerful and something that I think a lot more people should be utilizing. So with that being said, that is the end of our regular expression series. Thank you guys so much for watching. If you like this series, be sure to like and subscribe below and I'll see you in the next video.


Here comes the spider mag, so, Agnesites, in movie rumors, on nukethefridge.com. This is what it says, cause y'all know I love rumors.  Now we got that out the way, says Avengers Age of Ultron is set to open in theaters on May 1st. The movie is expected to be one of the biggest box office hits in history, but the following spoiler will grant Marvel fans what they have always wanted. A source has reported to nerdreport.com that additional footage has been filming involving Spider-Man. That's right, Spider-Man is in Age of Ultron. The source also added that a person possibly a stuntman was filming reshoots for Ultron on the Disney lot. Marvel Studios presents Age of Ultron, the epic follow-up to the biggest superhero movie of all time. Oh, fuck! What y'all thinking? Hmm? What y'all think? This is what I was thinking.  We want Spider-Man in this. And that's just how they said it till I'm telling you, in the office they said just like this, we're gonna have to up the ante. We want Spider-Man in this movie. Can you do that for me? Can you do that for me? And they did it. That's what I'm talking about. I can't wait to find out whether it's rumor or truth. And guess what? It don't matter anyway to me because he's still gonna be in self-aware. Am I correct or not? Okay? And if he ain't there, he's gonna be somewhere soon. So our wishes have already been granted anyway. But what the thing is, is just how soon? Correct? That's what I'm talking about. So let me know what y'all think about Spider-Man, possibly being in the age of old trauma.  Even if, say, okay, rumor, right? Say I absolutely 100% believe the rumor.   Man, could y'all imagine that just a big-ass battle? Because you know he's gonna make some type of epic, teenager-y type entrance. You know what I'm saying?  He's gonna swing out of nowhere and kick Ultron in the face or he's gonna web his eyeballs. It's gonna be something.  Ah, man. Post your comments down below, back to the sites. And let me know what you think. Can't wait till this comes out. One million subscribers. We're the 100,000 subscribers practice on the way. I will reach 100,000 subscribers tonight or tomorrow. Guaranteed, there are.


Happy Sunday everybody, I'm Joe Jaspy and that is 2021 Tops Gypsy Queen Baseball. Five box, random team break number two. One spot gets you two teams and all cards ship. A lot of fun stuff in Gypsy Queen. We've got all new captain's mini autographs and the return of the rare autograph lineup cards. Alright so a lot a lot of fun stuff here. Big thanks to this group for getting in on the action. Appreciate it. One spot gets you two so let's double up and all 30 teams are in. Let's roll it and randomize names and teams 11 times. Five and a six. One, two, three, four, eleven the final time. I did not Jack. I did not read the article in the Wapo about sport. This collector from SoCal used to clean pools and now what does he do? He just only collects? So after 11 we got Nick down to Nick. Nick K down to Nick K. Five and a six, eleven times for the teams. One, two, three, four, five, six, seven, eight, nine, ten and eleven the final time. After 11 times we got the tribe down to the J's. Alright so here's how it shakes out. Nick you have the Indians. Mary with the Padres. Sean with the Rangers. Corey last Bob Moji got my Dodgers. Ben with the Orioles. John with the Rocky. Stephen K with the Giants. Sean with the Angels. Corey with the Red Sox. Ben with the Astros. Leo with the Mariners. Eric with the Cardinals. Harry with the Pirates. Michael with the Braves. Harry with the Diamondbacks. Sean with the Reds. Stephen K with the Brew Crew. John with the Marlins. Michael with the White Sox. Corey with the A's. Sean with the Rays. Mary with the Nationals. Harry with the Mets. Leo with the Yankees. Chris with the Phillies. Eric with the Tigers. Corey with the Twins. Chris with the Cubs. Harry with the Royals and Nick with the Toronto Blue Jays. Let's sort by column B by team. I'm gonna pause the video when we come back we're gonna see if there's any trades then we'll have the break stick around will be right back. Alright welcome back folks. Still three nothing Yankees in the top of the seventh. The Red Sox trying to threaten again. No deals were done. Here on Sunday the 18th here in 2021 Topps Gypsy Queen Baseball. Random team break number two. Big thanks everybody here for getting in on the action. I don't know what popped in the first half but let's see what we have here. Good luck. Two on-card autographs per box. Belanger on the front here. A lot of fun parallels and short prints to chase. I think there are box hoppers here too. Save that for the end of the box. All-card ship and we'll do it. We'll do a little recap at the end. Tanya's 34th home run today. David Fletcher had like a 26 game hitting. 24-26 game hitting streak going. Broken today. 1-0 for. There are some baseball records that I think are just out of reach just because the way the game has changed in terms of like wins that pitchers may have. Just because pitchers, starters don't go as deep into games anymore. There's so much more specialized relief pitching. A lot of those pitching records can get a little weird in the future. There may be some old records that may not be broken. But that hit streak I think is made even more difficult because of that specialized relief pitching too. I don't know if that's ever going to be broken. Let's go Kike. Kike here now is going to be pretty clutch. You've got curveball, curveball, fastball, curveball. You're going another curveball. Fastball. Line dot to center. Yankees are out of that inning. That was your pitch Kike. Box number one. All right, good luck. Was there a booklet in the first half? I'd like to see another book too. So many cards there. Yeah, it's Vlad Jr. I would like to see him with a triple crown as well. And there's our first autograph. There's his teammate, Cavendizio. I'll make the MVP race really interesting. 37 out of 99. I think Trout one year had a better war, for whatever that's worth, wins above replacement, had a better war than Miguel Cabrera. But Miguel Cabrera had a triple crown. And so I like these tarot cards here. They do. But they gave the MVP to Miguel Cabrera because of the triple crown. That'll be interesting to see how it shakes out. Sometimes these can be numbered. There's Yellowich to 250. We've got a different parallel here. That's the 99 Joey Gallo for the Rangers. That'll be for Sean Oh. Blue Jays, by the way, Nick Koba has the Blue Jays, gets that Cavendizio autograph. The Brewer's Yellowich numbered card will go to Stephen K. Cards feel a little bit thicker this year. There's a good stock to it. That might be a font swap parallel. This is why all card ships, there's a lot of, there's a lot of different variations in short prints that you can find in there. And a redemption. So I would consult CardboardConnection.com. I think they have a pretty good rundown of what the different parallels are. There's Max Fried for the Braves to 250. And there's our second autograph. That's for the Rangers. That's Anderson Tejada. Sean O'Leary with the Rangers. Wait, so if that's a redemption, they're three. Got three autographs? Two on-card autographs per box, but there was two autographs and then a redemption. A bonus auto. There's Lindor to 50. Just checking the backs of the cards really quick. Sometimes there can be different backs on the cards as well. There's two crazy there. Lindor, black and white to 50 for the Mets. That's for Harry. There's Max Fried for the Braves. That's for Michael and the Bravos. And behind Eddie Rosario, you are due to receive, congratulations by the way, you are due to receive a autograph indigo parallel of, I think I have a checklist of what is indigo number two? I think 250 maybe or those are base parallels. What about autographs? Indigo autographs, I think 150. Chicago White Sox or Chicago Cubs. It's Andrew Vaughn of the Chicago White Sox. I like that how they print the team name right there too. Not to mark up the redemption. Now to Michael Brinson and the shy socks. Bonus auto here. And here is the three chrome cards in this box topper. Anthony Rizzo, Kieber Ruiz and Christian Pache. All right, next to box. Yeah, so Vlad's at 31 now. Is that enough to win the MVP? Wow, O'Dore. Tattoo's one. He knew it right away. 418 feet, 104 miles per hour exit below. He knew it right away. He hit it. Dropped it back. You know, I kind of thought I thought it was going to be the hawk too for a second there. The Yankees, the Yankees are up 5-0 in the bottom of the 7th. They're doing this, I think, without a number of players who are in the COVID protocol. Second box out of the 5. Good luck. And there's another Rangers autograph. That's Leody Tveris. Sean O'Leary. I like that gypsy queen design. All right, we got Ryan Castellani to 199 for the rocks. That'll be for John and the Rockies. So these are one of the variations. The Bazooka backs. I don't know how short-printed this is, but this goes to the Giants. Mike Yastremski, Stephen Kay with the Giants. It's cool though. Still looking for our second autograph out of the box. Got a Daniel Johnson to 99. And another redemption, and it's, yeah, it's Andrew Vaughn again for the White Sox. So Michael Brinton, I guess you're starting your Andrew Vaughn PC, whether you intended to or not. Ben pulled a Yachty Aussie dual auto booklet out of a blaster of gypsy queen a couple years ago. That's crazy. So that redemption is our second autograph, unless we get a bonus like the first one. So they were just, I don't know if you heard Buster only saying, he's like, he hasn't heard anything definitive, but apparently that, apparently they're just looking, teams are monitoring the Yankee situation, basically. Here are the font swaps right there. Oh, sorry. This Vaughn was a black and white parallel. So the first one was indigo, and the second one was black and white. So at least there are some different ones. It's kind of weird. Yeah, they're not actively looking to trade Aaron Judge, but I guess speculation from other front offices is that they're looking at what the situation is going to be. Basically, if the Yankees COVID situation drags on or gets worse, or if they lose a lot of games in that, during that protocol window, wow, that would be wild. And that's the no name right here for the pirates. That's going to be Harry. Can't imagine they'd actually do that. Would they? The Yankees. All right, box three. All right, do we have any other? Let's go to mlbtraderoomers.com. Oh, forgot about those box toppers. We've got Kershaw, Aaron Nola and Kershaw number, 31 out of 99. All right, all right, all right. Gavin Lux leaves game with hamstring entry. I don't like that. Jazz Chisholm and Garrett Cooper leaving the Marlins game due to injuries. Michael Lorenzen on the 10 day IL. MLB hoping to return to nine inning double headers plus modified with the modified extra innings rules in 2022. I'm not sure if I like that. Cardinals locked up their first round pick, Michael McGreevy. Darren O'Day season likely over after a significant hamstring injury. Angels promote Brandon Marsh. So I guess if you have a position on Brandon Marsh, keep an eye on him. Starly Marte reportedly rejects Marlins' $30 million extension offer. DeGrom, yeah, did you hear that DeGrom went on the IL today? Right, forearm soreness, never good to hear that. That's what Kershaw is working his way back from. Lindor is on the 10 day IL. All right, good for Lance Lynn. He's having good season. Lance Lynn and the White Sox agree to a two year, $38 million extension. Cubs will reportedly try again on Rizzo and Baez extensions before exploring trades. Nationals have interest in Chris Bryant. I think a lot of teams though. I think Chris Bryant's going to be linked to a lot of teams. There was that Cole Hamill's showcase. Wonder where Cole Hamill's will end up. That final basketball mixer filler sold out. Nice. Good. There was rumor that Metz wanted to trade DeGrom and a few others for Bryant. I don't think they'd ever trade DeGrom. Someone just, what is this? Who started that rumor, Rex? Some kid on Twitter? I could see them doing Cindergarde and a few others for Chris Bryant. But like, that sounds like a rumor started by a very optimistic like Cubs fan. Without any, there's any basis in that. I think that'd be wild. Like if it actually came from like sources say, I mean, that would probably be, why would they give a, I mean, they're in a playoff position giving up DeGrom. There is Andy Young, Harry in the Dimebacks, Bleacher Report. Is Bleacher Report, is that just like, what was the context of that rumor? Because that'd be crazy. Is it just someone saying, is someone just speculating or did someone, did this person actually, are they actually hearing something from front offices? That's a huge difference. Because I mean, I could just make up trades too and make up rumors at that point too. I'm looking at the backs of those. Yeah, DeGrom did get hurt again. He's back on, he's back on the IL, which probably furthers, moves down his trade valley. Not that, not that I think the Metz wouldn't trade him, but, well, hi Matthew Selvillefield. Well, Rex is saying, he heard a rumor that the Metz want to trade DeGrom and a few others for Bryant, for Chris Bryant, which does not seem to make any sense at all. So that's, so then I was asking, Rex, like, surely this, this isn't like an actual, this is just pure speculation by like a Cubs fan writing an opinion piece on, on Bleacher Report. That's wishful thinking. Unless, unless that writer has sources close to sources in the front office, you know, unless it's something like that, which would still be shocking. There's the Donis Medina for the Phillies. That'll be for Chris. Right. I mean, at that point, I mean, that person was probably reporting Miguel Cabrera is going to go to, Miggy's going to be traded for DeGrom. And behind McKinstry is Dane Dunning. Yeah. Metz are definitely, I think the Metz should be in buying mode, White Sox, Michael, but I don't know if they're going to be, I'm not sure if they're going to be in selling mode, not, not a DeGrom piece anyway. I could see, I could see Cindergarde though. There's Will Meyers, black and white. Will Meyers for the Padres. That'll be for Mary and the Friars. It sounded good, but it's morning track power. All right. Two more boxes to go. There's Chisholm, McKenzie and Giancarlo Stanton. 99 almost to Plinyard. City of Oakland released a proposed financial plan regarding a development of a potential new waterfront stadium for the A's. Report, Saravani of the San Francisco Chronicles. The A's are unhappy with the terms, which team president Dave Cavill called a step backwards in this discussions. What's going to happen with the A's? The A's have been looking into the possibility of relocation since May with Las Vegas appearing to be the most likely destination for the two Oakland teams, ending up in, ending up in Vegas as a, oh Ben's an A's fan and they're leaving. I mean it's, it's, it's tough. It's a money thing. And it's, I don't know. It's not, it's not, I don't know if a county like, a county like Alameda County can afford to, you know, have like taxes, tax money go to, go to a stadium. It's a good luxury. But then you don't want to lose like a civic institution like the Oakland A's, but it might happen. Trivia question says Rex. Did you find that Bleacher Report story? What are the two oldest ballparks and what years they were built? Yeah, it's got to be Fenway and Wrigley. Fenway must have been built like 1898, 1903, something like that. I feel like Wrigley's around that time too. Aren't, aren't those both like really old ballparks? They were, yeah, they were probably built in 1492. No, Ben won't follow them to Vegas. Your new team will be the Padres. You're not going to go, not going to go Giants regional team. Jackson, are you looking this up? 1912 and 1914. That's what Jackson's saying. I feel like that sounds about right. No, Giants haven't had a bandwagon fence has been. Yeah. Padres will be a fun team to watch. If you're a free agent, nice for the Mets, Ali Sanchez, 51 out of 99. Oh, you saw playing MLB the show. Fenway is 1914. Wrigley is 1916. Here's, that's out of 99 for the Mets. That would be for Harry. It's Dylan Carlson to 50. Looking for the bazooka backs as well. You've got Garrett Crochet to 250 for the, for the White Sox. Braxton Garrett to 150 for the Marlins. That's going to be for John and for the fish. That is Luen Diaz 136 out of 150. That must be the indigo parallel. It looks pretty sharp. That goes to John R and the Miami Marlins. So if the A's leave, they would think they're really go to Vegas. They'll have to, I think Vegas will build them a new stadium. They've got a AAA park out there, but I don't know if that's going to be, uh, Red Sox walked in a run. I don't, I don't know if, uh, I don't know if that AAA stadium can be expanded or even used as a major league stadium, although it might actually be better than the Oakland Coliseum. Yeah, we've got a Mookie Betts to 199 for the Dodgers. It's going to be for Corey. Yeah, that's what I was saying earlier, Jackson. It'd be, it'd be, like the Oakland would have, would have had their, their two big teams go to go to Vegas, right? Matthew's right. If Vegas gets an MLB team, P Rose has got to be into the Hall of Fame that very day. There's Keeber Ruiz, no name, and we've got a Mickey Miguel Cabrera 99. That's right. The Warriors did lead to, but they're still in the area. They're just right across the bay. But, but yeah, but they're technically out of the, the city of, of Oakland. Ooh, that was your pitch. Okay, can I have the Kings? No, the Kings just built a new arena, sparkly new arena, not too long ago, Matthew. Kings aren't going anywhere. They, they, at one, they once thought at one point it seemed like the Kings were going to go, but they sold the team, got new ownership, and they built a new arena. So I don't think the Kings are going anywhere for a long time. Jack's asking you trivia question, which two teams played in the first formal game of North American style football? Oh, is that another run walked in? I say it was the, I say it was the, the, the Green Bay Packers and the Chicago Cardinals. Yeah, no worries, Matthew. I sometimes I forget too, because I was like, oh yeah, the Sacramento Deaf and it's like, oh no, that's right, hint 1874. I mean, the Lake Erie musk rats versus the, you know, the, the, the, the Charleston meat packers. Now that would be the Green Bay, but the Charleston Fox trots. So two different dances. There's Louise Robert to 99. Yeah. The bean eaters. Yeah. Boston bean eaters versus the Brooklyn bridegrooms. It has to be something like random city that you would, you wouldn't think would ever have a professional football team if it wasn't for the faculty that was 1874. The answer is Harvard. I was going to go universities next. I wasn't going to go Harvard though. Harvard. I'm going to go like Yale versus the, the New York Golden Knights or something like that. Here's Josh Hader to 199. No, it's Harvard versus Jack V's Alma Mater and McGill playing in 1874. The first formal game. I guess not professional game. There you go. First formal game of North American style football when they used their fee. Who would have thought that that game, that those college kids were playing back then that 150 years later or whatever, turns out to be billion dollar industry, billions, billions of dollars. A no name front. Is that sunny gray? I think it looks like from the picture. Any more bazooka backs? I like those bazooka back cards. Wonder what the line was on that game. I'll tell you the total was probably, the total was probably seven. And I don't know, maybe uh, maybe Harvard minus one and a half. There is Jonathan Arues, Arauz. Arues? Arauz. I don't know where I got the other thing. Corey with the red socks. What was, what was the score in that game, Jack? Do you have, do you have box score for us at this point? That looks different. That's a 250. All right, noted Brandon, thank you. I know it gets a little weird this time of day. There's always like a two hour window in the day. It's fine Brandon, don't worry. I've been doing this for a while. I don't need any backseat driving. But it does, I do acknowledge that it does get dark at certain points of the day. So something that we deal with when that powerful California sun bursts through that window there. Abreu going to the Yankees. That'll be for Leo. It's fine, it's not like it's pitch black. It's James Karancheck for the tribe. Is that Granke? All right, and that's that. Let's see what's in the box hoppers that Granke's to 150. Could be a Cromotto in this last pack.  There's Moncada, Devers, and Marcus Simeon. Check that too. All right, here's the quick little recap, autographs, some nice parallels, some nice colors, couple Andrew Vaughn's in here as well. There you go, ladies and gentlemen. That was 2021 Topps Gypsy Queen Baseball, five box, random team break number two from Jaspy's casebreaks.com. I'm Joe. I'll see you next time for the next break. Bye-bye.


You cannot stop people promising you the earth and delivering nothing. You can't stop people promising you the world, over promising and always under delivering. Some people do that in your life all the time and it's killing you because they're close to you.


I want to welcome you to the architecture of a geodistributed SQL database. My name is Lindsay Hooper.  I'm here with Jim Walker VP product marketing at Cockroach Labs and Chris Casano Solutions Engineer Cockroach Labs. Let me tell you a little bit about them. Jim is a developer who spent a large part of his career helping people understand complex emerging technologies. He's passionate about distributed systems, data and open source technologies. He's a startup, a holic and loves building apps and new communities, and he has helped build companies like Hortonworks and Core OS. Chris prior to Cockroach was a senior solution engineering manager, Audara, a solutions engineering manager at Hortonworks and a principal sales consultant at Oracle. Welcome Jim and Chris. We're super excited to have you on the line today. One quick logistical note for you. I've put you all on mute to avoid background noise, but we do still encourage you to ask questions. Questions will be answered towards the end of the training, though pressing one should be answered in the moment. You can ask these questions through the chat function depending on what kind of computer you're using. I'm going to hand it off to Jim and Chris, and you guys can take it away. Enjoy. Awesome. Thanks, Lindsay. We are excited to be here with everybody today. I hope everybody in the audience is healthy and staying safe. These are some interesting times, but we're excited to talk to everybody today. I'm excited in particular for this webinar. Actually, we do a fair amount of them at Cockroach Labs. Because I think, you know, the Postgres world and being part of Postgres and that community, you know, I've been definitely involved in building on top of Postgres for a long time. You know, I just, I'm just so amazed at how big and vibrant the community is. And so we're excited to talk to everybody today about, you know, what is a cloud database and what is these, what do these things look like over the next couple of years? How we think about scale, how we think about resilience, how we think about meeting the expectations of, our users, no matter where they're at. And that's globally, how do I, you know, have like a great experience for everybody? And I think that's, you know, one of the challenges around applications is really data and transactions and how we need these kind of, kind of diverse workloads. You know, Mark, Mark Adrian is an analyst at Gartner, who I look up to as somebody who's, somebody who I feel is a pretty, pretty smart, great, great analyst. And he believes that the future of what we're doing is smart, great, great analyst. And he believes that the future of all databases is cloud. And I kind of agree with the guy. So that's kind of quick intro. So why another database? You know, we have relational databases. You're all obviously, you know, familiar with Postgres. You kind of know there's no SQL databases that things like, like Mongo and some of these other things like Cassandra and some of these other databases, you know, with the inception of cloud and with kind of the requirements to scale, the requirements to be resilient. And really kind of with distributed systems, you know, the real is an opening for a different way of thinking about a database and not taking legacy systems and making them fit into distributed systems, but let's just build a database that was really from the ground up architected for distributed systems. And that's, that's what we're doing at Cockroach Labs. And that's what we've been working on for, you know, a fair amount of time, you know, really since the start of this company, 2015. And so we fit in, and if I think about relational workloads, some of these kind of transactional system of record type workloads and applications that we're building as they move to the cloud, having a database that is not only SQL, but will scale and build in this resilience is kind of critical. And really kind of what we focus on. Lots of people find Cockroach Labs because of, you know, they are building out kind of a microservices based platform or application. They're breaking down a monolith into these things. And so if I think about this, you know, like the traditional way of thinking about a database, if I think, you know, back in the days of like Oracle, the way we would get redundant systems and get coverage, like global coverage for databases, you know, we have instances of Oracle, we use Rack, we use Golden Gate, it's Baltly, it's costly, it's difficult to get going, right? I think AWS has done a good job about pushing forward into kind of cloud database. The problem with AWS and Aurora is it's not really a cloud database. It wasn't really built to be a distributed system. I think distributed storage is awesome, but in a true kind of world of distributed systems, what we think of typically is shared nothing. And you know, that's something that we've gone a long way to actually have this kind of more matrix type database where each instance of Cockroach CB is its own single atomic unit, is its own kind of standalone piece that coordinates and takes part in a larger system. We don't live on top of anything that's shared, and I think that's one of the things that allows us to be a truly distributed system. And I'm going to go into how we did that in this webinar today. That's all we want to talk about. Like, what does it mean when I say don't take legacy tech and move it to the cloud and try to make it kind of distributed? What does it mean to actually tear apart a database and think through, like, how do we become a distributed database at the core and how do we build something from the ground up to be so? For me, if you're going to be a distributed SQL database, and I think about this as kind of a new emerging group of databases, well, number one, and I think everybody on this phone would agree, you've got to speak SQL. Like, I'm not going to require y'all to go out and learn a new language. I find it incredibly important that we should be able to use the interfaces that we have today. You know, we're wire compatible with Postgres today. You can use, you know, there's a world of ORMs that we've built if you're into that, if you like to use ORMs. And we're going to work the way you work today. And I think that's actually really critical. Requiring people to learn something new is not realistic. I think, you know, our application speaks SQL today. And I think, you know, gosh, I was learning SQL a long time ago and I think it's still applicable today. You know, a distributed database should ease operational scale. I think we can shard databases. I think a lot of probably a lot of people on this phone today have gone through this whole manual sharding process. It's difficult. There's a lot of operational overhead. Distributed systems should automate scale. And so how do we make that happen in the database? That should be a requirement. These databases should be always on and resilient, right? Another kind of like a core tenant of distributed systems. And if you're going to be a database and you're going to store data, well, you can't lose data, right? That's kind of important. So you got to be asset compliant and allow for transactions to happen. We, for instance, we implement serializable isolation. So this isn't like snapshot isolation and eventual consistency. This is true consistent transactions. Difficult to do a distributed systems. But I'll show you how we do that as well. And then this last requirement for a distributed database is tying data to location. Now, sounds simple, but it's really difficult. Typically as application developers or people who are dealing with the database will actually build a layer at the application layer to basically tie data to location. So all records of this type can only get stored in this set of servers. That's typically left to the application layer. If you're going to do that, you know, we believe that that should be part of the database itself because ultimately if you're going to be a distributed database, you're fighting against the speed of light because latency between data centers is no joke, right? And if you're going to do, you know, if a transaction takes four different phases or 10 phases and it's got to go back and forth between three servers and three regions, you know, latency is add up.  So, you know, tying data to location is actually really important from a performance point of view to give the right experience for the people who are actually accessing that data for the users of your applications.  It allows us to basically tie data to location to also meet, you know, privacy requirements as well. But I think it's a, it's a nuance that this is really about performance and really a critical thing. And so how does the data basically, how does the database automate that? Now, again, CockroachDB, this is a unique architecture. I'm starting at the top level. Each instance of CockroachDB, I could spin up an instance, right today on this, on this, on this laptop that I'm working on. I can point it at a cluster and it's going to participate in the larger cluster. Every node in the database is a consistent gateway or an endpoint, if you will, right, to the entire database. It's actually pretty important, right? So while the data is going to be stored and triplicated across multiple different nodes, and we're going to show you that, like, I can ask any node, I could have a thousand nodes and it's going to know where to find that data, which is kind of cool because I could spin up a node here. I could spin up a node in AWS. I could spin up a node in Azure or GCP. And actually, I'm going to have one database and every node is going to have access to all the data that's going to cross all those nodes.   You know, we actually built a, a location-based cost-based optimizer. So filming with that and how to actually, you know, that two performance around CDO. We built that actually in a database too. It's kind of cool. Now where it gets really cool, CockroachDB is this unique distributed architecture that because all these nodes are their single, you know, atomic unit, they actually coordinate with each other so that we can actually coordinate transactions because we're going to write data in triplicate and I'll show you how we actually coordinate transactions. But, you know, if a transaction comes in at Sydney and a transaction comes in at New York on the same record or the same row in a database who wins, the database actually deals with those conflicts and ensures that you're going to have no overlap, right? And ensures you're going to have serializable isolation for things like that. And so it coordinates for that, but it also coordinates in that there's no more manual sharding. You know, if you want to actually expand the database, just keep throwing nodes at it. The database is going to worry about basically spreading that data out in an intelligent way so that we can actually deal with performance and utilization of servers and whatnot. And then, again, we can attach data to a location because of performance and because of privacy concerns and this sort of stuff. So lots of cool things that we're doing here, but let's get into basically now the architecture of how this actually works and the differences here. And so, you know, lots of databases are databases, but some databases are actually built on other databases and, you know, sounds kind of meta, right? But, you know, we actually built a database on top of a KV store. So if you're familiar with KV stores, a key value store underneath the covers here, we actually use the database called RocksDB and that's between us and the storage layer because every database can be broken down into the language, the execution layer and the storage layer, right? And at the storage layer, we're actually using KV to store data to disk. And it's actually pretty important. It allows us to do some really cool things. We've completely re-architected the execution layer to do all this coordination. And then we've also, you know, we made sure the language is SQL as well. So across these three things. So let's just start with every table. Every table in RocksDB can really be ultimately at the storage layer thought of as this monolithic logical key space. And we order everything in here. Here's a dogs table if you will. And we're just ordering everything by name of dogs. So you can see it's basically in alphabetical order here. And so basically every table ends up getting that looks like this. Now it's a little bit different. A typical database, you just keep adding records to the table. Say it's an inventory table. And in order to find these things, we set up indexes. There are other tables that actually are meta and allow you to do some things. We don't need this because actually we're actually ordering everything based on key, right? So the keys are all ordered in CockroachDB. So it's a little bit different from the typical database structure, which you're going to find in Postgres and some other things. It allows us to do some really, really cool things. Now the sequel, the problem with this is that's KV. How do I deal with tabular data? How do I deal with this relational data so that I can actually run queries? Like that I select star from customer. That's a little bit different. And so the traditional way of looking at a table needs to be translated to the kind of KV layer, right? So we do that. This is part of actually, if you're ever like looking to solve insomnia, right? Like you go read the white paper about Google Cloud Spanner. I think this is one of the really cool things that, you know, Spanner really led the way with and the designers there that really drip this up. Really cool stuff. And this is really what allows for this distributed nature of our database. So the Google Cloud Spanner actually, that white paper that lays out a lot of stuff. So at the KV store, what we're actually doing is we're storing, we're going to translate that stuff all up into SQL. This allows us to do some really cool things. It allows us to efficiently distribute data. It allows us to do dynamic schema changing and allows us to do like this really cool thing around geo-partitioning as well. But I'll show you all that, right? And so let's just think about a table. Here's our dog's table. It has three fields, ID, name, and weight. Typically table entries kind of look like this. Now what we're doing at the KV layers, we're actually translating this when we store the data. Like this is the way it looks on the right hand side. This is how it looks at the execution layer, right? The SQL execution just thinks about it as a table. But at the storage layer, we're doing something kind of cool. We actually each take, we're going to each take each one of these rows and we're going to break it down so that the K, the key, right, is going to be basically an index, right? It's going to be the table name. It's going to be the ID, and then the column that I want to store. So in this case, it's the dog table, ID number 34, and name, and the value is going to be carved, right? And then weight is going to be 10.1. So where I had one entry where I was storing in the traditional way of the storage layer here, I'm actually, I have two entries. Now this allows us to cool things because we can actually change the primary key. You know, we can do some interesting things in this key, like this 34 can actually be a consolidation of multiple different fields so that we want to order things. But ultimately, if you look at this, we can just order these things. And now I know exactly where to put the next record in here into this table. And the KV is what allows us to basically keep this completely ordered pair of key values that that are going to translate back into the SQL table. So this is actually a really kind of cool trick of what we're doing here. I'm not going to go through this example, but just think through how we sort things in, in, say Excel sheets, you know, I could sort out one column and I sort out the next and I sort out a third. We actually can like embed this sorting. So this really kind of cool way of sorting data that allows us to do some really cool things, right? We have this monolithic logical key space. Now we take this key space and what we do is we, we break it up into 64 megabit ranges, a range you could think about as a shard. And instead of us having to kind of manually choose the shards, the database is just always doing this, right? It actually is always thinking about the ranges and we're going to use raft, which is kind of cool concept to actually show how these things get then distributed, right? And so here we have this table, we're going to break it down to 64 megabit ranges. Here we had just three ranges, right? They're small enough so that we can move them around quickly and we could split them. But they're also kind of the right size so that we can actually gain value in terms of how we index them and how we think about them. So if we're going to break a table down to this, we actually need a layer that's kind of allows us to understand where to find things. And we're just going to update this thing every once while as we kind of add more table and more data to this basically monolithic key value storage. This is very much like a B tree, right? If you're familiar with that concept. So order keys, they allow us to do cool things. So we don't have to do these crazy like told table scans. We're doing these very efficient race scans. So if I wanted all dogs between muddy and Stella underneath the covers in the database, it knows it only needs to go to these two ranges. Now these ranges are going to be physically stored at different servers. And so this actually allows us to be very efficient in how we use network and bandwidth and the various different servers that are in the thing. It actually really, really key concept, especially when we're thinking about distributed systems. So what happens when I want to insert a record into a table? I want to insert sunny. That's my friend's dog. And so I'm going to put sunny in there. Sunny's going to go. Let's see in between Stella and C. So I'm going to ask that range. Do you have the updated? You have space. Yes, great. Let's insert that record. We're going to insert that record. It's going to be updated in a couple of different spots. What happens when I want to insert and there's no range. Well underneath the covers, what happens is Cockroach TV actually automates the creation of a new range, right? We've split that range in two and we now have more extra space. So we can actually go ahead and insert that record into that range. Now we're doing this in real time. We actually will allow a range to go a little bit over in real time and then come back and actually fix all this stuff. So we don't affect performance like I don't have to actually split the range while the transaction is happening. I'll just note that, Hey, this range is a little bit full. I better split it. Great. And so this is where we get this whole sharding capability, right? Like cause, and I'll show you how we're writing data to the various different nodes, but think about what this means now from the way we think about like shards and manually doing this process. The database is just doing this, right? And so no more manual sharding. It just basically automates this whole like it's, we just split ranges, right? And so a new range range index was created or it was maintained. I could find that data. If I delete data, it'll actually consolidate ranges together. So it'll actually retract as well. So we'll both expand as it will both expand and retract the database too. And so that's not a really key piece of what we're doing here. And that's underneath covers. I'm a notice that the developer, the database just dealing with all this stuff. And it ensures basically this easy scale and kind of resilience. Now, when we write data into the database itself, like we're going to actually write on physical nodes, like physical locations. Like so I'm going to write, you know, I'm going to have a cluster with, with six nodes in it. Those are six different machines. If you think about, and I'm going to actually write data three times. Like by default, the database is going to write data three times. You're going to be three replicas of each of the ranges. So here we have four ranges. We're going to actually write the first range, Carl, Downey, Figment and Jack. We're going to write it three times across three different physical servers because we want to survive certain values, right? We want to actually distribute this data out in a more comfortable fashion. We use something called raft to do this. And a raft group. A raft group basically takes these replicas, like that, that, that first range is going to write it in three ways. And that, that little group, they communicate with each other and make sure everything's cool. Right. And they're, they're really what ensures that like, there's this atomic replication of command. The atomic replication basically says like, for that group, one thing is happening.  Right. And there's, there's this concept called a leaseholder. You might hear about it. If you know raft, like it's a raft leader, that's the, that's the unit that basically in this, this group of three copies of this range, they actually understand really what's going on. They're like the leader, most things go through them and you're insured, you're going to be right, but they basically kind of coordinate everything for each raft group. So in the implementation of a database for Cockroach DB, Cockroach database, there's thousands and thousands and thousands of these, of these raft groups going on and lots of coordination going on amongst all of them. So, so let me just, I'm going to, I'm going to fast forward a little bit just because of time and get into the stuff that's really important. So that actually, I'm going to have Chris show you guys how this actually works in the database. But when we write data, here's our table, right? Our dog table. We're going to actually write it to physical location. So here we are. Like this is what I was just talking about. The Carl through Jack range that shard is being written three times across three physical nodes, node one, node two and no four. You know, range two is going to be written on node one, node two and node three, range three, node two, node three, node four. Now what we've done here is we've actually distributed the data across physical nodes, the replicas so that we can actually survive certain values. I want to survive value moments like a disk or a rack or a data center or region. We can tune the database. Actually, you can use a command called alter table. And we can actually change this change this in real time in the database. The database will actually change where all this data lives. So you can fail different things, but it really depends on what you want to do when you implement something like Congress database, you got to think about two things. You got to think about latency and resiliency. What do you want to what kind of failure do you want to survive is a key thing. When you're thinking about the table itself and then, you know, what is the latency? Do I need data to live in certain locations? I'm going to show you how we do that as well. So we're going to distribute on this sort of thing. So if you think about this, well, this is kind of to even out storage, right? And utilization of disk basically across the cluster. We basically wrote the same amount of data on every single node. Sometimes a range, however, is accessed more than other parts of the table. And so the database is smarter. Also understand that utilization of each node is actually really important. So I can isolate a range into nodes that have you know, that are that are straightforward for a particular utilization so that that workload is going to have the same performance or optimize performance because I've placed it in an area where there's not a lot of other ranges, right? And so I can basically take heuristics of how that data is being accessed. You know, muddy is a really popular dog. That's my dog. Obviously she's really great. And I'm going to actually take that blue range there and I'm going to put it on nodes without a lot of other data so that I can optimize from a load point of view too. And the database is automatically doing this underneath. So not only are we placing data based on load or you know, by, by, by utilization of storage, but we can do it by load. Now we can also do it by geo partitioning. So the same concept that we had here. Here what I've done is I've actually overloaded the key with a value from the table for location. So, you know, you can imagine that table had an added column for country or whatever that is, or location. Here's the EU US East US West. And so now these shards or these ranges are going to be tied to servers because each server, each node is tagged with the location. And so we can actually geo partition data as well. So not only can we distribute data to survive failure, not only can we distribute data to, to work through utilization issues, but and access and workload, but we can also do it from a geo partitioning point of view. And all of this can be changed on the fly. We can actually change the primary key. We can change this, this key, the K and the KV store. And so that once we change it, the database is going to automatically start to move things around and make it correct for everybody. Right? So kind of cool thing about what we're doing here. And kind of unique and important because it actually sets up some of the key capabilities of what we're doing. So really quickly before the demo, again, we can rebalance replicas.  We just introduced a new node. The database is smart enough to actually just start moving data around to, to, to, to balance things. Right? And so I just, you know, they basically, it decomp, it writes the data, it recopies it in the new node and then actually decomposes the, the previous replica, yet raft still maintains the, the connection between all of those, those, those replicas, each of those shards, if you will. Right? And so that's kind of done. And we just take care of that. We can actually survive a failure as well. Right? So often a node will go down and Chris is going to show you this in a demo. Right? Here we have node three went down and had two replicas on it. Each of those replica sets is smart enough to understand. That's like, wait a second. I'm missing one of my replicas. I've got to have three. Like I can't, I can't live without having three men. Like what am I going to, so it's far enough to actually understand like the leaseholder is going to say, wait a second, I don't have a third replica. Let's create a third replica on a node that's available. And make sure that I have that in the, in the database. So you'll, you'll see something in the demo, like under replicated ranges is what we call that. And that's really when a node fails, you know, the replication factor has gone down a two for a moment until the database can actually recover from that. But all the while, any one of these nodes because each one of them is an endpoint could have taken that query and found the data no matter what, right? So you're never going to lose access to the data, right? It's active, active mode. We don't need active passive in the world of this. We can do distributed backup and restore for those sort of things for those kind of trouble issues, but we don't, it's just, there's no more like active passive system where if it fails and I bring on this, I bring the passive system online and you know, that thing's going to run for a while. And then basically I bring back up the, my original primary and how do I actually consolidate records across these two databases? Now that's tough, man. Like those problems that RPO RTO issues that people typically think through just doesn't happen with us because everything is active. Right. So this really simplifies a lot of things from an operational point of view for developers. Right. It's going to be sequel number one. So that's great. But this whole like this sharding thing goes away. The resilience thing kind of gets solved as well. And so we can also survive a temporary failure. So it isn't as harsh, but I wouldn't. So Chris, do you want to jump in and just show a little bit about the resilience in the, in the scale part of cockroach TV? Absolutely. Can everyone hear me? Okay. I can hear you just fine. And then by the way, everybody, there is the chat window. I am monitoring chat. So if there's any questions, do, do please feel free to do that there. So Chris, I believe you have the floor. Great. Hey, everyone. Good morning. Good afternoon. Good evening. Good to meet you all. I'm going to go through a quick demonstration or really around the fault tolerance of cockroach TV and how you can maintain performance by, you know, being resilient. And you can repeat this demo if you, if you'd like, if you go to our website, you know, docs dot cockroach labs dot com, we have a section over here called tutorials and there's some great tutorials on how you can learn how to use the database. The one I'm going to go through today is around fault tolerance and recovery. What we're essentially going to do is stand up a six node cluster actually already stood up. So we don't have to spend time. You know, you see me set up things in AWS. But we have a cluster set up. We're going to, you know, give it some workload. And then we're going to start removing nodes. And I'm going to show you how the replication factors work. I'm going to show you how the resiliency works while maintaining performance. Okay. So let's go over to the UI. And you're actually going to be seeing some new bits here too. So I'm actually showing this within one of our beta releases. Usually this is a solution engineer. No, no. As far as, you know, demo, demoing things on new software, but I've tested this out a couple of times. It works great. So I wanted to show you some of the new things that are coming within our next release, which is the 20 dot one release, which you should be seeing hopefully towards the end of this month, maybe if not beginning of next month. Okay. So here's the administration UI. You can see I have six nodes that I've created in the database. If I go to the metric section, you can see that I actually already have some load going on here. All right. If I look at the SQL queries, you can see that I have a lot of selects and updates that are happening. And over on the right hand side here, it's giving me an idea of the queries per second that are being performed under the current cluster. There's other things I can look at too. Yeah, Jen, go ahead. Hey, Chris, what's the workload that you're running right now? Oh, yes. This is the YCSB workload. It's the Yahoo benchmark. So we package a bunch of different workloads within the database. The one I'm showing today is the Yahoo benchmark. And if you want to see it running with the database, yeah, with the database binary itself, there's actually a couple of different workloads that people can use to test out the database as well. Right. So that's all in docs. I guess where I'm going. Exactly. I can even show that too. If you can just do a search for workload within our documentation, it will take you right to Cockroach workload. And you can use a variety of workloads here, such as TPCC. A lot of folks are, you know, are definitely familiar with that. There's a whole bunch that you can run there. But anyways, yeah, I'm running TPC. Excuse me. YCSB. You can see we have load there. We can look at the hardware metrics too. You can see that we have a lot of, you know, CPU utilization. The nodes I'm using are actually kind of small, because this is a demo environment. But you can see that we have workload happening. Now what we're going to do is if I go back to the overview, you can see we have six live nodes and we have 51 total ranges. What I'm actually going to do is I'm going to actually remove one of the nodes and we can see what's going to happen with performance. So let me come over here. I'm going to do a little command. Let's say stop. A particular node on my cluster. I'm going to stop node six. And I could have done this more crudely as well.  I'm going to do this more in a gentle fashion. So as you can see, I stopped the node. You can see the node is under suspect here. I have one node that's, you know, the cluster identified as not being available. If we go and look at the metrics, you know, there is a little bit of a dip, but you're going to see this come back up and the performance will maintain. And on top of that as well is that, you know, the, you'll notice that the database is actually very patient. Right. So we have a couple of different statuses here. We have a healthy status. We have suspect and dead. We're actually, because we can't communicate with a node, we don't just eliminate it from the cluster. We'll wait for it to come back. There's a default time of about five minutes where we, where we will mark something from suspect to dead. But if the node does come back up, it will just reapply itself to the cluster. And that's what I'm going to do here. I actually sent the, set this command actually to, to mark a node dead after about a minute and a half. So I want to get it back in there. And what you'll see is that eventually these nodes that, these ranges that look like they're under replicated are actually going to go back to being replicated because I just added that node back into the cluster.  Now right now I'm running this with a replication factor of three. So, you know, I can tolerate a single node failure. If I want to just tolerate more node failures within my six node cluster, I would need to up the replication factor. So that's something that's pretty easy to do. And you can do this even on a, on a, on a live database. So I'm going to show that here where. I'm going to come in and I'm going to change the, I'll show you this command in just one minute. It's basically an alter command that Jim was mentioning, mentioning before actually know before. Yeah, I'll run this and then I'll show you what this, what this zone configuration is. So what I'm going to do is there's a default zone configuration. And you can change this to specify how you want things to replicate what your ranges want to look like. If you want your ranges to be in certain regions. Or the number of replicas that you need that, and have an affinity towards a certain region, you can do that within, within this database.  This is actually going to up the amount of. The replication factor to five within the database. And you'll notice here and in just a few minutes, yeah, you'll see that I'm actually under replicated, right? Because now I only had, I only had three replicas and I need to up that to five. So the database is aware that, hey, this, you know, a configuration change was made. It's going to identify the ranges that are under replicated and start moving them out into the other different nodes. So you'll even see here that like the number of replicas that you're seeing in the nodes are going to go up as well. So we'll just give this another minute or so, so that we get into a healthy state. And then I'll show you how we can eliminate, you know, two nodes and still have the resiliency and performance under, under load. So it makes sense. Jim, any questions coming so far? Uh, no, either, like, we're that good at explaining things or we forward everybody to tears. No, I'm, I'm joking. No, it's, it's pretty scary for us because this is awesome. Yeah, that's good. You can see here, we did have a little bit of a dip, but everything was, uh, went, went back to his ride of rain. Um, you know, during the, uh, during the node loss, and you can see here that the service latency or, you know, the node, um, you know, fell out of service during the, during that time. So let's go back and check. So yeah, we have about, you know, one more range to get cleaned up, and then we're going to do the same thing. We're going to eliminate two nodes and we'll see how the performance looks.  We're in a good shape. While we're here, uh, there's other things that you could do within the UI that are actually really useful for diagnosing things that are happening or monitoring activity. You can look at latency of statements. You can look at the jobs that are happening in the database. Um, you can see there was an import job that was run here to actually see the database with the workload that I'm running. And there's a lot of advanced, um, debug features. There's a really rich rest API that we have so that you can see a lot of things that are happening within the database that all power this, um, that all power this, uh, this UI. Okay. So last part of the demo, let's go just do, um, now that my replication factor is up, I'm going to stop two nodes. Let's stop. We'll stop no six again. And we'll stop node five. And because our replication factor of five is up, you know, we can incur a two node failure. Um, and again, you'll see that we have two, um, two suspect nodes that are here. Um, and the metrics will go back to that as well. You'll see that pick up eventually as well. Um, and what we'll do too is we'll actually have one node go into a dead state and I'll actually bring back one of the nodes so that you can see what's happening here. I know we're getting a little bit tight on time. So just give us just a second. Um, take your time and people, people like to see, people don't like listen to me, dude. I'm just slide guy. This is much better money. Cool. All right. Well, I'm going to bring back node five. Um, just so folks can see this happen. You know, we kind of saw this in the last scenario where we have a suspect node that's going to come back and be attached back to the database. We'll see that suspect nodes go from, yep, from two to one.  And you can see how, um, you know, the database will continue on and continue to perform. Uh, my workload should be going for another couple of minutes. So we should be fine here, but yeah, after about a minute and a half, um, you'll see that this, this node, uh, that's in suspect will actually go to dead. That's pretty much it. I mean, that's, that's all I really have to show from a performance perspective, but, uh, it's just a very patient database in dealing with adversity. Uh, so when you do have failures, dis failures, node failures, and so forth, it takes a pause. It looks at what's going on. And then it, you know, patiently will say, all right, you know, I need to replicate over here. I need to balance myself out so that I can continue to, um, meet the workload that, that, um, that's being asked for me to do. Yeah. So Chris, a couple of things, um, I, you know, I think it's really cool. Like, look at this service latency, the 99th percentile for P 99 latency has actually gone up as you may changes. Another key thing, right? Um, you know, we're constantly looking at P 99 latency. That's one of the things we're looking at. Actually 23 milliseconds is pretty high. We can do lots of things to actually get that even further down. There's lots of tuning. Typically we're seeing those things under five millisecond. Right. And so, um, I think there's a lot of rich stuff in this UI. One of the questions that came through Chris was, um, How do you, you know, how do you run multiple nodes with the web UI? I mean, is there fault tolerance for the web UI? I guess like, how does the architecture of the web UI work? Right. Like somebody asked me, is this part of cockroach DB core? Yes. This is definitely part of the free version of our database. You can download. You can actually be up and running with this. Like I can do it today. Um, but how does the web UI architecture work? Like, where is that running? Is there multiple instances of this? Like, how does that. Yeah. Great question. So since we're shared nothing, uh, set up, you can run that, uh, the admin UI on any of the nodes. So, um, you know, if one of your nodes dies, you can easily just connect to another node and spin up the admin UI. There's nothing you really have to do from a setup perspective to make that happen. Um, it's not like you're, you know, you have to think about my application being on two different, um, you know, two different management nodes, make sure that they have a shared database to have all the metrics and so forth. It's a much easier setup. You just basically say, I want the admin, you know, UI running on this node.  Right. So a node is a node is a node. It's always the same binary. It doesn't matter. And that's like a true, like really important concept in distributed systems. Right. And I think that's, you know, it's kind of one of those things. I think this is actually really important. Now one of the other, one of the questions came in, how compatible is this with, you know, Postgres command line, Chris. Yeah. It's very similar.  So if you're familiar with, um, if you're very familiar with, uh, you know, Postgres, um, I can actually connect to one. Um, here we go. Yep. I'll go cockroach, SQL, uh, all this equals local host. Um, Oh, I forgot my one more flag. And yeah. So if you want to do, you know, show tables or, um, if you need, uh, help on a particular command, um, the syntax is very similar. Um, now we don't support every single, uh, postgres item that's out there. There's a few things like stored procedures and, and triggers that we don't support. But, um, for a lot of those features that customers are looking for, there's ways that we can support triggers through using our changed it capture option. Um, sometimes folks can move their stored procedures into micro services. Uh, so we don't support everything that Postgres supports, but the, you know, the wire postgres wire compatibility is, uh, is pretty spot on. Cool. Um, what happens when two nodes go down and say in a three node cluster, what happens that it is, is stop accepting connections? Is it, how does it gracefully handle that Chris? Yeah. So in the event that two nodes, that's a tough one, right? Yeah. So that, that's, that's actually a tough one. Right. If you have a three node cluster and you have a two node failure, um, what's going to happen is you're actually going to, you're going to lose quorum. You're not going to lose the data, but you are going to lose quorum. Um, and what happens is those Rangers are going to sit out there. Our support team is very good at bringing those databases back up to health. Um, but if you really want to prepare for resiliency, as, as Jim mentioned, um, you're going to want a more, you're going to want more nodes. You might want five, you know, with a replication of five, if you want to make sure you can fail two nodes.  Um, what do you want to survive through?  Um, you know, that's, that's the best way to think about this. Uh, like I said, you won't lose data, but the data won't be, um, accessible because we don't have a way of doing, uh, quorum reads and writes. Exactly. And I think that's actually pretty important. So again, I always think about the, I usually with the database, I'm thinking about the logical model. You kind of got to think about the physical model of like where the data is going to be, what you want to survive. That sort of stuff, right? And I think that's like one of those key questions. That's a little bit different about this than, than a typical database, right? And I think, you know, going through that with, with customers. And we talk about, um, I think on the website, Chris, we call them topology patterns, right? I think that's what we call that in our docs. And I think there's a whole bunch of, yeah. And so we, we walked through that with people, like how to configure the database. There's, I, I've worked at a couple of pretty cool companies. I've never seen a documentation as good as ours. And I'll just say that because I, not because I love my documentation team, but, um, we get a lot of comments. Like there's a lot of really good information about how to, how to configure the database to just survive these things on, in our docs. Um, another question, Chris, and I'm just doing QA right now. You guys, I'll come back and finish up with transactions at the very end. Um, how does band work? How does bandwidth work for synchronization, right?  And so for each replica set, let's just say I'm in a single region. You know, there's data that's being written in the single region. But what happens when it's like cross region?  Is there costs associated with that when you're, you know, let's talk about single region in that multiple region. Um, in terms of those, those sort of network effects. Yeah. So great question. So anything in a single region is pretty straightforward, right? So I'm actually, we have a chart here in, in the admin UI that shows the latency between the nodes. Um, so you can see here, the latency is really low, right? Because these nodes are all close to each other. They're in a single region.  It should be very fast. Once you start getting into geo distributed clusters, you definitely have to, you know, you're constrained by speed of light. Um, there are a lot of things that we built into the database. Um, you know, such as parallel commits and so forth. That allow you to do a single round trip for a particular right transaction. So that's really, you know, and it's a two phase commit. So it's a really, um, fast way that we could do writes, um, even though they're, they're, uh, even though their latency between the regions might be, you know, so much far, you might be going from the, you know, the Eastern part of the US to the Western part of, uh, the EU. And you're bound to there basically by the round trip of speed of light. Um, so yeah. And Chris. And Chris, every node can accept a right. This isn't like Aurora where there's a right node and it's in every region and you're kind of stuck with that one right note. Every single node is an endpoint. Every node can take a read, right? It comes back to like the atomic unit, everything is, it's all, like, like we were talking about people. Like every node is the same. Yeah. That's a great point, Jim. And you can be really particular too, as far as where you want to pin data so that you can make your reads and writes very low latent if they're being utilized in that particular locality. So great point, Jim.  Um, you know, that, that's not something that I would think about to be in the UI for a DB. Um, but it's actually really critical when you start thinking about a distributed database. Uh, and so we, we actually include all that it's kind of table stakes to, to what we're doing here. So, um, I've gone through a fair amount of the questions that we're in the chat. I'm just going to jump back into, um, just to talk about geo distributed application. Um, um, so the last thing I did want to talk through is, you know, Chris talked about being geo distributed and, and how do we actually take care of transactions, right? Cause if we're going to write data in triplicate across the cluster, well, man, like I got to ensure that that stuff is consistent, right? And so we do implement acid. I mean, when we think about the eye in, in acid though, um, we're real heavy on the whole, you know, serializable isolation. And so we've actually implemented that for a distributed system. So, um, we're pretty excited about that. And so I just want to go through kind of how that works as well. Um, really quickly, this is a, like, looking, I'm going to give you the best case. There are a lot of use cases, a lot of like corner cases here, you know, multi-phase transactions, transactions that deal with like 15 tables. I think it's really complicated. Our team has done a really good job of actually re-architecting the execution layer, the database to take care of these things. But let me just show you just, like some really kind of core concepts. So what we actually had to deal with here, let's just do an insert into the SOG stable. I'm going to take two records and I'm going to insert into this, this hypothetical table. Now, uh, the lease holders here are those, uh, ranges that are, uh, would have a heavier outline, that gray outline around there. Right. And so every SQL statement in a database actually gets broken down into multiple different transactions. Right. And so this case, the first transaction is begin transaction, which says, Hey, uh, I'm going to start this now. And then the second transaction is right. So I'm going to write sunny. The first thing I'm going to do is I'm going to find the range, the lease holder for the range that I need to write that data to. It's on node one. I'm going to tell that node. Okay, great. I'm going to do this. It's going to append a record that basically just says, Hey, I have a transaction and it's pending. So right now I've got this, uh, range. And so nobody can really do anything else. Like cause why I'm going to, I'm going to do some work here. So it's going to be pending. And what it's going to do is it's going to say, I'm going to, I'm going to temporarily write this data in, and I'm going to go tell the other replicas of this data, that we might not even know it's like, Hey, temporarily write this in. Okay. And so they're going to go ahead and do that. And it's going to, like, one of those notes is going to return back on a knowledge, maybe before node four, node four is a little bit further away. And basically at this point, I have quorum on the first part of this data of this, of this insert. Right. And I've actually now, I can actually wait for this thing. Okay. The app comes back. The transaction now is also going to say, Hey, wait, I've got to write the second row. And that's going to go into the second range Aussie. And I'm going to write that record. It's going to go do the same thing. It's going to say, all of these records at this point are kind of like penciled in. That's why they're all yellow.  All of the records are now committed. And I'm going to send back in an acknowledgement back to the originator. The application is asking for this. And now I've actually done this distributed transaction. Now I just went through a whole world of complexity in about. Two, three minutes. Doing this is not simple. Lots can go wrong. But I just want to give you a little bit of a sense about what core is, right? Two nodes out of three actually have to write data before we can commit it. Right. So I want to tell you a little bit about the core. There's a really odd number in terms of replicas, you know, Chris talked about, Hey, we can increase the number of replicas. That's important, but it's got to be a number, right? Cause I need core. I need an, you know, so two of three does if it's five, it's three of five. No, it's had to write that data. And so we go through that. We actually deal with that, that transactional layer. And this is how we actually are able to, to, to ensure that that transaction are happening serially in the database as well. I have four minutes. So Chris talked a little bit about pipeline and parallel commits. So typically when you write a transaction, there's lots of back and forth. I'm gonna start this transaction, it's pending, I come back, I'm gonna write Sunny, I'm gonna write Aussie back and forth. But what we're doing is we're actually taking the Sunny, the Aussie and this transaction and we're doing it kind of all at once. And we're giving it a staged, basically status if you will, right? So we can actually start doing all these things all at once and we're actually pipelining transactions. This is one of the ways in which we improve performance. And actually we can actually send this all over in one package. So the question about network and bandwidth, all these other things, we're actually doing some really interesting things underneath the covers to actually conserve the resources that are actually utilized in the database. Now, once all those three things commit, we can actually come back and have a commit happen and we've done this in say half the time, right? And so there's delta between what a normal transaction looks like in the serial mode and what we're doing from a pipeline mode greatly increases the latency and what's happening in the database from a performance point. And so that's my last point. That was the last slide. I am at, let's see, 150A here. I talked a little bit about the requirements at the very beginning. What is distributed SQL? A, it's gotta be SQL. B, I hope we showed you how we can simplify this scaling of a database. I hope we showed you how you can actually have a very resilient database without thinking about other systems. Let's let the database deal with that, right? I showed you a little bit about transactions and how we can actually deal with serializable isolation. And then finally, and we didn't really get into the demo, but I kind of showed a little bit how we actually able to dive into a location as well. So we are at the top of the hour. There's lots more. Our docs is a great place to get started. Go download CockroachDB core. It's on our website. We give you ample opportunity to go find it. You can start a cluster of minutes with Cockroach Cloud, swipe the credit card, get a three-node cluster, up and running pretty quickly. But I would implore everybody, please do. There's a survey link here. It's cockroach slash 2VBW2GW. If you could, please do that survey. That would be really, really helpful to us. We'd love to get feedback in terms of, I'm gonna dump this into the chat really quick here. Let me just grab this guy. I'm gonna dump that into the chat for everybody so that you can actually do the survey if you like. That would be awesome. If you like, from me to everybody, to everyone, not just our organizers. If you all can take that survey and let us know how helpful this was, it would be amazing.  So thank you, everybody. We are at the top of the hour. There was one last question about transactions. I will absolutely answer that. We'll get that out to you in an email to this person who asked. And I believe that's the end of our webinar. So if there are any other questions, please do feel free to reach out to us. I think info at cockroachlabs.com. I'm Jim at cockroachlabs.com. Chris, are you Chris or Chris C? Chris C. Yeah. He's Chris C at cockroachlabs.com. So we're happy to answer these things, but any of the questions that were here in chat will make sure to get answers for everybody. So with that, thanks again, everybody, for taking your time out of the day right at two o'clock, well, at least the East Coast time. Listen, have a great day. Stay safe. I hope everybody and their families are well and everybody they love ones are all doing well in these crazy times. And thanks for taking the time to learn about us and have a great day. Thanks, Chris. Thanks, everyone.