Alrighty. Thank you for that wonderful introduction. I hope it's all clear to everybody. Keith and I have done webinars a few times together. And so as noted, we are happy to get questions throughout the course of this. I know sometimes you wait until the end and answer things, but we are happy to get questions along the way. We're going to give a little bit of a presentation upfront, but we're going to get deep into, you know, we're going to have some fingers on keyboard and show you. A demo here as well on this, on this topic. This is something that is very near and dear to my heart. I'm, I have been kind of in the Kubernetes community for a while. And I'm a real advocate of all things that are happening in our community across lots of different ways. But in particular, you know, running workloads in Kubernetes that that use data and then use state and, and these sort of things. But more importantly, in this context, how do you do that globally dispersed?  I think that we have, and it really comes down to, you know, I think of that as federated applications. You know, the SIG Fed group in the Kubernetes community has had some fits and starts, but it's always because there's like some really difficult things to solve. And I hope what we present here today is valuable to everybody and can help give some context into some of the challenges that that are faced when we, when we do get geographically distributed in our Kubernetes deployments and how do you run our applications. So we are going to use CockroachDB as an example of how we're solving for this problem. Well, that's what we do. We are a database that was kind of designed for this world, but, and that's what we're going to talk about. And sorry for not capitalizing Kubernetes here, you guys, I'm so sorry. I apologize to the entire community. But by means of introduction, Keith, you want to give a quick introduction of your experience in the, in the community and kind of where you come from. Yeah, absolutely.  At the top of the call, they're like, you got to be really polite and nice to people. You know, that's not what I do, Jim. Why did you do it all the time? You invited me to that. I love you. So Keith McClellan. So I have been in the greater cloud native space for what, seven years now. I started in my, my career in containers at Mesosphere. So, you know, kind of a precursor to Kubernetes. I have, I have been kind of leading our Kubernetes efforts here at Cocker Slabs now for a little over two years. What the thing that kind of brought me to Cocker Slabs was one of the things that's hardest to do in a cloud native ecosystem is manage state, right? So that's kind of why, why I joined the company, right? It's, this is a really hard thing to do. And Cocker's TV quite frankly is the easiest database I've found to run regardless of where you're running it, which makes it perfect for running in Kubernetes. Yeah, in pods on Kubernetes. And yeah, and likewise, and Keith, I go to Keith all the time with some really deeper technical questions. So y'all feel free to ask whatever you want of Keith. We can go to lots of different levels. I originally got introduced to Kubernetes. I was at a company called CoreOS and I, I too saw the problem with some of the state, like I'm a next developer application needs database. How does that work? Right. And so that's how I got introduced to Cockroach. And I'm happy to be here now. It's, it's interesting in things. So let me give it a little quick introduction, five, 10 minutes on it and some slides and we'll get right into demo, you know, Cockroach. And I think what Keith and I both saw was this is something that was architected for the cloud. But ultimately I believe it was architected for Kubernetes. You know, Cockroach is direct descendant of Google Cloud Spanner. It was actually the Google Cloud Spanner white paper, which was the kind of the, the, the, like how we got started here. Three Google engineers, Spencer, Peter and Ben are three founders actually started this company to try to build Spanner, but not tied to, to, to Google infrastructure. How do you build an open source version of that? So that's really the, the, the foundation of where we started, but this is a relational database and all the beauty that you get with the relation database with familiar SQL, with consistent, you know, isolation, a strict, strict sterilizable isolation. But more importantly, this isn't a document model underneath where you're kind of accelerating complexity when you start to change things or the more model, the more documents they have, the more difficult. Let's use the, the simplicity and the elegance of, of a relational database, but make that distributed. And that's really what cockroach delivers, but we implement, we are implemented as a series of nodes. You can imagine each one of these running in a pod, spin up a new node, point it at the cluster, and it just takes care of rebalancing and dealing with kind of where data lives in that cluster. We'll show you that. We could scale further. We could scale across regions. And when you start to scale across regions, this is where Kubernetes starts to have some problems, right? Are you going to have, can you run one Kubernetes cluster across multiple regions and clouds? That's difficult to do. And federating a single control plan across all that, that's some tough stuff. Lots of companies out there trying to figure that out right now. And I think it's interesting. But for us, we just federate at the data layer. We just create one single logical database across, you know, all the different regions. And we're going to show you that in cockroach. But again, being able to do that, it's not simple. You actually have to deal with the speed of light. It becomes a big issue. Now we are going to show you today, not only can we do that across regions, one single logical database. Well, we could actually deploy a single logical database, ask any node for the data that's going to access the data wherever it lives across three different cloud providers. And I, and you're going to see a diagram of what we're going to do, but we can do that as well, which there's a lot of questions. There's just such thing as a multi-cloud application. We have a multi-cloud database, like one single logical database. Any endpoint in cockroach can service reason rights. But cockroach in the name really comes from our resilient nature. You know, depending on what you want to survive from an HA point of view, whatever your failure domain is, is it a node? Is it a rack? Is it a server? Is it a rack? Is it a data center? Is it a region? Is it a cluster? And so there's a lot of stuff that we do within cockroach that allows you to survive the failure of a node without service disruption. What we're doing is we're ultimately writing data in triplicate across multiple different nodes. And then we use raft. It's very similar to the way that SCD works in, in Kubernetes. In fact, we contribute to upstream, to upstream SCD. And a lot of the stuff that we've done is actually under the cover is actually helping Kubernetes. A lot of the things that we've found to fix an SCD. So if you look at raft SCD slash raft, there's lots of innovation that we've done there. And that's a lot of the area we contribute. This is a, our code base. I like to think of as a, a PhD in distributed systems. There are some really interesting things that we've solved for as we've kind of battled the speed of light. What's really interesting, I think, when you start to think about multiple different clusters and the way that cockroach works, because we are a series of different nodes, you can simply set up a cluster and it's one single logical database. Any node can service a read or a write. And that's really important because each node is, is the same binary. In fact, if you wanted to bring up our management UI or DB console, which Keith is going to show you, you could point that at any cluster and God, man, Keith, I hope you don't kill the node that you're actually doing it on later. But that's it. You're going to get into that dude. We've, we had some fun building this demo. I think you're going to love this demo. But our UI is actually service from any one of these you can literally. So like it's, it's, it's kind of a core distributed principle of what we do. We can ask for data, but find it anywhere within that cluster. And that's not a node that's in the U.S. West, but if that data lives over in Europe or U.S. East or whatever, it could find that, right? But we locate data. We allow organizations and customers to pinpoint or pin data to a particular location so that we can beat the speed of light so that we can actually locate data closest to users. Now people love this for other reasons, like can I comply with data privacy regulations to say all German data lives on German servers, that sort of thing. So there's some really unique capabilities and cockroach, but it's very well aligned with the distributed nature of Kubernetes. And in fact, built within the same core principles, the same way that pods work, the way that we survive, the way that we can scale, very, very similar. In fact, the same kind of, you know, at CD and RAF type stuff, right? And so some core principles at the base of what we're doing, very, very similar to, to, to Kubernetes. So that's a quick intro. It's actually an important intro because it's actually going to allow us to explain some principles that are going on in Kubernetes underneath when we get to kind of the networking stuff and everything else.  Yeah, absolutely. So when we start moving to the cloud native, actually leave, let's stay on that previous slide for a couple of Oh, sorry, buddy. I was going to the QA and I got lost. Sorry. No, that's okay. So when we start moving to, to cloud native disaster recovery, the, we need to treat a different class of events as if they're an everyday experience. So when we're managing your own data centers, the most common type of failure domain that you're managing for is rack failure where you have, you know, you'll have a switch at the top of your rack that switch might fail. That'll take out all of the communications to any of the servers that are in that rack.  Right. As we move to the cloud, we've had to start treating either availability zones or regions as kind of the failure domain, depending on what we need to be able to survive. And that means that we need to automatically heal from that type of a failure, because as soon as a human has to be involved in a, in an event, then that's what basically what makes it a disaster. Right. Because humans are disasters and computers are highly available. Right. So, so when we, when we start talking about this, we want to make sure that detecting a failure is autonomous. That the healing process is, is automatic. That we don't lose any data. That if we do have a service disruption, that's RTO recovery time objective, that it's measured in seconds, not in minutes or hours. Right. We want to, and we want to make sure that the people that are tasked with fixing the problem are the people that are also tasked with building the solution of the problem. So in a lot of cases in traditional DR, where you have two sites, the application team effectively created the problem, but the solution is really on the storage team because the storage team is then restoring back from backups and tape drives or, you know, offsite storage or whatnot. So instead, what we want to do is we want to be able to detect a failure and continue to operate. And we're going to demonstrate how a one of the various ways we could do that here today. I think what's interesting here, Keith too, by the way, I love the process owner. Where do we think about these things? And it's actually extending up the stack from storage all the way up into the application layer. And I think we architect for disaster and we are, we architect for DR now pretty much at every layer. Right. Like I think there was another version of this. We were talking about it from a security point of view and certificates. I mean, there's lots of different ways or points where you want to think about DR in a cloud native world. Right. Yeah. And so what you really need to do is you need to change your perspective on how you think about DR. So, so legacy thinking on this is, is designing for failure. Right. Right. This thing has failed. I need to get it back up and running. How do I do that? What we're doing is we're designing for survival. We saying, we know this thing is going to fail. How do we make sure we continue to operate even when that happens? And that she slight change in perspective informs an enormous number of design decisions around how you build your system and where you invest your time and energy. And what the, the side effect is that it ends up actually being less expensive to design to survive. Then it does the design to fail because for, when you design to fail, I've got to have a complete cold standby of everything. So I have to buy two times my infrastructure. Right. Just to be able to recover because I don't know, you know, there could be a fire in my data center. Right. That's the type of disaster I might need to survive or recover from. If I'm designing to survive, what I'm actually doing is I'm in our case, we're splitting across three data centers. We're going to jump to this here in a second. If I need, if I lose the data center, I just need to be able to continue to service all of my requests. With two of the, with my two remaining data centers. So instead of having two times my infrastructure, I really only have 1.5 x my infrastructure, right? To survive a data center failure. So I can actually bring my, my costs down by being more distributed and using a database like cockroach DB to enable that. And then, and then avoid all the, the manual labor costs of manual remediation of an event. I mean, that's not even active system. It's not even the labor costs. It's the mistakes. Yeah, exactly. I mean, so when I was setting this thing up this morning, because this demo's got a lot of moving parts, which is why we're, we're going to show the, we're going to show the, the Julia child's, uh, finally, final baked piece and not the, I like to think about it. It's one of those fishing shows where they have a scuba diver down there underneath the water and they put like a fish on the hook. Yeah. Very much, very much like that. Um, you know, when you're designing, when you're designing to survive these things, um, there, there's just a lot of stuff that can go wrong. So when I was setting up this cluster this morning, I put in the commands in the wrong order and I had to blow everything away and get started again. And that's the kind of thing that happens when you have a traditional DR, a human runs the wrong restore command or a human does like most disasters are human created. They aren't actually infrastructure created. Like I think I don't remember what the exact statistics are, but it's like seven out of 10 disasters were caused by a person doing something they should have either not been able to do. Or even if they should have been able to do, they should have been double checked before they were able to do it. They should have been able to do other types of things that cause disasters. Are there like me, they fat fingered it. You know what I mean? Like it's hardly everything, anything go around and always does. Right. So, so designing to survive when someone does something stupid, um, like we all do on a pretty regular basis is, is really the most important thing. Um, so, um, if you want to jump to the next slide. Yeah. Um, I just, it's just a really point. It's funny, Keith. I often what's the trade off of, you know, in my opinion, I think, in the new world, the model world, we designed for things to be available and optimize for that. And disaster becomes something you don't, you don't, you don't react to. It's actually you designed into the system almost, right? And so it's interesting. So yeah. So, here's a picture of what we didn't do. I didn't set up three cockroach DB clusters and three different sites. And then kind of try to set up some asynchronous replication application across them because I might lose data that wouldn't be globally consistent. This is this is actually really hard architecture to manage because this is typically what people do though, right?  If you were to do the try to span multiple sites with a traditional database, what you would do is you'd have instances of the database in multiple locations. You'd have some sort of synchronization process. You also have some sort of a sharding process to be able to kind of make this stuff distributed, at least partially, it'd be really painful. It'd probably break a lot unless you had some really smart engineers who basically built a distributed database for you. But if we move to the next slide, I'm going to start show even if you're doing this and you're distributed in a single environment, you're still not solving the problem of latencies at scale across multiple clusters, right? And I think that's, and you're still managing three clusters, but you're managing three databases as well, which operational nightmare. So, right. And well, and the reality is, is your failure domain there is a pod or exactly node, right? And so, um, so what we did was, um, we created a single logical cluster across three different cloud providers, um, we're, we're using the different Kubernetes distributions in each of them actually. So we've got EKS in Amazon, GKE in Google and AKS in Azure, um, mainly because those were the ones that were convenient for me to spin up.  I would have loved to have made one of these, uh, an open shift region. And now that Rosa is, uh, is publicly available. I think I'm going to add a fourth region to this demo, but I didn't get a chance to do it this week. Um, under the covers, we had a couple of network peering options that we could have selected. I think folks, if, if anyone that's on the call has, has been on one of our previous webinars, you've seen us do this with something called, uh, Submariner, which does an IP sec tunnel across the sites. Um, it's probably what I recommend for, for most production deployments today, it's a pretty performance solution. Um, uh, but to, to kind of highlight, uh, a different way of doing this, I chose to use something called scupper. So scupper is another, uh, open source project. It creates something called a van or a virtual application network. Effectively we're doing TP, TCP over HTTP. So, um, so scupper creates these kind of TLS tunnels between each of the sites and, um, and then uses that to encapsulate the TCP, um, packets. That, uh, would normally be transported between our nodes. So our nodes talk over, over GRPC, right? So, so the GRPC traffic is then going over these scupper tunnels to, so that we can get, uh, communication between all of the nodes. So Keith, this is like solving. I think when we think about federated clusters, one of the biggest issues is networking, right? Cause you have different networks and then that, that's what you're solving here. The other is security, right? And so I know we didn't do this in this demo, but you have solved for that as well in the past as well, right? Yeah. So in this case where, um, I am pre-sharing, um, TLS certificates. Sure. Um, one of the great things about, um, another open source technology called vault from Hashi corp. Um, is it also uses Etsy D, um, and uses the same kind of underlying protocol as Etsy D uses raft. Um, so you could, um, we could have just as easily set up scupper and set up vault and had a single, um, certificate authority, right? Sign our certificates. Um, that, that's something that we'll probably add to this demo over time. Um, yeah, we have shown it at to your point before you with Submariner, um, Submariner is a little bit more sensitive to having that as a requirement than scupper is. Scupper lets you pretty easily inject like a sub sign CA, which is what I did for this demo. Um, whereas, um, uh, Submariner is, is going to do its own. Um, it's going to reach out to cert manager and actually request certificates, which is the right way to do things. Scupper hasn't got that feature yet. That's the only reason I wanted to highlight the key that we talk about, you know, designing for disaster, um, and actually thinking through and, and, and it's, it's part of the overall architecture. There's so many different layers to think about federating of clusters is not simple. You're still going to have to manage three clusters here, but what are the things that you can do in this distributed system to do it across multiple? And I think that's a, you know, and I think the, the networking thing, we're getting closer to solving this at scale with, with lots of different solutions here, but we're talking about networking and three different cloud providers here. So I don't want to underestimate the magic of what's going on in this, this diagram, like that's pretty phenomenal dude. So yeah. And, and we're peering all three of these networks across three different cloud providers where the infrastructure for the cloud providers doesn't provide any capability. That's right. They don't want to, by the way. They really don't want to. Yeah. Yeah. And so like, yeah, like you talk about multi cloud, uh, or even hybrid, like if you want to run OpenShift on-prem and you do it, that's just a multi-cloud problem in my opinion, multi-hybrid. It's, it's multiple, multiple regions. You're going to have this problem. Right. So yeah. Um, yeah. And I think it's, it's, uh, this can, this is happening. Is it easy to do this, Keith? I mean, um, I set this up, this, this demo up this morning. So it's not that hard. Um, the, it's, I wouldn't say it's easy. I think Scupper is probably currently the easiest way to do this. Um, I think Submariner is a bit more robust, but it's a little bit more difficult, although there's some big changes that have come to that project in the last month that make it a lot easier to install and working with, uh, one of the guys on the team over there to, to update our Submariner demo, to use the new install pattern, which I think might get it closer to this. Um, what I really like about, um, Scupper is that because it's over TLS, it's over HTTP, I don't have to do any kind of, um, I don't have to get the network team involved and get, uh, firewall rules set and whatnot. Um, the downside is that it's, um, it's got more overhead because we're wrapping all of our TCP packets in HTTP, right? So, um, so it, it's better for environments like this where we want to kind of try something out and see how it goes. Um, whereas Submariner is probably what I would use today if I needed to do this for production workload. Um, what, what's really cool though is both projects are making huge leaps forward, um, in a very, in a real rapid way. Um, I've already been, I've been interacting with the Scupper team over the last few weeks when I was trying, like, when we started talking about this demo, what three weeks ago, Jim, they already put two bug fixes in a Scupper for me to make sure that this demo was, was working for us. They've been phenomenal. Uh, I have every confidence that this is going to be every bit the production grade solution that, um, Submariner is going to be, and it's going to have the advantage of, um, being more, uh, being more flexible in its deployment pattern than Submariner. Um, but I don't, I don't, I also think Submariner is probably going to end up being the most more performant solution.  I think there's different approaches that are emerging, and I think the right approach for each implementation is going to win for that implementation. And so I just, I'm just excited that you were like Keith, I know how hard you work and this, this, the, the concepts are here to actually do this. You just need to go out and find them and do it. And that's the funny thing. And that's what's interesting about this community, but let, let's move, let's get out of slides, dude. So let's talk about what we're going to do here in the demo. Yeah. So if you move to the next slide for me, um, so we're running something called TPCC, which is a workload. It's a, it's a standard space benchmark, TPC.org. Feel free to go read about it. It simulates a warehouse workload.  Just a hundred. Well, you're like just 150 warehouses, right? Yeah, we're running 50 warehouses per site. And that, that workload client is just kind of, you know, generating some load against the cluster. Yeah. And then we're going to use something kind of neat. Well, one second before you do that, by the way, the TPCC workload, we actually package that in the binary with Cockroach, a little commercial. We package a couple of different workloads within the binary that if you wanted to start using Cockroach, Cockroach demo, you can actually run the TPCC workload against our, our database today. But yeah, and we test Cockroach to be up to 140,000 workloads. So that's like three, roughly three orders, four orders of magnitude larger than what I'm doing today. I was this is just kind of a, this is more of a functional demo than than it was meant to be a performance demo. And then we're doing something called using something called KubeDOOM, which is literally what it sounds like. It's doom, like in software 1997's doom that's they released it as a GPL project and someone hacked it to do chaos testing on Kubernetes pods. So we're going to be killing some Roach demons in KubeDOOM and it's going to kill pods in our AKS region. So a couple of which is kind of fun. And we're going to show the database keeps running. It may end up killing the workload generator in the east because it is not monsters don't appear differently based on what the pod is. That's right. So I might, I might act as a pod of monsters, a monster. I might shoot the TPCC demon. It actually is not thread safe either. So it might, I might kill KubeDOOM itself, which could be fun. So it's a little bit meta using doom to kill its own process.  The pod is going to come back. Right.  But in the trick here is, and you may actually kill the admin UI that we were hitting, or maybe you're doing it from another region. No, but the point is, is like, once a pod dies, you're killing it. We're killing the pod, not the instance of cockroach. We're killing the pod. Kubernetes automatically is going to restart that pod and cockroach is going to start in that. And the database is not going to have any sort of. Well, the database will continue. The database will continue to run regardless of how long it takes for those pods to recover. Exactly. The UI, I think I'm actually reading the UI out of the Google region. Oh, good.  Yeah, so I shouldn't, although I am using load balancer. So one of the other things to keep in mind, this is this is a specific discover. So a van uses forward proxies. So when when I pull up the screen, each region can see all of the pods for all three of these regions. And so in the east region, it's going to see the central and west pods. Those are actually proxy containers. They're just all they're doing is acting as kind of a transport layer for the for the community, the nodes to communicate with each other.  So why don't we go ahead and if you could stop sharing your screen, I will start sharing mine.  Love this.  There's something like 40 pods running in this environment. So so there's a there are a number of. Hey, Keith, real quickly, there was a quick question there. I mean, one of these regions could have been on bare metal, right? I mean, it's there's no I mean, whatever, like, yeah, regions or regions, the clouds, the cloud hybrid environments. It could have been it could have been equinex metal. We just chose these three because that's what we had. It's good. We chose these three because I had accounts that spin up infrastructure for these three. I could have these. We could have just as easily done this in digital ocean or done this on bare metal if I under under your desk. Yeah, I think we probably would not have wanted to run it on my laptop, but we could have fair technically. I mean, I I tested Coob, do my laptop. It did work. So left hand side of my screen, the first three boxes, the red, green and blue boxes. Those are just the workload generator running in the background. So we're processing transactions. I kicked this off about 45 minutes ago. So we've been processing transactions for a while. The black area of the console screen is showing you the current status of the pods in East. So this is an Azure. So you can see the the cockroach DB dash East dash zero through eight pods are those are the actual local nodes that I'm going to potentially be killing. There's also the internal proxies. These are the outbound proxies for for the cockroach DB pods that are local. And then we have the inbound proxies, which are marked as central and west. OK, there are also a couple of other things going on here. We have a client container, which is the what's actually running TPCC. We have the service router and we have the service controller. The only thing you're not seeing, because this is showing the cockroach DB namespace, is the coob doom pop, which is running in the coob doom namespace. So I didn't switch to that. So you can see we're running. 500 queries per second. I didn't optimize the performance of the queries here at all. So ignore the P 99 latencies. If you've watched any of the demos that Jim and I have done in the past, you know, I can get these latencies for TPCC down to, you know, 100ish milliseconds for P 99 have just didn't choose to do that for this particular demo. I did make sure that we could dig into each of the regions and take a look at each of the nodes.  OK, so I've run queries running across all of the nodes in the cluster. All right, so now for the fun stuff. Going to go over to doom. I'm going to make sure that I can still fire walls. I've got no clipping set up. So you can see the names of all the cockroach DB pods.  I should still be kind of immune for a little bit. Oh, wow, Keith, really. Bloody pod killing.  You're going to start to see some of the pods. See on the left hand side, one of the East Proxies died and restarted. We got all kinds of stuff is dying. Oh, I ran out of and then you're terminating Kubernetes, just bringing them back. Meanwhile, the database is not having any sort of impact. Now I'm shooting them with shotguns so they're dying faster. Because before I was using the Gatling thing, I can switch to the one that like will probably kill me to the BFG.   You can see we're still creating containers. We killed the TPCC process that was local, but my queries are still running on the West and in the central. See, three nodes actually still haven't recovered yet. They're recovering for us. We don't have any unavailable ranges, which means we're continuing to process data as we go. And now that this suspect node list is down to zero, our under replicated ranges will go back to zero and the cluster will be completely healthy. So Keith, one of the things that we didn't kind of describe to the audience in cockroach, when we write data to a cluster, we're actually writing data in triplicate. So, you know, the replicates set is, you know, these three things that are running. So while you're doing that, Keith, so we're writing data in triplicate. We're writing across multiple different nodes, physical servers and that so that we can survive that. So an under replicated range just means that one of the replicas of that raft group is missing. And so the database just knows that and it's just telling us, oh, wait, you're missing some data here. What's interesting is that at no moment in time was there an unavailable range. A range you think of as like a tablet or a shard. We automate all that underneath the covers. So the data, no matter where it was being accessed in West, East or in central was accessible. And that's the critical thing, or we didn't have any impact to the availability of that data. But we did have, because we killed some pods, that stuff went away. Now, when the pods came back, we're just using stateful sets to actually bring that pod back to its current state, reconnected to the storage it was using. Somebody was asking, you know, what's the best practice for deploying cockroaches that, you know, well, how does that storage work? Well, each node is actually has its own storage connected to it and its own PV claim. Right. And so ultimately, stateful sets is managing that in Kubernetes.  Right, Keith? Yeah, that's correct. Yeah. So. So did you want to show the impact on the some of the metrics, Keith, or no? Yeah. So I am waiting for the cluster to decide that it can get its metrics. Oh, right. And I will reshare.   I killed CubeDoom. I killed the TPCC container. I think I killed all of the forward proxies in East as well as most of the database pods at least once. Right. It looks like I don't have anything left. I have looked. So for East, I killed half of the database pods. So anything that's here that says 4-H, you're no longer sharing, by the way, Keith, did you know? Oh, sorry. Yeah. There we go. Sorry. So for anything here that says 4-H didn't die. Yeah, because we're running for four hours. Right. But the ones that said there.  We killed, it looks like three or four of the Congress DB pods, and I managed to not kill the Scupper Router or the Scupper Service Controller, which is good. Which is good because that would have killed all of the proxies. That definitely would have taken, that's effectively like taking the entire region out of service. The database will survive that too. Let me ask you about that, Keith. If you had done that, I mean, it would have basically, I think we did this last time we ran this, right? It would resolve over time, correct? I mean, it would just basically automate and heal, correct? That is correct. Yeah. The challenge that we're having right now is I think I was wrong. I think I actually killed the pod that was hosting the admin. Oh, so you got to go to a different pod. Yeah, so. You have a different IP. So again, what Keith can do is just point our, point the browser at a different node that's running. Because I didn't set up a global load balance. Yeah, that's right. Yeah, otherwise, yeah, it would have been, there we go. Oh, there we go. Yeah, you're back. Let's see. Yep. There we go. So you can see, we definitely had an impact to our queries per second while this was going on, which is not surprising. We did just take out a third of the cluster, basically. But the database state available, it was always processing queries. And it's starting to recover right now. Because one of the load generators died, it won't get back to full unless I restart that load generator.  Which is different. Yeah, normally that would have a load balance in front of it. So it would have, it would have continued anyway. You know, our latency numbers kind of jumped around a little bit, which is not surprising. But the point is, is that the cluster survived. Right. And healed itself. Right. And I really didn't have to do much anything other than talk to you for a couple of minutes. And this is one of those things where it's like, you know, we've talked to customers and are like, well, we didn't even know an event happened until we went back in time. And because stuff just basically survives, right? And I think that's the trick here. It's like, you know, you got to actually go find when you have issues. Our biggest challenge is when customers want us to show failure in Kubernetes, it's kind of the reason why I wanted to build the demo. It's actually hard to do without, like, straight up deleting the stateful sets, because Kubernetes will regenerate the pod so fast that we don't even have, like, a substantial service average. Yeah. Yeah. All right. Well, Keith, I love your demo. I'm going to stop sharing. Yeah. I do, I do like doing chaos testing with Doom. It brings me back to my video gaming heyday. So it's always fun. Keith, there were a couple of questions in chat. And if anybody has any questions of anything we went through, I'll go through a couple of them. I think Jim answered a lot. Thank you, Jim, for answering these. But somebody was actually asking about a project. I think they were asking about Submariner. It's just Submariner I.O., right? Isn't that the one? Submariner.io. Yeah. And then scupper is scupper.io? Yeah, SKUPPER.io. That's right. That's right. Yeah. So if you're looking for that, I think those are the two projects. Let's see here. Jim, there was one question I didn't do a very good job answering about U-Limits and Docker and Kubernetes and kind of how we deal with that. Keith, are you familiar with U-Limits and whatnot? Yeah. So we don't run into a lot of problems with U-Limits generally. This is definitely a low level Kubernetes thing. When using the managed service providers, Kubernetes, we don't have a whole lot of problems with this stuff. Sometimes we run into it with self-managed. Generally speaking, you fix them outside of Kubernetes and then restart the Kubelet on the POD host and then that gets inherited. But that's not something that we normally need to even really worry about. We do have some instructions on what to set those U-Limits to and our docs to support the database because we do have minimum requirements there. Right. So another question that came through, and this is a real basic kind of cockroach question, Keith. How are pods distributed across nodes? Since those staple sets are used, does this mean it's restricted to one POD, one node? What happens if we want to score horizontally scale? Kind of a softball of a question. Yeah. So by default, we do use anti-infinities to stop multiple cockroach DB pods from launching on the same physical server, largely because we need to treat the server as a fault domain as well. And that's how. So now in a multi-region deployment like this, that's less important because we're treating our top-level fault domain. So from a replication perspective, the top-level fault domain is the region or the cloud. So we're already not going to have multiple replicas on the same physical server by default anyway. But you can enable running multiple cockroach DB pods. Generally speaking, you're going to get better performance in that scenario if you run larger individual cockroach DB pods and then scale out by having more POD hosts. Yeah. That's generally the best practice. It can be done. You can disable the affinities. But then you have to be very careful about how you set the locality flag so that we don't put multiple replicas of the same data on the same physical node. And Keith, I mean, that's a pattern to think of regardless of cockroach. I think that's just something that as you're building your own applications and services and you're running on Kubernetes, that's really good advice. Because I think it's one of those things that you don't know until it hurts you. Yeah. For a stateless application, running five copies of the same service on a node is not a big deal at all. But for stateful applications, generally, that's not a best practice. That's not to say that there aren't scenarios where we might suggest it for certain requirements. But in those scenarios, we have to. So we didn't cover a whole lot of cockroach DB architecture stuff today. But when a node or a pod joins the cluster, it announces its location in the topology. So in a scenario where I'm going to delete all of the anti-affinities from our stateful set config, I need to add the host name of the pod host as part of the hierarchy for the locality so that the database knows not to put multiple replicas on the same physical hardware. That would be the only thing that would have to be done there. But reformers is generally going to be better if you just use larger cockroach DB pods in that scenario. Well, cool. I think we actually hit all of the questions. I mean, Jim, I'm sorry, Jim, too. I got to call you Jim, too, live, buddy. Thank you for jumping in. I love that we had the voice. I think we should call him the hatch. The voice of Jim came in there for a second. It was great. Jim, did we hit everything? I think we hit everything, yeah. The few questions left. But I think we addressed most of them. Yeah. All right, well, if there are any other questions out there, we're happy to talk through. I think your code, the repo, it's all available, right? Keith, you've published all this to GitHub? Yes, although not quite yet because my notes are still it's in GitHub, but it's not public yet. But we'll get it out there in the next week or two. Oh, there's a question. There's a question about scaling down cockroach DB that I would like to answer. I missed it. I'm sorry, buddy. Yeah, it's brand new. So someone was asking about scaling down and seeing it as failing nodes. And is there a way to do it gracefully? Yes, we have a node decommissioning process. Right. So it is this is a pattern that is a little bit more challenging to do in using our current config. We are working on an operator because, effectively, that's like a day two operation, not a day one operation. So the stateful set config is great at the day one ops that will handle gracefully scaling down your cluster in a way that's not going to create data unavailability. The other option is to just do one node at a time and make sure that that under replicated range count is at zero before continuing to shrink your cluster. That's a bit more of kind of a hard muscle way of doing it, but we'll survive that as well. You've got to be careful. You can't just take out. This is true with any stateful service? Any, I mean, again, this is a truth for all things deployed in Kubernetes.  You're going to have some problems. Yeah, so we need to have a quorum of the replicas for each of the ranges available for us to be able to answer queries against those ranges. So if we have a quorum, if we have a raft group of three replicas, we need to have two of those replicas available to be able to respond to queries. So that's why we would only recommend scaling down one at a time. If all of your tables were a replication factor of five, you could scale down two at a time. And at seven, you could scale down three at a time. It's just that's just kind of the nature of consensus-based replication. That would be true for any stateful application running Kubernetes, not just cockroach to be. So Keith, let's shift the topic a little bit. And there's a couple of questions that are coming in related to something. And this is a day two operations thing is what we're talking about here. How do we scale down on these sort of things? And we get asked about operators fairly often. And I know you've do a lot of work on that. What's the current state of our operator for Cockroach? I mean, because honestly, Cockroach didn't need an operator to do any of this stuff that we were showing today. It's literally kind of aligned perfectly with this whole the Kubernetes architecture, but we do have an operator. So what is our operator delivering then? Yeah, so our operator is making it easy for users to do common administrative tasks against the cluster in a way that's safe. So all of the things that the operator is doing, you could already do with the existing stateful set. But it might involve you updating the animal files and mounting file systems. So a rolling upgrade, for example, we have a manual process for doing a rolling upgrade in Kubernetes today. It's pretty straightforward. But the operator is going to manage it in a way where we're going to do things like fully health check the cluster and not just the pods before rolling the next pod. We're also going to do things like make sure that when you upgrade, you're upgrading to a valid version from the upgrade that you're going to and you didn't skip a feature release. It's not that you couldn't already do all this stuff, because you couldn't already do all this stuff. We wouldn't be able to build an operator to do it.  So it's those types of operations that we're working on. I'd like to think of you as making things more graceful. Well, I see the operator. Like the operations. I see the operator as a way for getting how to operate CockerCB on Kubernetes out of my head and into everyone's hands. Exactly. That's the reason why I've been spending so much time. And by the way, that was the original intent with the operator pattern way back years ago when it started was how do you take the knowledge and expertise out of somebody who's an SRU as doing these things and create some sort of framework within Kubernetes? It's exactly what it is. So there was a question on if we're going to support MySQL alongside of PostgreSQL. We're wire compatible with PostgreSQL. We've gone a long way to do that. There is no plan immediately to do anything that is like MySQL. We have to make a choice. It's a lot of work, by the way. Well, so the other thing is that we're serializably isolated. We didn't really talk about database at all. We're serializably isolated, which is PostgreSQL can run in serializably isolated mode. Exactly. MySQL can't. And so it would be much harder for us to provide a MySQL compliant endpoint, because there's no way for us to signal to the client that we're running in serializably isolated mode. Whereas with Postgres, we can do that. Yeah, I learned something new. Thank you, Keith. I love when I learn things new, especially from you, which is all the time. Or never. Or never. I know 100% of the time I'm making it up 90% of the time. Right? I don't even know the quote. I can't even start. So well, let me go back to the slides. We'll stay on screen here, Keith. We can go back to this is it. So where was I? So I just realized 52 minutes into the webinar that because I'm using the fancy camera, people can see that my headphones have messed up my hair for the day, which I think is awesome. Yeah, and the problem with a fancy camera is like I have to shave, and, you know, anyway. So we are a database that was designed with Kubernetes in mind, y'all. I hope we showed that to you today. But I think there's a lot of principles that we've done in our code base that might be applicable to what you're doing in your code base or what you're actually trying to implement. I like to think a cockroach in our code base is a little bit of a course in distributed systems and some of the cool things that we have done. Keith talked about deploying and making things a little bit easier to do. We do have a Kubernetes operator that's absolutely available to all. Lots of companies using us today to do these sort of things. And Keith, what's the percentage of deployments that are on Kubernetes that you see? I mean, you work with a lot of customers. Yeah, so we know of the clusters that have telemetry enabled, we know that roughly 40% of them are running on Kubernetes and another 30 of them are running in containers, but we don't know for sure if they're running Kubernetes. So anywhere between 50% and 70% of cockroach DB environments are running Kubernetes today. These are long-running clusters. So we define a long-running cluster as one that's been up for more than seven days. So not something you run on your laptop and shutting down at the end of the night. Also 100% of our cockroach cloud clusters are running on Kubernetes. So there's a lot of internal dog fooding around our use of Kubernetes. Yeah, and so yeah, so you could spin up, start running cockroach cloud today off our website. It's all available. Any of this stuff that we went through is available on our website. But I think most importantly, and Keith, oh my god, how do we go an entire session with and out saying nice things about docs? I can't believe we made it this long. That's because there weren't any docs on how to get cockroach DB running over scuppers. Well, that's true too. There's this now. But our docs does do a really good job of, if you want to dive into how we do our transactional layer, there's questions around how we do, typically in our docs it's there. If it's not there, we do have a community Slack channel where people are always on and always answering. So if there are any follow-up questions or other stuff, I mean, we're pretty open and transparent about everything that's going on here at Cockroaching. We're happy to engage and we would love for anybody to be part of this community and to tell us what they're doing or if they have any questions about how this stuff works, Kubernetes and Database or without. And we're happy to engage and a proud member of this community. So Keith, thank you as always. Thank you. Yeah, and that was, it's a wonderful demo and I really, really like it. So I hope this was valuable to everybody today. We tried to make it a little entertaining. I love the CubeDoom thing. But this is something I've never seen before. I've never seen an application run across multiple Kubernetes clusters until I saw this demo and Kudos, buddy. I mean, it's super impressive. Thank you. Thanks. All right, so Marisa and Linus Foundation, we are done. Thank you very much as well. Awesome. Thank you guys so much for that wonderful presentation. Keith, Jim and Jim as well. That's Jim too. And thank you. Yeah, Jim too. And everybody of course for participating and joining us here today. As a quick reminder, the recording will be posted to the Linus Foundation's YouTube page. Should be up by the end of the day today. Okay, we hope to see you guys back here again for some future webinars. Okay, have a good one. Thank you. Bye. Thank you.


Who has the button back here that was a random story. I've got We had we had one guy getting serious trouble a teacher He went to it was a big kid. You know there was a confrontation. He went to push the kid here the kid ducked Hit him in the jaw and broke his jaw Oh, yeah, he wasn't hitting him. It wasn't like a closed foot is he went to push the guy and he yeah And he ducked and he hit him in the jaw Yeah, oh, yeah Yeah, no, he wasn't a lightweight either, you know the the faculty guy Yeah, it was it was pretty bad All right enough of junior high and high school Right, yeah, yeah, that all had a purpose Well, here's our final week and this is if you like Thinking about weird stuff in the Bible. This is the you thought the other stuff was weird It's gonna get sort of amped up here a little bit but we're gonna talk about sacred space a little bit again and And the concept of God being in control of chaos, which is another way of saying control Creation but really even more broadly than that being in control of everything and how in Israel I thinking those two things get married in the temple in certain ways and then we'll hit a certain point where Because of the they thought that way about the temple that essentially the temple was the center of of the cosmos there are certain things said in relation to the temple and The throne of God, which of course is in the temple that impacted, you know certain people Between the testaments to think certain ways about the sacred calendar the festival calendar of the Torah and Prophecy, so it's gonna be some weird stuff, but it's really it's really kind of fascinating because There are elements of it that kind of play Into the New Testament and some other things that are more familiar to us But this is sort of a free-for-all tonight, but it's just some kind of interesting stuff Now I had this slide up before when we talked about sacred space and about To sort of wrap your head around the way they looked at their Bible In a number of places you have to be thinking mythically and again, that doesn't mean oh, it's all just a fairy tale What it means is that there's there's a very real supernatural element not only to Episodes in scripture but basically everything all of life is sort of permeated by supernatural activity and meaning so Thinking you know mythically is really important because when writers and their readers are thinking this way What they say will often transcend sort of a simple literal reading Because they want to communicate certain ideas Through metaphor and symbol they want to drop certain vocabulary not so that your mind Parses that word like you would do it if you looked up and it looked it up in a dictionary But that all the baggage that comes with that word Just floods into your mind the illustration. I like to use For our own time period would be something like Las Vegas Okay, you know you we can oh, that's a city in Nevada. Well great Yeah, it is but there's all sorts of ideas crew To the term loss That's that's again just like what no, we're not going to get into that But that again that there are there are terms, you know, there are places there are people there are institutions Even events in scripture that when you brought them into a narrative You did so not just so that somebody could sort of oh, yeah, I can I could it takes like two hours to get there You know like like this literalistic feel But you would be thinking about the place's history and all the stuff associated with it Again that that has you know some real impact again on thinking mythically So we we use that to talk about sacred space which we our short definition was where God is or has appeared And how that was set off from normal space A couple of new elements here where God is by definition Is where there is peace order justice Which again the Hebrew term for that is shalom really all of that is shalom Stated negatively it's the absence of disorder the absence of injustice the absence of anything that's that's chaotic and by chaotic I mean Not just that things are messy and need arranging It's the the the theological concept of chaos has to do with A state of existence that is contrary to the way God wants it Okay, God wants order. He wants peace. He wants life to to run as he designed it to run When it's not that's a chaotic situation So every place that lacks the above Again these the bad things It needs to be made fit for sacred space or everything that has you know the the problems needs to be made fit for sacred space Again a more sort of official definition Chaos describes the state of disorder that would exist in the absence of divinely imposed order On the cosmos in biblical and ancient Near Eastern literature The supreme gods or the god of israel brought order to the universe and subdued the forces of chaos Somebody bigger than people Has to bring bring order to creation That's a little bit above our pay grade our job description And so people in the ancient world would assume that well, that's the gods. That's what they do That's what they're supposed to do If you're in egyptian you would sort of transfer that idea to pharaoh. Well pharaoh is the god around here So, you know, if if something isn't working, it's his fault You know, there's just this, you know, if the Nile doesn't flood and our crops just you would get half the yield And then people start starving.  This is why the plagues were really a big deal because they upset the natural order of things And specifically pharaoh couldn't do anything about it except make it worse You know, when he brought the magicians in they did, you know, it imitated it. Well, that really helps now It's like it was bad now. You made it worse. You know, why can't you do anything here? It was a real assault On their whole worldview Uh back to the quote here the chaotic Unpredictability and latent threat of the sea Often was a factor in this We mentioned the sea before because it's wild. It's untamable. You can't live there You know, it's just it's not where people belong The sea often became a metaphor for unpredictable chaotic dangerous life threatening stuff It's another reason why When when you have a situation like the sea is parted Okay, that shows who is in control of that. It's not just a miracle in the physical world sense It's it's It's the claim of supremacy. You know in relation to every other god Because there's only one who and not only that but that the one who's claiming to be in control Well, there are troops go in there and they wind up dead, you know It's this sort of thing and so this this it's an event that gets referenced In a number of ways and the whole idea of Taming the sea we could go through I'll go through a couple of them here in the old testament But you go through five six seven eight nine of these in the old testament and then when jesus walks on the water When he says peace be still it takes on a whole different flavor There's only only a few times we've ever seen anything like that. It was it was you know, yeah way of israel So like what's going on here Again, it it meant something to them on a real you know sort of cosmic level Now you have literature in the ancient world the bale cycle. I mentioned here at the bottom Is an important one Because bale is the chief rival to the god of israel and in this bale cycle. It's just a collection of tablets His main conflict is a battle against the god of the sea yam yam In canaanite is the same as hebrew yam. It's the same word because their languages are related bale's eventual victory symbolizes the triumph of order over chaos in addition to the sea itself chaos could be represented as a Sea serpent or a great sea beast or dragon That lives in the sea. Why would they do that? You know, is that like a is that like an anatomy lesson? Is this like, you know, real Reporting on some real animal well, maybe some of them thought that there was really a dragon that you know Who's tail and his mouth, you know encircled the entire world and maybe somebody thought that But it became a symbol of The badness the dangerousness of this place So When you came across things like sea and the dragon These terms show up in the bible. They show up also in the bale cycle leviathan rehab and we're not talking about the woman Okay with with josh It's not a slam on her and the deep These terms wind up in The hebrew bible and in some places they actually quote Some of these stories Like the bale cycle. This is psalm 74 These lines basically line 13 down through 15 come right out of the bale cycle Okay, why are they doing that and the the canites thought that bale did all this stuff bale's the one who subdued The sea monster He brings order to the sea to the creation Crush the heads of leviathan all this stuff and the psalmist says no no, it's not bale. It's The god of israel you know first line in 12 it god is my king from of old You read a few lines up a few lines later. It's going to be Yahweh of israel. They would do things like this To be a theology lesson It's not bale who's in control of this stuff. It's Yahweh of israel. They would do it to all different kind of deities It's basically a slap in the face To competing theologies biblical writers did this a lot in this case again, you have the reference if you look at the rest of the language, I don't think I have a a Laser here But look at the language down here You've established the heavenly lights in the sun the day is yours than the night you fix the boundaries of the air. What does that sound like? Sounds like creation language Sounds like something right out of genesis. Well, what happens in genesis? You know wait till we get there in genesis We have the waters are calm the waters the deep the spirit of god is hovering over the waters Hey, that's the way that creation is described here. It's violent Okay, god has to act to bring that into order And it's done through metaphor genesis. It's just you know still and waiting for god to get to work on it Psalm 89 you have another one You rule the raging sea. Oh lord You crushed rahab like a carcass rahab is another one of these beast names That you get in the ancient world the heavens are yours the earth is yours also The world all that is in it you have founded them Creation for an israelite was not just the story of Well, you know, we don't really understand this but 2000 years later There's going to be this guy named darwin that this is going to really kick in the butt. Okay, that isn't the point That's the way we think of creation stories because we have a problem Like like like an evolutionary worldview or something like that We'll take the accounts and start thinking about the things that our culture forces us to think about What they were thinking about is We need someone bigger than us to make us a habitable world And to make it livable It has to be orderly or else we're going to die We're at the mercy Of nature Somebody has to control that and keep it under control and that was how they thought about Creation, which is why they put they set the bringing about of creation In weird language like slaying a sea monster Like that just sounds so odd to our ear Psalm 74 why would you associate creation with killing leviathan? I didn't read that in genesis No, you didn't it's just sort of already happened in genesis Psalm 89 crushing Rahab like a carcass and the earth is yours the heaven is yours you have founded them Oh, yeah, no kidding. But what why describe it this way? The reason they describe it this way is they're trying to teach the idea of Where god is there is order god is the one in control of all order He's the one that keeps us from being exterminated by creation essentially All that You know, we're leading up to something here. You get the same language here. You get the dragon Rahab Was it not you who cut Rahab in pieces who pierced the dragon? Was it not you who dried up the sea yam the waters of the great deep to home? That's that's genesis language Who made the depths of the sea away for the redeemed to pass over Again god in control of the chaotic forces that would try to exterminate israel In all this language is deliberate Here's our genesis reference In the beginning very familiar Darkness was over the face of the deep to home spirit of god hovering over the faces of the waters Face of the waters and god saw at the end Everything was very good. There was evening and morning and that was the sixth day Okay, very good Tov ma'od Is not the word for perfect In hebra. That's tamim. This is an entry from a hebra lexicon How a lot of you care to look it up But The point is Is that when you get to the eden story? Eden like I said two weeks ago eden was a place that had specific geography It wasn't the whole world It was the place where god was it was the model for what life could be everywhere else Was uncertain All that other stuff it was it was very good God had you know brought you know the forces of chaos sort of under restraint under control But man it could just it could just erupt at any point We could have you know storms we could have earth you know all we could have all these natural forces kill us off and so When god go you know goes and creates eden puts you know adam and even the garden he commissions them To subdue the earth and fill it. Okay Yes, they're supposed to work in the garden and maintain that but they're supposed to multiply and go out and bring the rest of the world Into a state like this Like eden that was their job So that you know god's presence his control his rule would spread over the earth Among people and everywhere that they were at That's the vision That's the original vision For humanity for human life in god's world A trusting that god as you obey him That he will bless you And bring order to your world It's very simple set of ideas now The fall of course brings an end to that There's an eruption of chaos eden is no more Which is why sacred space from that point forward Has to sort of get a toe hold somewhere God doesn't give up on the plan He doesn't annihilate humanity He doesn't eliminate everything and just say that was a terrible idea Of course god really can't do that if he's omniscient Okay, god anticipates these things, you know, we know this about the plan of redemption the whole bit But from that point on it's god trying to maintain His promise his plan To work with humans To sort of start the ball rolling again And it happens in in different stories god trying to do this He enters into covenants with people he first well he creates is real out of nothing You know first it's trying to work with humanity as a whole that You know it works until you get to babble After babble you have god saying okay, i'm gonna i'm gonna call this one guy abraham I'm gonna start my own people through him That's why in the story there are direct parallels between abraham's life and adam's life Between the history of israel and the history of adam You're supposed to think about both of them at the same time God's trying to do the same thing again. That's why you have the tabernacle Looks like eden. That's why the temple reminds me of eden You know we talked about this two weeks ago All this is deliberate planning deliberate messaging And tonight, you know, I want to sort of focus on How they thought about sort of the It's not the final installment, but for an israelite what they thought was the final installment of god returning to earth restoring his presence And then everything would sort of mushroom out would blossom out in theory And the lord would return to earth and rule the earth and that was Jerusalem that was mount zion. This is where This was now the sacred mountain wasn't sinai anymore. It's not eden eden Of course is also called a mountain in ezekiel 28 not just a garden. It's none of these other places where god appeared Did things said things We are finally in the land that god chose That god promised for the people that he chose and created We're finally in there. We finally have a king The man after god's own heart, you know again this this template kind of guy And we have a temple So everything's supposed to be wonderful now Yeah, we know the rest of the story and how it turned out, but the way they thought about the temple Even here you have ezekiel 38. Ezekiel of course is writing when the temple's about to be destroyed And he's telling them why this is going to happen But the way they thought about this place about zion and the temple is where I want to camp sort of the rest of the night because it's Again, it's interesting. It's kind of bizarre Very foreign to us, which I think is why it's interesting because it's we just don't think the on these terms But as far as Jerusalem in ezekiel 38 12 Jerusalem is described as the center of the earth Now, you know, like if you did your Surveying measurements and all this stuff. Is that really literally how it's going to work out? No Well, that there's an error, you know the biblical writer must have flung geography or math or It's like look the center of the earth again. It has nothing to do with literal math and geography The whole point is that it's the center Of god's activity. It's the place from which He is now going to fulfill his original plan, you know fulfill his covenants all these things It's it's the headquarters. It's the beachhead. It's the nerve center, you know, think of it that way And you'll get the point. Of course the center of Jerusalem was the temple That was where everything focused on it's where the presence of god was Now we looked at this a couple weeks ago and I mentioned that hey, if you compare the pictures tabernacle Temple there are some differences And the differences are Kind of strange in some respects. We're not going to hit all of them, but you've got two pillars here Joaquin and Boaz Which you know, if you're familiar with like the freemasons man, they make a lot of hay out of that The two pillars there But I want to focus on number two number two and number one number two is the bronze sea And number one is the bronze altar now the altar is just a bigger version of the one in the tabernacle But you know, you don't have this In the tabernacle Okay, that's because it's going to signify something New we're not not not new like it's never been thought of before but in a new way Anybody Can look up first kings seven real fast. I want you to read Whoever can do that these four verses Just to get a little bit of description in our heads And this is not a devotional passage This is not one of those I memorized this, you know in college First kings seven 23 through 26 made the sea of cast bronze Yep White was five cubits and a line of 30 cubits measured in circumference below its brim Were ornamental buds And circling it all around 10 to a cubit all the way around the sea The ornamental buds were cast in two rows when it was cast It stood on 12 oxen three looking toward the north three looking toward the west three looking toward the south Three looking toward the east the sea was set upon them And all their back parts pointed inward It was a hand red thick Its brim was shaped like the brim of a cup Like a lily blossom It contained 2,000 bats Give us another verse or two He also made 10 carts of bronze four cubits was the length of each cart four cubits It's with three cubits its height And this was the design of the carts we had panels and the panels were between frames On the panels that were between the frames were lions oxen and cherubin And on the frames was a pedestal on top below the lines in the oxen were reeds of plated work every cart Had four bronze wheels and axles of bronze And its four feet had supports Okay Just again just some added detail there What do those things I mean just a little little bit of a We have plant ornaments We've got you know, I want to say lions and tigers and bears We've got the animals there. We've got the cherubim Anything coming to mind Okay It's again, it's this wilderness again the wild you know garden of Eden the cherubim and all that kind of stuff You've got 12 oxen You got three the four groups of three They're all pointing in different directions You've got these stands that the thing is on With wheels Okay, it makes you think of Eden What else would it might make you think of? It's going to come a little bit later If I say cherubim and wheels Ezekiel, yeah, okay. Just so just hold that thought Again, it's sort of a menagerie And What's interesting about it we want to focus on we're going to focus on a couple things It's already in you know, you when you go into The temple proper You're going to have the giant cherubim there You're going to have you know all the decorations of the garden and again the wildlife and all this sort of stuff We talked about This sort of brings some of that outside And you've got a sea it's called the sea of bronze Now Myers in her uh Anchor bible dictionary article says this just to hold out a few excerpts here It's really big in fact that the amount of water in this Is a lot more than you would ever need for washing Okay, even if like you were pigpan. I mean it just it's just it's huge She says just as spectacular as the size and the sea of bronze was its ornamentation Under its rim was a series of cast decorations two roads of gourds The rim or the brim itself was made of lily work Most amazing as of all was the way it was supported on four sets of bronze oxen with three oxen in each set Each faced the direction of the compass Keep that in mind the compass With their hinder parts facing inwards supporting the basin Now albright very famous biblical scholar Who is he died in 1971 or something like that? His parents were missionaries I'm a big fan of albright even though he sort of gets poo pooed now because He just believed too much of this stuff Taught at johns hopkins for many years. I think I think he's a wonderful guy but albright Sort of you know went off on a tangent here and he thought well, you know, what about the The oxen here and then you have the cherubim. So it made him think in one article. He wrote which I don't I don't quote here of Ezekiel's cherubim faces, you know 12 elements 12 cardinal points 12 You know with the compass points the four compass points it made him think of the vision in Ezekiel and the reason he thought that was because well You have to know a little bit about the vision of Ezekiel. Okay, it was not a flying saucer All right, so let's just get that out of the way It's also not like an early Leonardo da Vinci plan for an actual flying object What the what the 12 or what the cherubim were This is stock description of a divine throne. I could show you lots of pictures of divine thrones They are on oxen The oxen will you know will have wings They'll have the different faces. Okay, this is just this is what a royal throne looked like It had wheels. You could move it around. It was portable All that stuff fire again the presence of god sign. I all these things sort of accumulate in in Ezekiel It's not a coincidence that in Ezekiel The four faces of the cherubim correspond to the four cardinal points of the Babylonian zodiac Ezekiel's living in Babylon. He's writing to people who are living in Babylon All of this imagery you can find in Babylon including the eyes The eyes in ancient text were references to the stars If you had animals with eyes in them that was a constellation. This is astronomical language The point of Ezekiel's vision is that Marduk of Babylon Was viewed as the ruler of the cosmos Marduk controls space and time The passage of time the epics of human history And you Jews are sitting here in Babylon by the river Kivar Why because Marduk just kicked Yahweh around Okay, that that's Babylonian theology in Ezekiel's day You're here because your god got beat by our god And what Ezekiel is saying is he lays the whole thing out the whole worldview out That a Babylonian would understand and who does he put on the throne? It ain't Marduk Marduk isn't there The whole point is that yep We're here in Babylon, but The god of Israel is still on the throne. That's the inaugural vision of the prophecy And then he goes through and he explains.  Of the heavens why are the heavens important because the heavens are how you measure time That's how you measure the passage of time sun moon stars who created those things That would be the god of Israel Who's in control of those things if you're in control of those things you're in control of history Because those things are linked with time So if you were reading Ezekiel The message is very theological god is still on the throne. He has our destiny in his hand It looks really bad and it is But he has our destiny in his hand When you superimpose that to temple architecture The message is This is the center of the cosmos This is the place from which god controls time and eternity and human history There's a couple of other interesting things about it Again the bronze sea if you just look at the gallons it's 17 000 gallons You know 27 feet you've got the qubit measurement there Four feet high again. This wasn't in the tabernacle. Why? Because this is a this is a picture if you control the sea if the sea is calm If the sea is in the place Where god is Time and eternity and history are well in hand It's fixed. That's why it's not in the tabernacle because the tabernacle was mobile Not only couldn't you get all that water, but how would you move it? Okay, this is a fixed object When they get in there, they build it and it's not only that But even this is a slap in the face To Babylon, Mesopotamian religion Uh Yeah Myers continues in the temple of Marduk There was an artificial sea Marduk is the chief god during the you know the time of the exile It was called the ta'amtu sound suspiciously like Tiamat the deep Okay, some Babylonian temples had an opsu sea a large basin again to commemorate this idea It represented the waters of life at the holy center Ancient Israel shared this notion of a watery chaos being kept under control by Yahweh and again the forces of chaos the things that would would destroy us The great molten sea near the temple's entrance would have signified Yahweh's power and presence now In Ezekiel 40, I'm going to go back. I want to pick up one now. Let's do this There's one there's another object here So this is again a symbol of god's control of the forces of chaos And it's a counterpart To what the Mesopotamians have as well Now when it comes to this object It actually gets called this is a really kind of a strange thing The altar hearth in Hebrew is Har El Four cubits from the altar hearth projecting upwards on and so forth Har El Is the Hebrew equivalent of a Sumerian term Sumerian Mesopotamia Babylon. Okay, they're they're all using the same languages there the Aralu This term in the same passage Is to be connected with Akkadian Aralu a term for the nether world About which the chicago-assyrian dictionary marks that it was inter alia a cosmic locality opposite of heaven Although we should note it does all right that the ancient israelites may have understood this term as the mountain of god That's literally what har al means the mountain of god cosmic mountain Sacred space. Where else would you expect that but in the temple? Okay That's just a page from the chicago-assyrian dictionary making the point The cosmic locality opposite of heaven. Let's think about that Now it's the nether world in Mesopotamia, which is the realm of the dead in israel How would you think about that? Think back to the whole world tree thing the thing that connects heaven and earth and under the earth It goes through the altar, which is where you sacrifice Okay, there's a death metaphor going on there There's a substitutionary metaphor going on there. You're going to wind up in the nether world unless you're rightly related to the god of israel Okay, there's a salvation metaphor going there So you have two objects in the temple that talk about god's control over the things that want to kill you and this notion of To avoid eternal death We use this altar Again to be a substitute for you. There's a lot of theology packed in just those two objects And it's not just sort of operating on an earthly level. It has everything to do with again You know who's in control of all of life and death who's in control of life and death And our eternal destiny who's in control of all of history Again, these are big big thoughts and they are tied to this place Jerusalem this temple and even within the temple some of these objects This is the cosmic center of everything For an israelite Now let's talk about how this relates to how they thought about what happened in the temple You got stuff going on every day You've got The bronze sea there God has all the forces of chaos under control We've got some of the architecture The four cardinal points, you know With the oxen That's telling us God is in control of the heavens and the earth and the passage of time and history and all this stuff You got that going on In the temple. Well, we use the temple. It doesn't just sit there So we have daily sacrifices.  In other words, everything's going to be orderly. You'll have a good life Won't be a perfect life because you're not perfect But God's going to stay here.  You're not going to, you know, you know have You know, end up spending an eternity of divorce from from your God All these big theological thoughts the temple was supposed to remind them of that And as you went through your year You would do certain festivals You know you to commemorate certain events you'd have Passover Right, you would have the feast of tabernacles the feast of booths You know all these different festivals that commemorated how God Had intervened for you in the past How he had kept you from chaos How he brought you through the sea how he created you out of nothing All of these things that would operate on the calendar were supposed to remind you Of these episodes as an israelite and they all took place At this This place, you know the temple now in the temple god has a throne This throne in some passages and namely in Ezekiel one, but there are a few passages and kings and chronicles It's referred to as the merkava Gold star if anybody knows what merkava means the Hebrew term It means throne chariot Think of Ezekiel. You got the throne and it's on wheels It's both. It's dual purpose The thing can move, you know It's it's a throne chariot And Again, this is when we get to Ezekiel we see the most explicit vision of it And it's standard where you would portray a throne, but it takes on importance And conceptually in different ways This is by rachel ellior. She's an expert in merkava Stuff in Judaism In the holy of holies the divir of Solomon's temple two gold plated cherubim shield to cover the ark with their wings Their appearance revealed to david as a divine pattern. I mean god actually says what you're building here is according to a pattern That's in the heavens, okay It's described in parallel passage and chronicles which explicitly links the cherubim with the heavenly chariot throne in that passage the the The lid of the ark is called the merkava this throne and The merkava was a representation of the ritual order of cyclic ritual time In Ezekiel you got the four faces the cherubim correspond to the four points of the zodiac Why because it's constellations we can watch the constellations. They have an annual circuit It's orderly. It never changes And Jewish thought went something like this Well, look at the perfect order of all that That's pretty fantastic In that the heavens to quote psalm 19 declare the glory of god There's like eight different verbs in there for speech and communication In psalm 19. This is just a fantastic thing. It's it's regular Not only doesn't it ever change but like who could change it? You know the god made this and god's in control of it And they thought well, you know, how does that how does that work? I wonder what would happen if we start like keeping a calendar And counting days between events And why would god tell us like hey, you know, you we work six days and on the seventh you rest And what's this business with seven? Why you know, why do we have multiple sevens?  Well, it turns out The merkevo was a visual representation because it mapped time Of the ritual order of cyclic time measured in Sabbaths of days These are called weeks if you've ever done the 70 weeks of daniel, you know story weeks and days Weeks days and years you can use the terminology for all of them The fourfold annual seasons in turn subdivided in accordance with a fixed sevenfold order Similarly the concept of sacred time derived from the seven days of creation Accordingly, there are seven days in a week Counted in Sabbaths of days seven days of performance performed by each priestly course serving in the temple Seven days of consecration seven week intervals between harvest if you took the whole virtual calendar and divide it by You can actually divide it by four It's evenly divisible In this calendar Let's see here if I have the one up here Yeah, let's go with this one The priests the priesthood that came out of the old testament exilic period Again think of Ezekiel Ezekiel is when they start the exile And they're developing you know Ezekiel's writing about all this stuff They came up with a calendar that works with perfect mathematical symmetry and it corresponds to all these biblical numbers It's kind of a marvel. That's a mathematical calendar. It's not an astronomical calendar They believe it began on day four of creation. Why day four? What's created on day four the timekeepers The sun the moon and the stars So that that's the first day of god's calendar So they begin on the fourth day And all of the numerically significant events if you keep what's called a 364 day calendar with all these divisible by seven You know 49 all these numbers If you keep that calendar every feast day every Sabbath every Passover falls on the same day every year It never varies It is perfectly mathematically precise It requires no adjustment And all the numbers again correspond to these numbers that you see in the bible a lot these multiples Now they believed that it was so perfect that it has to reflect the mind of god God set this up. It's in time in tune with the heavens It never alters. It never needs adjustment If we follow it, we can be sure that we are never out of sync. What's going on in earth on the temple In the temple is never out of sync with what's happening in heaven As in heaven So on earth They had a whole developed theology Of what we do here Is not only a mirror of what god is doing It not only is what god wants us to do, but he commanded us But if we don't do it Then chaos erupts This is why when you have very liturgical religions, okay, Judaism, you know, especially in ancient Judaism Some of this works, you know into Questions in christianity about have you ever wondered like why they fought so much over When to have Easter This is a debate that went for centuries They like fought wars over when we date Easter This is why Because we got to get it right To be in sync with god We must get it right. Now the reason they had the problem is because the Pharisees did not use this calendar They used a lunar calendar Which requires them to add a month every now and then To make everything realign And if you were among the people who the kumran is where the where the dead sea scrolls are associated with This is why they left the Pharisee the Pharisee priesthood Just before jesus day and during jesus day. They left and went out in the desert To keep This calendar In a temple that they didn't have they didn't have access to it because the temple's back in Jerusalem So they actually like have these long texts where they imagine themselves doing this stuff So that god would know hey somebody's tracking with this It just sounds incredibly crazy and bizarre, but they thought they're the only we're the only guys between you know Life on earth and complete annihilation Because of this calendar stuff Again to our area. This is like this is like loony tunes But again, they thought why Pharisees? Why are you refusing to map time? According to the way god told you to map it. He laid it all out and they had a big fight over it That's actually why they left And to our area. That's just crazy stuff Now I'll just give you one thing if you take their calendar And you map out jubilee cycles Again, you can do it. Oh, you just need a computer nowadays just computer programmer gotta be really good at math They expected messiah's appearance to be sometime between 3bc and 2a d That's a freebie for those of you who've seen my lecture on the birth of the messiah Because they're the only ones in antiquity that actually plotted it out correctly They had the right window Everybody else didn't Okay, it's just to me. That's just really kind of interesting that they There's a big question in academia. Were the priests at kumran when jesus showed up? Did they all convert? Because it's like Okay, we were expecting that And and you know academics like to fight about that stuff because there are similarities between their theology and the new testament There are things things that carry over But they actually they actually got it right Uh, you know, you can't get it If you follow the Pharisees calendar, and here's here's, you know, this is the one last thing If you use the calendar followed by the Pharisees the lunar one where you have to add months every now and then The birth date of jesus wouldn't have fallen in the same period. It wouldn't have been a jubilee year, but but If you use the biblical chronology, jesus began his ministry in a jubilee year And why is that significant? Luke 4 14 through 19 The day jesus begins his ministry He walks into the synagogue at nazareth and says what? What does he quote? He quotes the jubilee passage He quotes isaas 61 and mixes in a little bit of the leviticus stuff So this is the year that the captives are set free, you know all the you know, he quotes the passage and he says this day This scripture is fulfilled in your hearing I mean, I actually think he meant that Like he knew what was going on Uh, but again, this is this is just sort of really it's bizarre. It's a little bit crazy but The people who are living in the first century who are tracking Things like calendar and chronology and messiah and by the way, they're not too crazy because Doesn't any you know, we have people who do that today I'm not one of them. I'm not a prophecy geek. Okay This I find interesting But I'm not doing the blood moon thing and all that kind of stuff But they were doing it in the ancient world and they were using a very specific method to do it And it was based on the role of the temple in sacred space Certain objects in the temple and what they symbolized That that's how they that was their system And it has these weird synchronicities With what you actually see in the new testament now last slide If you go to ezekiel 40 through 48, which is a passage about what? The temple the idealized temple There are 60 references To jubilee numbers And they're multiples Not six 60 That's a lot And again people who are into calendar and chronology have noticed that why are all these numbers Either a jubilee or half a jubilee or a seven or something like this Why do they sync up with this view of the calendar? again If you're able to think about this beyond just sort of a literal correspondence Oh, it's about building and a construct a building Well, it'd be nice if they put a put a roof instructions for a roof in there The the temple in ezekiel 40 for the 48 does not have a roof Okay It lacks certain other things that you would expect in the temple based upon the temple of Solomon and they're not in there either So it just again, it just makes me wonder this i'm interested in this subject Boy, that's a lot That's a lot of things that's a lot of ways to when people are reading about a temple It's a lot of ways to make people think about jubilee Release of the captives Setting the captives free Redeeming the land Can all these concepts that go with jubilee and if you look at how the new testament talks about the temple Who does it talk about? I just gave it away Jesus Jesus is the temple. He refers to himself as the temple And he refers to you as the temple because you are members of his body Again is all of that a coincidence See, I I don't think it is a coincidence. I think it marks a really really intelligent frankly divine mind They they knew what they were trying to telegraph and it actually worked in real time Which I think is kind of remarkable So the purpose of tonight was again to do a little crazy stuff But some of the crazy stuff again What I want you to to encourage you to do is when you're reading through scripture Assume that none of it is throw away material Okay, there there may be a reason why It has this set of instructions this set of numbers It might just mean more than Especially if it has to do with with ritual and sacred objects and something god is doing Okay, there might be something more behind it And then always checking the new testament to see what they do with it They do really unusual things with certain passages that wouldn't occur to you but seemed Intuitive to them or they would turn around and say well You know when we were when we were over here We remembered what Jesus said to us and it didn't make any sense when he said it but now it makes like kind of sense And they'll they'll they'll run with it It's high it's hindsight. We have the benefit of lots of hindsight. We have 2,000 years of hindsight Hey, they had they had hindsight and they could look back and and things just sort of pieces You know fell into place. That's what I'm trying to encourage you to do I don't expect anybody to remember this or really necessarily even you know care to go back and go through it the lesson for tonight is Read carefully and don't assume anything is there by accident Because it may not be It may not be they may think about it quite differently Yes, yes, it's all It could all be more profitable than you can even imagine, you know And again that that to me that that's entertaining is not the right word, but it's it's what It's just kind of neat Okay, I mean that that's kind of a dumb word But it's just kind of neat that there could be so much packed into the description of an object Well, who cares about you know, the okay, the altar is bigger big deal. You called it Har-El. What do you do? Well, it's the mountain of God. That's why it's a big deal and the Babylonians have one too, but ours is better You know, we're we're basically, you know poking them in the eye They're just things like that lurking in the background of a lot of passages And it you know, it takes work to ferret some of that stuff out But that's why they call it bible study You know bible studies not bible readings spend a little time on it newt a little bit And you know, you'll find some really interesting things Just trajectories you can just run on And I like it. It just makes me appreciate Again, the it makes me appreciate scripture Uh, something more than um It's more than Shakespeare. Okay Um, it there's there's just an intelligence behind it. I'm not trying to diss Shakespeare anybody's a Shakespeare fan Uh, but there's just an intelligence behind it that you won't find in anything else So anybody have any any questions any? Now that we're into crazy time crazy talk, but Go ahead There there are images that are common to both No, there's the there are points of both that telegraph the same ideas, you know one is Order you know holding chaos in order the other one is again time in history and the cosmos That's a really good question because we're not actually ever really told You know the scholars assume well, it's water. What do you do with water you wash? You know, it there's no passage that ever has them washing in it. They're never directed, you know It it was just kind of there. It's supposed to mean something Yeah, I think when they lose the temple by definition, you know, you're going to lose most of that Now you could go back You could you could observe Passover Technically see that The regulations for Passover in Exodus 12 are actually a little different than they are in Deuteronomy For instance in Exodus 12 you're allowed to hold Passover in your house You're not allowed to do that in Deuteronomy there are other differences But that's the most obvious and Deuteronomy presumes that they're in the land and then it becomes a central Ceremony for the nation like collectively But before that you could do it in your house So this is what it's it's an illustration of the kind of thing that Jews did We don't have a temple anymore We want to keep Passover So we're basically going to do the best we can here You know, we're going to keep the Passover We're going to hold in the house with family and all that kind of stuff So you could still do things like that but as far as the sacrificial system until they build the second temple You know, which is modified then by Herod second temple is roughly 516 bc All the way into the second century And presuming they're allowed to do it They could they could do it but there was a period in between there that they just they didn't have any of that That's when you get the synagogue developed when they're in exile All they can do is teach they can teach scripture They can do Passover in their house They can try to keep, you know, some of the laws that I mean, you're still going to have priests You're not going to really have the regular courses because there's really no point to that because we don't have a temple But you can do some of those things and keep the Keep the system alive in as many ways as you can You know, without that It's if you're really orthodox. Yeah, you do the best you can without that, but but that's why they push Yeah, yeah So, you know for for the ultra orthodox, this is why the rebuilding a temple is a crucial thing, you know to restore everything Um, they they do the best they can No, this is pre pita Anybody else all right, thanks for coming


Foundation to be launched in Reiki Rick's honor? When he said cotton eaters do not die, they multiply he knew exactly what he meant. Therefore his family will ensure that his memory and legacy live on. During his emotional funeral on Tuesday March 1st, his family revealed that they would create a foundation in his honor and it will focus on young people. Reiki always emphasized the importance of taking care of the youth and that is what many people remember him by. He did not just preach the importance, he also practiced it. A family member, Kumi Naito, revealed this during his funeral service and said when they last spoke, he pleaded with him and his mother-in-law to help him bring his vision to life. In our last conversation, you said mom and I need to help you with your core vision, which is to support young people. You were so concerned about young people at risk but you also wanted young people to have the same chances that you had to break into music, arts, and culture, he said. Mama and I and the rest of the family will work with your fans to bring this vision to light and ensure there will be a foundation set up in your name to serve young people in Africa, he promised. It was an emotional send-off indeed and it was revealed that his body had been cremated. The funeral, Bianca Naito, his wife of nine years spoke about how he helped her break away from her shell and how their kids loved and cherished him dearly. When I think about nine years together, here are the things I am most grateful for. Our journey as parents started at the beginning of our relationship. He was an amazing father to Jordan. They had the most incredible bond from day one.  Our children told Reiki and me the importance of family values. He loved teaching them new things, he wanted them to always be open to new experiences. He always wanted them to stretch their thinking, expand their horizons. He took a keen interest in their hobbies and passions. He loved them deeply, he was their protector, she said. As a partner, he always encouraged me to come out of my shell and to allow myself to step into the light and conquer my shyness, he wanted me to shine.  Ours was deep and pure love.