with that tag. So yesterday was as wild and dramatic as I can remember. People were stopping me in the airport to ask my thoughts and these implications. So let's first set the table with the guys who got the tag, the non exclusive exclusive, all of it. No, it happens. The Marjaxon will begin. St. Carn Barkley gets tagged well done by them. Get a lot of kudos to the Dianne's Josh Jacobs, friend of the show, Tony Pollard, Evan Angram, Duran Payne. So let's start with this Marjaxon one because we have to dig into the situation a little bit more. It was, you know, the tag wasn't a surprise, right? That's where things have been trending. Negotiation sort of broke down. But the non exclusive tag is a little weird because it means that Lamar is given some freedom. He's given the ability to go and negotiate and try to reach a deal with somebody else. And then the Ravens have a chance to match. The Ravens, you know, having a little confidence here that they'll be able to match or that nobody's going to make that deal. And if Baltimore decides to let them walk, Lamar's new team has to send them to first round picks. So to me, this means or what it should mean is that there's a great chance that we see Lamar rocking another uniform next season, right? That is what would make sense. You know, if I'm the Panthers, for example, from the Panthers, I'd give up two first round picks and a heartbeat for a 26 year old Lamar Jackson. I would break him off. But to everybody's shock yesterday, there were these reports that, you know, a bunch of the teams and you'll take a look here. I think Field of the Yates tweeted them out and I couldn't believe this. There's five teams that have been linked to Lamar that have come out and said, or at least told insiders, we're not pursuing negotiations with them almost immediately. Falcons, dolphins, Panthers, Panthers, Raiders as you're seeing here, they're out of the running. The dolphins I get with Tua, but the four other ones, those question marks at the quarterback spot are staggering. They are glaring and then it leaves the Jets, it leaves the Colts, it leaves the Texans as really the only obvious quarterback and ED teams going into 2023. So it's really strange and I don't want to conject and there's a lot of that going on. I'm really curious to get Chris Collins with thoughts on it, but it's even weirder when you stop and think about the following. The Broncos, they were super willing to give away two first rounders last year and then some for Russell Wilson, who's nine years older than former MVP Lamar Jackson, right? And then they made him the second highest quarterback in the entire game. The Browns, who I mean, they're an anomaly and everybody hates that they did this and maybe that has something to do with this in the fallout with guaranteed money and all of that, but the Browns were super willing to surrender, not one, not two, but three first round picks and gave the biggest fully guaranteed contract in history to someone who a lot of people don't think deserved it and certainly was in the midst of a huge investigation. So you're telling me not one of those teams would be willing to give up two firsts for Lamar? I don't get it. Where are you guys at on this? At Up and Adam show. And by the way, it's not out of the realm of possibility that one of these teams do step up and take a big swing for Lamar. And I think if anything yesterday taught us that, you know, there's a lot of theories out here. Maybe we should just let it play out and see how it goes. But something weird is going on. All I know is that I hope that the former MVP gets money and is happy and gets the stuff that he has earned and definitely deserves. But we'll talk about the collusion part, all of that as we see it unfold. Giants quickly. How to boy Giants? What am I supposed to say here? New York media, all of that. I worked in it for so long. Avoid the drama. That's what they did. All this lingering drama is Daniel Jones. Are we going to put a ring on it? Yeah, they did. They said, baby, I love you.  And at least the next four years for $160 million. And they did it a little dramatic. They down to the wire as men always wait the last minute to make things happen. But I do think that there's legitimate questions he has to answer. They took a gamble. That's what it did. Like, first of all, they had to get a deal done with him so they could do what they wanted to do, which is smart and franchise, say, Quan Barkley. He was there, MVP last season. He's what makes this thing go. And he sort of ensures that this ensures that he's not going to get to go anywhere when they try to work out something long term for him or get a win with him on this tag. Now, you know, $40 million sounds aggressive per year, but it's the state of the business. Like this is the going right people. Salary caps been ballooning. And this is what guys get when they're up for an extension. Look at Derek Carr. He got nearly $40 million a year. Geno Smith got $35 million a year. And both of those guys are seven years older and have fewer wins than Daniel Jones. So there is a risk here. And they're not paying him for what he did last year, what they saw last year. They're paying him for what they think he can do. But they're saying, listen, this is a 25 year old quarterback. And I think he can, he can build off of this season with Dayball. And yeah, there's an unknown collection of receivers, but he did without that led New York to its first playoff when in over a decade. So you're not, you're paying for what he could do for you. And Shane and Dayball, I thought, well done by them. And they made it clear.  We believe in our quarterback. We put a ring on it in this Instagram official. And I do want to say this, seeing that Daniel Jones made me realize this. Brett Veach and the Chiefs, and I've treated this yesterday, I was like, obviously, but what a masterclass by Veach and company with this Patrick Mahomes extension. Everyone, and if you look at what these quarterbacks are making, Kansas City hasn't locked up for nine years. Sure, he'll renegotiate at some point. And he should, but he's just the fifth highest-paid quarterback in the league right now. Super World Champion two times. They can $5 million more than Daniel Jones and Dapper Matthew Safford. So there was a lot of work that she's doing handing out half a billion dollars to a 25-year-old. You know, they're not going to be able to keep their roster intact or get anything done. Veach and Co bet on what we all know is going to happen. The cap blowing up, and they look like geniuses now. So the moral of the story is pay your quarterbacks early. And I'm sure Chris Collins' worth coming up here will echo that sentiment of Burrow and the Bengals. Please, Bengals, go take care of Joe now. Go take care of Joe and get it done. And quickly, I don't know much to say about this, and I'm sure Chris does, Aaron Rodgers, this budding romance, this off-season fling going on between the Jets and Aaron. Nothing is imminent, apparently. Rodgers is looking at his options. It's very, you know, the insiders yesterday were all, they're on a flight and there's graphics being made. And then like late last night as I was getting back into town to turn it into, oh, we don't really, it's kind of happening. It's kind of not, we don't know what's going to happen. So, you know, with the Jets, Derek Carr was theirs if they wanted him. And it's nothing against Derek Carr. He's a good quarterback and the Jets would be a playoff team with him. I'm saying that to you. That's what I think. But the point is that, you know, they could contend with Derek Carr in their center. Was this a good idea to pass up on Derek Carr at the chance of upgrading yourself with Aaron Rodgers? You're not competing in this league to be good. You're competing to win it all. So I recognize that the Jets certainly recognize that. And I would just say that I like that the Jets are even on the plane. It is hard to do that.  It is hard to do the courting process, especially with a character like Aaron Rodgers. So just the fact that we know Aaron's not washed, he's ready to go. He wants to play again. And we are seeing this Jets brass, the Johnsons and the, you know, the Salas and company and Joe Douglas, that they're making the moves because they want to contend that they say, listen, we got Garrett Wilson. We've got Breeze Hall. We've got a top five defense. We have his back. He can do big things in New York and we want to do big things in New York. If they pull us off and they bring Aaron Rodgers to New York back on that plane or otherwise, this could be one of the most successful roll dices that we've ever seen in this league. And I just love it for the Jets and that fan base who's super excited.


The Carnegie Mellon vaccination database talks are made possible by Autotune. Learn how to automatically optimize your MySuite call and post-grace configuration at autotune.com. And by the Steven Moy Foundation for Keeping It Real, find out how best to keep it real at stevenmoyfoundation.org. Alright, welcome guys to another David's Vaccination Cinema series. We're super excited today to have Benki Ruckhaven. He is a senior manager of product at Green Plum. We've been working with him for several years now. Benki got his PhD at WPI back in New England when I met him several years ago. So we appreciate him for being here today because today is actually a holiday for VMware. It's not for the university, so we appreciate you coming on your day off to talk with us about this. As always, if you have any questions for Benki as he's given the talk, please unmute yourself. Say who you are and ask your question at any time. We want to feel like this is a conversation and not just him talking to the empty space of Zoom for an hour. So, Benki, thank you so much again for being here on your day off. Thank you for having me here. Hello everybody. My name is Benki and I want to talk to you about query optimization. You all focus very much on query execution and storage in my memory databases. This is a little bit maybe away from your core area of interest, but maybe there will be some of you who are interested in query optimization. I wanted to share with you the past, it's been 10 years since I started in Green Plum in 2010, more than 10 years. And when we started, we started building this query optimizer and we thought we could ship an enterprise level query optimizer that can handle millions of queries on terabytes of data and terabytes of data and it took four or five years of hard work to get it shipped. So, I want to show you from an engineering perspective as well as technology perspective how we achieved it and what is the intricacies that you have to know when you are building such an optimizer. So, I'm going to be going through some of the code. I'm not going to do a code review, but I'm going to show point to you certain parts so that you can understand what I'm talking about. And many people will come to you and say, hey, we are modular, we are extensible and stuff like that, but once you look at the code, it feels like where the hell do I start? So, I want to show you where it's modular and how it's modular and how we build it. So, there's going to be back and forth between code as well as the slides. So, who am I? As Andy said that I did my PhD at WPI and before I graduated, I got tired of being poor and tired for asking my husband to take me on a trip to vacation. So, I said, you know what, I'm going to get a job at Green Plum and my former manager, Mike Was, who is now the founder of Datametry, he likes to go hire PhD students because they have the difference between how much you get paid as a graduate student and how much you get paid as a first job is so high that you sometimes take the low value. So, watch out. So anyway, so I started off as an engineer and then I moved on in 2016 as a product manager and I currently handle query optimization, federated query processing. I have one of my former colleague, Francesco on the call. Hi, Francesco. So, which we do connectors for Hadoop connectors for high connector for JDBC connector for S3 or any cloud storage. And I recently started off machine learning and graph processing so I managed these three products and I have a great team working with me. So, so let me give the first couple of slides is an introduction to Green Plum. I wanted to send out a poll before say how much of you know parallel database and stuff that last week was rough. So, Green Plum is short introduction to Green Plum. Yes, Andy, you had something. Again, not all of you are my students like me from IBM here. So it's not all of my students. All right. Thanks. Thanks for that.  And basically what you have you have multiple.  So we have a coordinator node which is connected or a high speed Ethernet switch with multiple segment hosts and each segment has a segment multiple segments are typical architectures a segment host has four segments. So what you do is you take your data and you distribute it across these things so there are no duplication except for replicated tables but let's keep that aside. So you distribute the data across the tables, and then you can do join or select locally rather than shuffling the tuples across. So now you can have petabytes of data that is stored on these different segments. So we have multiple green plum instances, which have like thousands and thousands of nodes. So for example, Samsung has one and couple of other customers have one. So they have so many data processing units and now you distribute it and the way you distribute it is with a distribution policy. The distribution policy is either a random distribution or has distribution. One thing I want you to know you can think about Green Plum as each of these segments are nothing but a post business. You can think about it that way, right, the execution and then you have the data distributed across them and then you query it. So just a quick hint about random as the name suggests so you have these order staples have color coded the order item order keys. So you randomly spray on your from your coordinator node to all of the different segments. So you may have two orders from two line items from two different from the same order across different segments. For example, here you have this order. So here you have order one and order one in two different segments. That means if you want to do a join or something you need to reshuffle things. Instead, you can have a hash distribution where all of the order ones are in a particular single second. So that is another way you can distribute it. The beauty of having this hash distribution is now you can have co location of joints. So for example, if you have line item orders and the ordering joint condition is on the order key. So basically what you do is each segment will process all of the orders line item join and then then ship it to the coordinator node which can then present it to the customer client. Everybody asked for a range partitioning. You guys don't support range partitioning because anybody asked for it. So there are two kinds of partitioning. One is this is distribution and then there's partitioning where within orders you can have based on dates. Within partition you can have multi-level partitioning. First level could be year, second level could be state, third level could be Timbuktu or whatever like color of the product or something. So you can have multi-level partitioning that is called vertical partitioning. This is horizontal partitioning where it's distributed across the segment. We currently don't support leave partition to leave partition joints but that is one of the active research that we are trying to do. So say for example, let me just scribble here. So you have sales table and then you divide it by year for example. So you have 2022 here and then you can further divide it by say for example state. So and then you'll have Massachusetts blah blah blah. But so this is the distribution and this is the partitioning. And then after partitioning of state you could potentially say, hey, I want to do it by county. So the year 2021 sales are all here but within the 2021 sales there's a state and a county. So now you can still join based on the county per county. What is the sales? Say for example, if you want to find out if dead people vote in the last election by county or something, you could do it this way. So just kidding. Okay, so let me clear the annotation. Did I answer the question on this? But you can't say partitioned by 2020-2022. You can't do that. Currently you don't do that. Did anybody ask for it? No. So what they try to do that the partitioning at the single segment, that's what the vertical partitioning will help you with that. So that is what they have wanted is given a particular year, sorry, say for example the first level was a state. Everything was in the state and then the second level is year. They want to have range joins. So for that you have partitions. But the problem is we don't do leaf to leaf joins right now. If that makes sense. So the next section I'm moving, that's a quick introduction for Green Plum Disclaimers. I took a couple of slides with the citation and their request. This is Dr. Christen's one or two slides at that. So I'm just giving you a citation up there. The first couple of slides. So let's just give me a life journey of a query. You have this movie schedule and you want to join and you have an actor. And basically your parser will take this query and generate a parse tree. So basically you have a project and you have a select and a partition product. And then you do some rewrites and you split the two predicates of title, the join condition and the select condition out. And then you take this join condition and make the partition product into a inner join. And from there you say, oh, I can now then further do some optimization. I can push the select down into the movies. And then I can find out whether I use this index, that index, do a nested look join, hash join, merge join, whatever join that option available. So this is a basic lifestyle of the query optimization. So if you, this is going to be the whole talk, this rundown that I've given and how we have implemented this and how cascade optimization helps us. That's what I'm going to pitch to you for the next couple of minutes. The rest of the talk. So we, as I said, we are based out of Postgres. So Postgres has a very established 20 years research of a planner. So, but why did we decide? So I want to pitch you why in the next two, three slides, why did we decide to generate a new, build a new optimizer? So basically most optimizers, at least when we started, split the whole optimizers based into two things. Join ordering, which was the bread and butter for everybody. And then the rest of it. And join ordering, there's so much research, so much research that has been done in join ordering, how to enumerate the plans and what kind of order, how many, how big the join graphs, how to do, if you have correlated the statistics, how you can exploit that beautiful work that has been done. And CyBase keeps coming out with beautiful research every year, one or couple of years in how to handle large join graphs. Okay. So this is great. But if you see, just to see a TPCH queries, one of the queries of TPCH with six tables, this is one of the papers written by Mike Vaz and Cesar in 20 Sigma 2000. They said, if you look at in the parallel database world, the amount of search space this join order solves is just 100,000. There is a billions of other plan alternatives, which interplace between aggregate, CTEs, correlated queries, apply all of those things that is still left and that's still not touched. How do these, this rest of the green space interact with the join order in the black square? So that is what the Postgres planner lacks. They're making, they're making a lot of inroads every year. Definitely. But when we started out the project with Green Plum, they said, hey, this whole space, there is no interplay between all of the rest of the things and join order. Now you will say, hey, thank you. What is this rest of the things? The rest of the things is basically if you have partition, how do you do static and dynamic partition elimination? How do you nest queries? How do you collocate your distributed queries? How do you do aggregates? If you want to do local filter, intermediate aggregate, local aggregate or final aggregate, can I push aggregates between below joints and have collocation of the execution? How do I do window functions? How do I do CTEs from TPCDS in an efficient way? How do I pull the predicates and push it in such that the plans are better? What if the data is replicated and what is the data distributed? All of these is what the green blob talks about. It's like, hey, I need to know. I need to not just worry about the join, but I also have to worry about the interplay between complex queries. One thing I want you to know, if you have not worked in an enterprise product, the queries and the things that I used to work when I was in the PhD, which was TPCDS and stuff, they look like child's play when you see customer BI tool-generated queries. Some of them are 20, 30 pages long, sequels that are auto-generated for some tool and then you need to figure out all of these parameters and queries and components. How do you generate a good plan for that? All right, so we wanted to build an optimizer and when we looked at it, again, this was a little bit old when I was graduating. So usually in car, I don't know whether you all know this, but if you see infinity, I think it has a Mercedes engine. And most of the Hyundai or any other Volkswagen, most of the cars, different models have the same engine with minor tweaks. So you have many models that share the same engine. But when we started, we had everyone writing their own storage layer, everyone writing their own execution layer, everyone writing their own optimization layer. So basically what we saw was one model, one engine. So right now we have many of the enterprise operators have successfully done it, separate storage and compute. But what about query optimization? We want to do separate Spark execution engine, separate execution from the system. What about optimization? We need to also separate. We can also do what Orca wanted to prove is, hey, if you can separate storage and compute, if you can separate the execution engine from the rest of it, you should also be able to separate query optimization out. And that means if anyone wants to build on top of it with the same API, not that it's trivial, but it need not take six years for you to build, maybe it takes you about a year and a half to build it. So that is the pitch that we wanted to do. And so, and one of the things why we wanted to, the goals that we wanted to achieve is, hey, Green Plum has millions of queries on 10 terabytes, hundreds of terabytes and 10 terabytes, tens of terabytes of data, and it's distributed. So performance is very critical. That means if you get a wrong plan, it's not about a query that is going from 10 seconds to 100 seconds, it'll never finish. So right now I have, for example, Morgan Stanley is our reference customer. They cannot one with the Postgres planner, several of their reporting queries that they have to, they have to go and say to SCC saying, hey, this stock seems like a problematic stock trade. So I need to report this. So these queries with Postgres will not work because they are finally focusing on joins and not of the rest of the problems. So and the second thing is when we started in, when I came into Green Plum, it was Postgres 8. Now we are in Postgres 9.5 and we are in the GP7, which is the master, which is not yet released, is on Postgres 12. That means the planner and the executor and the storage is changing underneath us. So that means every time I bump in my Postgres version for the rest of the ecosystem, Orca or the optimizer needs to keep up with it without major surgery. That means our API has to be clearly defined and distinct enough from the system. Only thing that I need to change is the, like the driver that connects to the database. So I have to make sure that it's really extensible and really modular because if I have to, if Postgres, say for example, came up with Brindindex or some other index form, then I have to support that. And how do I do the cost model? How do I do the cardinality estimation for spatial joins, which Postgres supports? How do I build all of those things while the database underneath is changing? So that is one of the goals that we wanted to do. So Postgres, as I said, was not designed for distributed data. The code complexity for the planner, I'm talking about Postgres planner. The planner, the code complexity is quite high. It's okay for quick pictures, like singular features that you want to do, but if you want to do a swath of features and build on one on top of each other, it's much harder. Adding new features are challenging and it was, it was, that's why we felt it was time to build a new big data-based driven query optimizer. And we didn't reinvent the wheel, we used cascade optimization paper with Grit's Graphi and others. They have a beautiful framework. So we said, hey, let's learn from that and learn from 20, 30 years of query optimization knowledge and build something that we can improve quickly, we can enable future developments and we can connect to multiple platforms. This connects to multiple platforms, we kind of failed, but I will tell you why. So before that, yeah. Good question. Like, so in terms of like being, I mean, the multiple platform was hawk. You'll come back to that later. But like the, Postgres now, like, as you said, they add new features and you have to now support that in your cost model and certainly incorporate that in also your, I guess the question is like, there's a bunch of knobs you can tune in Postgres to make it the optimizer to do certain things. Do you ignore all of that and basically say you have your own cost model, you have your own understanding of what Postgres is actually going to do and therefore you can do a better job than their planner. Correct. So the first avatar was, I don't care what Postgres is doing. I'm not going to take in all of the bad habits that the Postgres has. I'm going to try to figure it out on my own. I just, I'm going to use what your execution, say for example, if Postgres didn't support merge join, I'm not going to generate a merge join. I know what Postgres executor, how the partitionings are represented in the catalog, how the data types and casting are represented in the catalog. If you have to do coalition, how the coalition is represented in, I need to know all of how Postgres internals work, but I can generate a plan. I can generate a cost model that's totally orthogonal to Postgres. Right? But now what problem happens is now that the worker has established, they want a convergence of at least a one-to-one mapping saying, hey, these are the Postgres things. Are there some things in there that you can migrate over? And we are a little bit, we are a little bit muddy in the water where we are saying, okay, these things, say for example, the page cost is a very commonly used thing that they try to hint to optimize the planner to pick index scans instead of sequence scans. So those kind of, there are certain knobs in Postgres which are really important for customers, the BI customers, and that one we want to migrate, but we are trying to keep the balance of, you know, what is the right thing to do, what level, because we just don't want one-to-one mapping for everything. So what happens in a modular structure is, so you have, this whole system is say for example, a green plot. The parser comes in, generates a parser tree, and then we convert it into what is called DXL, data exchange language, then that's how Orca communicates to the world. And we have a small driver, a translator sitting inside the host system that can take Orca's language and convert it to what Postgres can do. And while we are optimizing, we have many metadata information that you need to know, and then we go to the catalog to find it out and give it back. And with all of this metadata, we can now generate a plan and it can then be executed and produced outside. So basically what this is doing is Orca is just communicating with the host system, whatever the host system is. Currently the easiest host system is any Postgres-based host system. You could pretty much lift and shift things with some amount of work, okay? So whether if you want to, we actually had a project of trying to do it with Postgres itself, but given that the optimization cost for Orca is a little bit higher because we searched larger plan space and for short running queries, which is what for all TP queries, sometimes the cost is a little bit too much for the customer to bear. So we have something called layered optimization, but currently our primary focus is green plum, but we have something in the back corner to do that. So let me just focus on metadata provider. Basically what we want to do is to say, hey, there is this Orca and it communicates with the rest of the world and tells me, hey, I have an int8 and I have an int4 and I want to do a comparison. Can you tell me what it is? And then it comes back and saying, Dave, this is it. Hey, what is the hashing operation that your system uses for this data type? You tell me, I don't want to know. Hey, I have this expression, this constant expression, one plus one, if it is not an integer, your executor engine can generate a different value for it. Let us assume it's a non-integer, right? Two dates, like you tell me how to compute. So if I have to do constant expression, I create a constant expression and I say, here, cost system, you tell me what the value is for this and then I'll get back to constant. So any kind of metadata saying, what is your partitioning scheme? What is your distribution policy? What does your table looks like? What are the indexes? What are the constraints that you have? All of those information, you go to the whole system. We have something called a metadata provider API that can connect to this. What this gives us is if you have a file-based API, what I can do is I can pull up the whole database in a file and anything that I want, I can just go to this file-based provider and why is this useful? So I'm just going to jump to the code and show you what I'm talking about. So this is a simple test. This pretends to be the database. So basically what it says is, hey, this is the sequel, this is what, this is just a comment. Then I say, these are the hints that I'm trying to do. And this is the metadata. This is the statistics for some of this column that I have. And this is the statistics, all of the statistics, this is the entire statistics bucket that you have in PD statistics, but it represented in our format. Then it may have, say for example, let me go down, it may have, what are the cost function between int, and this is the, this is the int for a courseable cost function between these two data types. Then you may have, there's too many statistics stuff. Should have pulled a better example. So say for example here, most of it is statistics, but you could have data, you could have column, you could have type, you could have partition, any of those things you could, you could base pretty much, the file will provide you what the data, what the optimizer needs, whether it is operations, statistics, schema, indexes, any of those things, constraints, any of those things. You said that, it wasn't clear, but if you have a one plus one in the expression, like two, where two equals one plus one, for the one plus, is Orca going to then evaluate one plus one, or you go as the database server? If it is an int, if it is not an int, we will tell the, it's called constant expression evaluation. We'll just say, this is a constant, but the whole system, you tell me what the value of this is, because I don't want to be in the business of telling what the whole system will executed as, if that makes sense. Yes, I understand. Yeah, yeah, yeah. I remember the MySQL guys talked about how their query afterizer would actually run select queries, while in the middle of a planning, given query. That is a little bit different. What you're trying to say is, say for example, you have a sub query that could actually be evaluated to get a constant. You could say A equal to select max from foo. Right? You could just say, select max from foo, go and execute it and come back and tell me what the value is, so that rather than a sub query, I have a constant. Yeah. That is a little bit different. So that is an, that is called an init plan in Postgres and Green Plum. So that is a little bit orthogonal question. Okay. All right. Thanks. Okay. So I'm going to skip this. One of the things of having Orca as multiple, the vision we had was to have multiple databases using a very powerful optimizer engine. That you could have a dedicated optimizer engine and you could just send all of your jobs to that so that you can, because optimization can take a lot more memory and you don't want your coordinator node to be hung up on if you are searching a lot of space. So, but we haven't, that is maybe a pipe cream. I don't know. So, so one of the things we wanted to do is to make sure, since we are going on, we are going to, we keep adding new Postgres versions. So we want to make sure if anything is affected, it's a very modular and extensible. So for example, if you have the memo need not change, memo is how we represent the plants or the third space need not change. And operator, if you have new logical or physical operators that you need, you want to have to create those logical operators. And then I'm trying to come up with, so if you go to a cart race, go cart race, you will see when you are going in that big, gasoline filled room, you'll have new engineer, new people joining into the race in between, right? So what we are thinking, the query optimization is that go cart, the track is the query engine, the X form, all of those things. You want to add a new operator, just put it in there so that it will go into that engine rather than re-got the whole thing. And I'm going to show you how we do this in the talk, but that was our goal that we wanted to do. So what does that mean? So we divide the Orca architecture the following way. So we have optimization tool, this is the optimization tool. So if you have new operator, say for example, you have Jennifer Wittem or someone comes with a new joint operator, all you have to do is a joint operator that you do, whether it's logical or physical, and how you go from one to the other, and then Orca will take care of the rest. Provided it meets some of the criteria, and I will show you what are those criteria. Then there is something called property enforcement. Say for example, I want something to be ordered this way. Then how do I enforce the property of ordering in a certain way? Another one is cost model. So we had multiple cost models, we had three cost models that we built.  And that's actually sometimes how the cost models are done. So we came up with multiple cost models across multiple hardware on the cloud, on a standalone system. And then whenever the user can now pick which cost model they want. And it will not affect the whole query optimization process. And the job scheduler, the memo, all of those things don't need to care what operators are there, what models you are running, what transformations you are doing. That is just like a machine that says, this is what I need and this is how I'm going to make the cookies. Let me dig into what I'm talking about and what are the different steps in query processing in Orca and how we do it. So the first step is pretty much push down, preprocessing step, basically what we say is anything that you can think about that you should do it without like, it's like saying, hey, you should walk 10,000 steps away. That's like no brainer. You should do it for good health. You should have less fatty foods like instead of New York state trying to, or city trying to tell you you just logical things that you should do it without anything. So these kind of preprocess that you just do it off the bat. The next one is exploration.   And then you do statistics derivation, then you implement from logical to physical. And then since we are, we are MPP system, all of the MPP specific issues that we face, those are the things we didn't do. And then we need to do a costing and then we need to do a plan generation. So what are the preprocessing? For example, say, say if you have a derived table, that means select star from within brackets, select star from foo, order by A. So there is a derived table where there is order. SQL standard says, if you have an order within a derived table, it is not guaranteed outside that you can remove that. You can remove that in our order by, because if it doesn't have a limit, if it has a limit, that's a different thing. If it doesn't have a limit, the SQL standard says you don't need it. So why would you sort a billion rows to not guarantee that it is not going to be used outside? The order is not preserved outside. So another one is, hey, if I want to do a collapsing of joints, collapsing of unions, if I do collapsing of joints, then I can now look at the whole joint graph to figure out what it is. What if there is predicates on a particular table? Say for example, I know for foo table, all of the values are less than 10. And then I'm joining with bar table, which has the values from one to million. So I can use the predicates from the table foo, saying that, hey, all of the values, the values are less than 10. Now since I'm joining on the same column with the other side, I can now infer new predicates on the other side so that I don't have to hash it if I'm using a hash joint. So there are plenty of simple effective rules that you will always use immaterial of cost or skew or any kind of data distribution. So these are the things that we do so that we can make sure that before we come inside the query optimization, things are sanitized. You're doing this on a logical query plan or like how? It's logical query plan. So if you want to, if you have say for example, if you have, if you know one sub tree is always going to be false because some constant expression evaluation tells you that this predicates, if one is not equal to one, you know it is going to be false, that the whole sub tree is false. Therefore you can remove all the whole sub tree. So any form of things that you can do without any cost or skew or any of those things, just do it beforehand before you enter. Then what you do is then you start doing exploration, exploration is saying, hey, I have this joint. So let me use this. The query I'm going to be using is T1, T2, joining on A and B, ordered by A. The first table T1 is distributed by T1A and the second one is distributed randomly. That is T2 is distributed randomly and I'm joining them on A and B and then ordering them on A. So what I do, I have an inner joint. So I have an inner joint and then I have get T2 and get T1. So now what I do is I push it into a memo. A memo is you can just like basically, if you imagine an Excel sheet where each row represents equivalent things. So I put it into an Excel table where the first one is, hey, I'm a joint and my first child is group one and the second child is group two and get T2 goes here and says, I am T2. I'm group two and get T1 is group one. So basically you shred the plan into a table or format. Why am I doing this? Then all I have to focus is on a single level. Every time, if I want to optimize a joint, I just look at the single level. I don't have to look at the whole tree to figure out how to do it. So it's a top-down query optimization. So you just look at each level and then you trigger it. Now you'll tell me how do you do this? That's what I'm going to get into.  So I have to do associative commutativity. So I'm going to flip the joint order. So instead of one two, I'm going to do two one. So that's why I'm still doing in group one. I don't have to look at group two and group one. Group zero is just I'm flipping the joint order. So what does this mean? You can do any kind of joint order here. Basically you have an inner joint where T1 is the outer side, T2 is inner side or T2 is the outer side and T1 is inner side. That's the only thing. So any form of joint order. Say for example, you had an aggregate. So you had an aggregate. You said, hey, I have an aggregate. I want to split it. I want to split it into local aggregation where I can do local aggregation on each segment. And then I'm going to redistribute it so that I can do global aggregation. What do I do? Within that group, say for example, it was group four that was group four, which was the one that was doing the aggregation. I just look at the aggregate operator node. I split it. And then the local, the global aggregation will be in the same group and the local aggregation will be inserted into a new group as its child. Is this clear a little bit? Any questions? Okay. Yeah. Yeah. Okay. So then I say, whenever I'm trying to do any form of aggregation, I always want statistics along with it. Right? The steps one and step two are interplayed always. So how do I know? So how the way the cascade operation does is there is something called required properties and something called derived properties. Required properties is what I as a parent node need my children to provide. For example, if I'm a function. So here, here I'm only showing physical, logical operators. You could have scalar operators also. They will also be in their own individual group. Say, for example, if I was a function, I want to know, I want to, I have to tell my parents that I'm a set returning function. That means what is coming out of me is not a row, a scalar, but a set. Because that may be needed for how it is, how the data is going to be processed. Say, for example, if I know for a fact that I'm a null returning function, possibly, then I can try to figure out, can I do some optimization tricks to use this knowledge that this scalar is null returning? What if I'm a volatile function? Say, for example, to date, if I'm saying insert int and I do to date, this to date has to be executed only once. So I need to know where this function needs to be executed. So all of these properties, each function will say, each scalar operator will say, I can derive that I am a volatile function. I am a set returning function. Then my parent will ask me, are you a set returning function? Oh, you are. Then I have to do this way. Are you a volatile function? Oh, you are. Then I can do something. So the parent doesn't need to know the intricacies of what the child is doing. The same way the statistics says, hey, I'm an inner join and I'm joining between A and B. That means for me to estimate the cardinality of A and cardinality, the joint cardinality, I need to know the histogram of A and histogram of B. So what do I do? I require my group, my children, to give me statistics of A and B. And the group says, hey, I don't have the statistics of A and B. So I'm going to ask my metadata accessor, go to the catalog of Postgres, and get the statistics and represent it to you. Hey, this is what it is. And I cash it. Within a session, we don't assume statistics. We don't have even a statistics change. Within a session, that is what visibility is. So in the same way, we'll get the statistics of B. And now the parent says, oh, you derived the histogram of A and B. Now I can use that to join and then predict what the cardinality of the inner join is. So these are the two examples. Given the time, I'm not going into that. So then what I have is, okay, now I have statistics while I'm doing query optimization, I always know whether I should apply this rule or that rule. But before I go, I'm going to show you the code a little bit. So, so if you see here, so this is the NRA join. So the NRA join says, hey, these are the transfer. This is a logical NRA join. It says I can, I have several transformations that I can apply. For example, I can go from NRA join to a dynamic programming join order. I can do greedy join order. I could do min card join order. I can do a DP version two, which doesn't have a cliff as the dynamic programming with a clip. A little bit simpler would be here, for example, inner join. Hey, I can implement inner join as index apply. I could do a bitmap index nested loop join. I can do a hash join. I could do a nested loop join. All of these transformation transforms, basically each operator will register itself. What are the transforms that you can apply on me? And then the transforms will say, the precondition is I need a tree of this. I need the scalar properties of this. So now from an engine perspective, all it needs to say is, hey, I have a row in the memo. I'm going to take the operator. I'm going to say, what are the logical transforms I can apply on you? And they say, this is the logical transforms I can apply. And then it just applies this transformation.  I need to know cardinality. Can you go get the cardinality? We have an API that the join operator. So the join operator itself has how to generate statistics for it. Who to call? How do I, what are the transforms that need to be applied? Am I a good candidate for cardinality estimation or am I a bad candidate? For example, if you have a large nested tree, as you go up the join order, your cardinality is going to be really screwed because you will have attribute level correlation. So if you have an option of doing joins of two base tables, you have a higher probability of getting it right, rather than a join operator on top, which has maybe skew or some, the cardinality and the bubbles up so high that it's really useless as a candidate for cardinality estimation. Andy, you had a question. That's not an enforcement rule to do that, but it's like a preference, it's like a priority, right? Each operator will say, am I a good candidate or a bad candidate based on my tree? Okay. Because the parent knows how deep the subtree is. So given a memo group, it'll say, you tell me whether I should I be, if say for example, you come up with a cardinality estimation at one point in the optimization, and then you create in another tree which has a higher probability of correctness. Then you will ignore the cardinality that you created before and use the new subtree's cardinality estimation. So it's like a binary thing, like I'm good or bad, not like, okay, all right, yeah. We wanted to do it as actual probability, like a zero to one, but it was too over-engineering, but then a simple, it's a three flag, whether a good, bad, don't even think about it. Got it, thanks. So then what do you do? You go into the physical implementation, and in the physical implementation, when we also have merge join, but I didn't use merge join as an example here, you come up with a hash join and a nest loop join. So as I showed you, each inner join or any join operator tells you what are the implementation operators, transforms that you can use. And the transform itself will check to see, have I met the preconditions for this to be applied for a subtree? For example, you should not be optimizing or implementing a subtree of a join, of any of my plan, when parts of it, your children are not implemented. So say for example, you have a sub, you have a tree and the root, you're trying to do a physical implementation, you will never do a root as a physical implementation if that some of the subtree is still not physically implemented, because it doesn't make sense. For example, that's a simple example. Then okay, then then we come to the last. So we have all of the alternatives of logical things, then we while doing that, we did the carnality estimation, then we did all of the logical to logical transformation and then physical to logical to physical transformation. But still this doesn't look at, hey, am I co-located? Do I need to shuffle some things? Do I need to order? Do I need to do any materialization? If it is to save on computation cost, all of that information comes into the distribution enforcement framework. So for example, you have the coordinator note that says, hey, finally the customer wants it on the coordinator note, that is a singleton, and then it also, they also want it to be ordered on t1.a. That is my requirement. So I'm just looking at these two properties, which is distribution and order. And the inner hash join then says, okay, I'm going to hash distribute the first side. And the first side, this is the top one, the inner join is in one memo, the group one is in another memo. The group one then says, hey, I have a physical implementation for you and it's distributed by, I derive that my distribution policy is t1.a. Oh, the parent looks, the enforcement framework of the optimized, the engine will say, oh, the parent requested t1.a hashed, you are distributed on t1.a, yeah, it's a perfect match. So I don't have to do any enforcement. But the second side, oh, now I have, I want to co-locate both t1 and t2. So I say the outer, the inner side also has to be hashed by t2.b. But now group two says, hey, I am randomly distributed. Now there has to be like, a person, a judge who comes and says, I need to read the requested property and the derived property does not match. So what do I do? I add a redistribution of t2.b so that now the requested property and the derived property are matched. So this is how basically the memo, so now if you have a replicator, for example, if you have a replicator table, I introduced a new scan for replicator table. What would t2, if t2 was a replicator table, it'll say, hey, I'm replicated. That means I satisfy hashed. So the enforcement framework, you don't have to rewrite your whole optimization engine to have this work. If you want to use, if you want to create new kind of storage or distribution policy, any of those things. So say for example, Andy, your question is, if I say, hey, I'm range partitioned, if you want to come up with a new distribution policy all you have to change is how each operator generates its distribution policy and how the enforcement is done. Nothing else. So the last step is, okay, so we figured out the hash problem. Everything is collocated. You still haven't solved the order by. So the order by now comes in and says, hey, I need to enforce the coordinator node gets everything in order of t1.a. So now I have two alternatives. One, I have to sort it after the hash, after the hash join, and then I gather it to the, gather merge it to the coordinator node. Alternatively, I gather merge, I gather everything, and then I sort it on top of that on the, on the coordinator node. How will you make that decision? If the number of tuples are small, then you might as well do the sorting on the coordinator node. If it is too large do it at the segment level.  So the parse tree part and the Postgres plan statement part, you need to write, like meaning anyone who wants to use Orca, they need to write. The rest, Orca will handle it. So that is the overview of end-to-end optimization of Orca works. And what are we looking at? Okay, Venki, you finished all of this. What is it that you want to do further? Currently join order predominantly focuses on how we are doing collocation. But it doesn't do how does collocation interplay with partition elimination? Because you could have a plan that is perfectly collocated but does not do enough partition elimination. So how do we accomplish that? The second one is with Postgres and others, we have spatial predicates. How do you do cardinality estimation for spatial predicates? Then we have enhanced in, you have multi-level nested inquiries.  So we have 10 levels of in predicates. How do we unbox that and make it unoptimized? So currently the way Orca does it, it generates so many alternatives that it has the optimizer, even though the plan generated is correct and fast, the time it takes to generate that is too slow. It takes a couple of seconds, which is too expensive for Orca to take a hit. Another one is index plan generation. How do we, as more and more indexes are coming in Postgres, how do we support all of them and how do you do better cardinality estimation, especially with correlated statistics, how do we do it better? Then we have something called global deadlock detector. So because we are a distributed system and we have updates, you can have panel updates and you don't want to take, right? You want to take only the locks that you need. So how do you do, how do you generate a plan that takes less locks? The last two is, we may not get to it this year, but I really would like to do is parameterize queries because every engine has some queries that are built in and you just send parameters. How do you, currently Orca looks at it as a black box? That means it's not taking full advantage of it. So we want to do a better job with that and multi-phase optimization. So what is the example for multi-phase optimization? So, Menki, you gave me hundreds of transformations, but you are letting the engine figure out the order in which these transformations are fired. So can I have phase one, apply these transforms? Postgres doesn't need all of these distribution things. So cancel those things. Don't even look at it. So we have a way in which we can order the optimization in stages and at each stage, we can say, hey, what is your cost? Can I bail out soon rather than going all the way to the end? So this is a multi-phase query optimization that we have the prototype for, but we have not productionized it. A priority firing for the rules, right? Correct. In the sense, you want to end-to-end rules in levels. So say, for example, for a short OLTP queries, just run five rules if you can get it, get it. Otherwise, just ask Postgres planner to pick it. Orca don't even try it. So we want to... Or say, for example, one customer says, for me, this is the most important, so I want that to be fired first rather than anything else. So some form, Orca has a means already to say, you set the levels and we will fire the rules accordingly. Perfect. I'm cutting to your top 10. Prepared statements here.  It's like 40%. These are 40%, and we pretend everything is great, or currently we don't even... Actually currently Orca doesn't even support it, falls back. Falls back to Postgres. Correct. So we love Postgres planner because it's our insurance policy. But anytime we don't get it, if we smell we are going wrong, we just say, hey, you have an insurance policy, just take the insurance policy. What percentage of queries do that? Currently about 70%, 30%, Orca 30% Postgres. So for instance, all of the catalog queries, we don't even touch. Because we say, we don't need to even touch it. So if you have more OLTP queries, then it'll be much more Postgres. You have to look at your workload and see what it is.  But since someone calls, execute another pair statement, then you can take the variables being passed in and then use that to declare. Exactly. It's not perfect, but you can do that as a starting point. Yeah. So the code is right there. It's all open source. The Orca code is back in Orca. And a list of publications are here. Please do check us out. So we have quite a bit of publications more to come soon. So that was a short, not so short description of what, how do you build a optimizer roundup with cascades? Questions? Thank you. No problem. I have a ton of questions, but I don't want to be selfish. I'll open up to anybody else if anybody wants to go.  So my question over here is that if you look at the modern query engines like even Delta, for example. Yep. More and more we are pushing the decisions. They can, they can actually, for when you do a hash join for some blocks, you run out of memory. So it's very bad. So then for those blocks, they turn it to merge join because you use a lot more, a lot less memory. Correct.  And also during the runtime, you collect the statistics for that particular query. And then you can change the distribution. The plan at the middle of it, which data actually is doing it today. So more and more really, because particularly when you go to Disney or the engines, they don't have a lot of stats. And this is data lake and the stuff varies a lot. So you can just go collect a lot of data. So more and more is being pushed to that. So is this really prepared to lose such things? We haven't looked at, we haven't, we haven't looked at the data lakes aspect of it where, our potentially maybe it'll be challenging. We've had a real tech, it was all for because you need to be able to take the query, come thin and then it's like a one shot thing. You would have to be able to say, Hey, maybe the query plan has told you about, here's an updated view of what I'm seeing whenever the query and then apply your rules. And I think it'd be a major rewrite. Right. Or you could have the sub trees where you are fine.  So say, for example, if you have a data lake where you're pushing sub parts of the queries into different green plum instances, for example. So within a particular green plum. So if you have a federated lake of multiple green plums, for example, and you query comes into one green plum instance, you could potentially divide up into sub trees where a particular subtree can be paused and read them. Maybe I don't know if you haven't looked at it from that perspective, it is going to take a lot of work for that. We had a question. Yes. I got another one. So how aggressive are you in use of material as views? Not that much right now. Okay. We have a beta version of material as views on green as a whole, but we don't fully support material as views in GP seven, but GP six. You have to manually refresh it. It's not otherwise. No. Do you do the position part or not? No, we don't right now. Okay. So what about the correlated sub queries, which is the bane of existence in the MPP is because of any square problem, right? Because for every node, then it wants to do correlated across all the nodes. So it becomes in a square. So we do have a whole lot of slides for that. We do support a lot of correlated execution. We try to decouple most if not a most a good chunk of it. I, how about this? I will, I will send you the paper for what we have on it. Let me take that. We do some. Yeah. Go ahead. So you have seen what we did in dv2, which one of the best paper I will actually know it for the data engineering 1995 or six. We did the magic to correlation because we had the magic sets inside dv2. So we use that one. It is very complex because most people do it and then it's buggy and they don't work. Most. Okay. Well, I can show you what we have. I can, I can send you both what, what are the kinds of correlated queries we support and what the code to look at. So I can send you both. I'll connect. Okay. Thank you guys. I got your name. So I, yeah. All right. So my question is, um, my first question is more of engineering side, right? It's meant to be modular. And that's me that you drop in different plugins and get that you have to have these providers to get information. But like, I mean, you didn't like, you talked about how like, this is all about green plum, but the original goal was to support green plum and hawk. Um, I don't know three. So like, I guess if I had a brand new database, how much time roughly would it take to, to save my optimizers based on, based on Oracle? Like weeks, months, years.  Okay. Like if you basically, if you don't have to rewrite how the catalog and representation and everything is about six months, but if it is not, it's much harder. And that partly because it started off with a lot of optimal goals, but then we had to get to GA. So we went very Postgres heavy rather than like agnostic. So if you are a non Postgres person and you want to adopt it, it's going to be take a year. Okay. And like, when you say Postgres heavy is because it's calling like information schema queries to go get things. Basically the issue where the time, the vast majority of time that you are going to be spending is translating the logical nodes in Postgres to logical nodes in this one, all of the metadata constraints, all of the execution, also generating the execution plan. If you are a Postgres based execution engine, you don't have to do all of that. You can pretty much lift and shift most things and only target on your own area of need. So say for example, if you, if you had a in-memory database where the execution engines are like, you know, an extension of Postgres and the parser and everything is all, and the metadata is all Postgres who would take six months, but you may need some hand-holding and we are more than happy to hand-hold. And is there anybody other than VMware Green Plum that's running off Orca? Like the cal side has at least been, there's been a bunch of databases that use it in different forms. But as far as I can tell Orca is only being used for Green Plum. Right. Or any Green Plum extension, Green Plum derivatives. Got it. Okay. Okay. I guess a last sort of open-end question is like, since the cost model of the architecture is like, you know, pluggable, in theory, in the course of me, some come along and implement a cost model that happened to rewrite all the other rules and things like that. Have you guys played around with any of the learn cost models that have been showing up in a bunch of research papers or not? We would, we have not, but we would love to. Okay. We can give them something to do it. Okay. There's a lot of papers in this space.  They hooked it into Postgres. And I think of the DV2, but I don't think anybody's running production. I was just curious.


أعوذ بالله من الشيطان الرجيم بسم الله الرحمن الرحيم الحمد لله رب العالمين بارئ الخلاق أجمعين باعت الأنبياء والمرسلين ثم الصلاة والسلام على خير خلقه العبد المؤيد والرسول المسدد حبيبي إله العالمين أب القاسم المصطفى محمد وعلى أهل بيته الطيبين الطاهرين وصحابته المنتجبين صلى الله عليك يا رسول الله صلى الله عليك يا سيدي يا رحمة الله الواسع ويا بنجاء ويا عبرتا شهيد كربلا يا ليتنا يتناكن عكم سادتي فنفوزا فوزا عظي أعوذ بالله من الشيطان الرجيم بسم الله الرحمن الرحيم واللبنا هبلناتنا قرتا عيون قرتا عيون متقين صلى الله علي محمد وآل محمد أعوذ بالصلاة عن إمام زلحسن والحسين أعوذ بالصلاة عن فاطمة الزهراء مع جميعك عندما حصلت إلى العالم where you feel that it's time to get married you've finished what you've had to do, you've graduated you started working, you've raised enough money and you've decided that it's time to get married you meet people, you meet some, maybe a handful, more, less finally you meet someone who you're most comfortable with it seems like the right person, the qualities that you've been looking for and indeed the qualities that you want are found in this person whether it's the man or the woman and you get married and everything is fine, الحمد لله everything is perfect at other times there are others that are not so fortunate they meet several people they have several qualities in mind that they're looking for whether it has to do with looks, with intelligence, with personality with faith and religion finally they meet someone who they believe has either all of the qualities that they're looking for or most of the qualities that they're looking for however after a while either after they get married or before they get married of course after they get married that's a disaster they discovered that this is not what they had signed up for this person turned out to be different from what they had perceived the qualities that this person has now are fading away now this person's true colors are coming up and showing and then obviously what happens? the marriage fails if they're engaged the engagement breaks off and if they're married the marriage fails if they decide to divorce or if they decide to stay together then it's a reckless marriage it's a marriage that has no bond no love and no harmony of course either way this is disastrous if they remain together with no love that's one disaster if they decide to end it that's also a disaster because a relationship like this can leave a lot of marks can leave a lot of scars can leave a person emotionally drained and now that this person wants to remarry wants to meet someone else will have a lot of insecurities will ask a hundred questions now in their mind about the new person that they meet what if this new person turns out to be like the old person what if this person is a fake what if this person seems to have good qualities but then when I marry them they switch they turn out to be something that they're not right? so this person develops a lot of insecurities now this person is afraid of getting into another relationship this person is afraid of getting to know people because they're afraid this person might switch on them they begin doubting everyone you know a lot of people before marriage and after marriage they're two different people you see a person before marriage this person is romantic he calls 15 times a day he's sweet he's nice he's polite he brings roses he takes you out he's very different all of a sudden you're married 2 months, 3 months after the marriage and he's completely a different person now he's angry now he breaks things at the slightest comment he'll snap he'll start yelling he's not as religious as he seemed before the marriage or it could be the other way around the lady before the marriage she was perfectly fine she had a good personality she was sweet now after the marriage seems like she has a sharp tongue she has to comment about everything a negative comment nothing satisfies her why is it that a lot of times people find their spouse or the person that they've met to change all of a sudden on them what happened why do people change all of a sudden do people change you see it's not that people change yes to you this person's attitude changes this person was very nice before the marriage now after the marriage he's not so nice it's not that he changed or she changed most of the times this is how the person was before the marriage but that's the side that you did not get to see that's the side of her that you didn't see that side was hidden that side of his was hidden why because he doesn't want to show you all of his sides right before marriage he's trying to gain customers he's trying to win your heart so if he has a bad temper after a marriage most likely you have that bad temper when before marriage as well but he wasn't showing it to you if she has a sharp tongue after a marriage most likely she had a sharp tongue before marriage as well but you didn't know because she wasn't showing it obviously because she's engaged she's trying to get married she doesn't want to break off that marriage she doesn't want to ruin her relationship that's a lot of times this goes for many people before marriage the days, the weeks, the months before marriage they're engaged they're meaning people they hide their bad qualities they hide their bad qualities and they emphasize their good qualities and that's something normal you know when you go to people's homes as guests they're sweet, they're nice you know the husband and wife they're smiling, they're laughing they crack jokes the kids respect the parents the parents respect the kids but then when everyone goes back home they start punching each other because that was a show obviously there's guests they have to put on some sort of show the husband and wife probably maybe they don't talk but in front of guests they have to talk thus this is part of human nature we choose which side of ours that we want to show to people and a lot of times in marriage this is the case it's not just your spouse or the person that you're marrying we too, you too you're not showing all of your sides you're not showing your grouchy side your angry side your sharp tongue you're showing the side that is the most attractive because you're trying to win someone's heart so when you see a person all of a sudden change after 2 months or 3 months or a year from the marriage they didn't really change it's the qualities that they had before the marriage but you hadn't seen that you didn't really get to know about and this is not hypocrisy this is not hypocrisy otherwise all human beings are hypocrites no, this is natural instincts we were raised this way we're biologically engineered it's like some animals that camouflage right? why do animals camouflage? to protect themselves so that they don't get attacked humans also camouflage they don't camouflage in their bodily colors they camouflage in their attitudes so to win people's hearts no and I hope I, you know I didn't make things really bad for those that are trying to get married now that you've met someone someone you've met recently they seem like a good person he or she they have the qualities most of the qualities that you're looking for but how do you know how do you know that this is the right person for you how do you know that this person is not going to switch this person a year from the marriage is not going to seem like the anti-Christ how do you know for sure that you're going to be compatible and that you're going to have a happy wedding happy marriage and everything is going to work out perfectly after the wedding how do you know that this is the right person for you well to be sure that this is the right person some people go to extreme measures some depend on horoscopes okay well I'm a Gemini and he's a Scorpio and I don't know do they get along supposedly let's say they get along okay well the horoscope says we get along inshallah we'll get along others go and do a background check they go and do a police report about this person their background their history what they've been what they've done have they been to jail that's also extreme others depend on استخارة they go to the sayid or the sheikh I'd like to perform an استخارة I want to see is this person compatible with me or not tonight my dear brothers and sisters in the limited time that we have tonight we'd like to see how can we know for sure that is this person that we've met that we'd like to marry is this person the right person for us should we depend on an استخارة as some people do should we take an استخارة should we just take our chances and marry this person and hope for the best should I go and visit a fortune teller to see is this person going to stick with me or not what is it that we should do do you know that this person is the right person for us tonight I'd like to introduce a 10 step procedure to know number one the first step do not rush do not rush I know most of you that are not married obviously you know rush you're 30 years old more less you've waited a long time you've worked you've made some money you have enough to build a family you've waited long enough enough is enough now you want to start a family you want children you know there's some gray hair popping up in your beard and the train is not going to wait for you however as long as you've waited doesn't mean you should rush it doesn't mean you should marry that person within a week you barely know that person let's get married if you've waited 30 years you could wait another 2 months you could wait another 3 months to get to know this person why the rush a lot of couples they make the mistake as soon as they meet the person after a week after 2 weeks after a month and they're already planning the wedding what's the rush and unfortunately this is part of human nature people are always in a rush وخليق الإنسان عجولة خليق الإنسان من عجل man is hasty man was made from from being hasty wanting to rush and the hadith tell us that being hasty is not good rushing and things is not good whatever it is العجلة من الشيطان being hasty that's part of the instructions from shaitan he tells you to rush rush into things whatever it is this is a long life commitment the person you will marry this is a long life commitment for richer for poorer till death do us part you're signing up forever until death do you part so why do you want to rush good to know this person don't rush there are some couples that rush within a week within a week they feel that they've made their decision they made up their mind or within 2 weeks or even within a month that's not a lot of time what will you know about a person within a week what will you know about a person within 10 days or even within a month not a lot so take your time when they buy a house they take 6 months which is more important buying a house or getting married finding a husband or finding a wife a house is always replaceable don't tell me that a wife is also replaceable I know she is replaceable but when you have kids are your emotions also replaceable are your kids also replaceable some things are not replaceable صلى الله عليه وسلم the second step first you don't rush second ask about this person obviously when you meet a new person you don't know everything about this person you don't know about their past their history their أخلاق their manners their faith their religion you don't know everything thus you have to ask you have to ask that person's friends I'm not saying go and do a background check I'm not saying go to the nearest police station and do a background check on that person you don't need to do that you don't need to get paranoid but ask that person's friends current friends and former friends the local Islamic center or حسينية that person goes to your prospective spouse go to that center go to that حسينية ask people what have you seen from this person is this person a regular does he come to the حسينية does he attend the events ask the local sayed or sheikh or the Imam of the Islamic center what do you think of this person do you see him on and off what do you think of his أخلاق if this person was married before ask ask his ex wife ask ask her ex husband why did he get divorced what happened what was the reason do your research there's some people that do research when finding a job finding a house buying even a car they do a lot of research they spend months during their research they look online but when it comes to marriage for some reason they forget to do their research they forget to ask ask about this person's reputation ask what people think about this person this is very important people's perception about the person that you like to marry that person's reputation and let them know let them know this young man has come and proposed to you he's proposed to your family let him know that you're going to ask about him so that he doesn't get offended because perhaps people will tell him that you know so-and-so family they're asking about you you know you're religiosity so let him know so that he doesn't get offended or let her know that you've asked about her from family members from community members so they don't get offended and by the way if you're ever asked about your marriage you have to be honest that it's not permissible to lie in such cases those who lie in such cases that's the worst kind of lying you have to be honest if you're asked what do you think of this person and you know certain things about that person about their past about their habits you have to be honest you can't lie that person is asking for your genuine opinion they're asking you for advice you can't hold that advice you have to be honest and by the way it will not be considered غيبة it will not be considered backbiting even if you had to say something that are bad about that person these are one of the exceptions of غيبة one of the exceptions of backbiting you're asking for advice when you're asked for advice regarding marriage you are allowed to say everything that you know about this person you are not allowed to conceal your knowledge say your opinion say what you think of this person and it will not be considered غيبة you have to be honest give your opinion this is the second step ask about that person from the local Imam from the community members number 3 number 3 get to know this person get to know this person once you've heard good things about this person about this lady about this young man this person has a good reputation now get to know him you might say why did you first say ask about him or her and then get to know shouldn't we get to know him first or get to know her and then ask why did we switch no if you ask me of course this is not حلال or حرام this is not set in stone it's up to you but if you ask me personally I say first ask about the person ask about her or ask about him get people's perception about that person know what that person's reputation is like what if you ask about her and she ends up having having a very bad reputation do you still want to get to know her if she has the worst reputation for example people are telling you no stay away from her or stay away from him do you still want to get to know that person and waste your time and waste his time or her time why first get to know that person because reputation is everything my dear friends reputation tells you a lot about that person how people perceive that person how people perceive her him in the community what did they know of this person did they see him at the Islamic center does he participate is he religious does he pray does she wear hijab that tells you a lot when you ask about a person's reputation you've done half of your homework the other half is getting to know that person so if you ask about a person and that person has a good reputation go ahead get to know him or if you get mixed some say yes this person is good they vouch for that person others ify you could still get to know that person but if a person has an absolutely bad reputation people are telling you stay away this person doesn't have a good name this person doesn't have a good reputation why do you want to waste your time why do you want to waste your time and bring a person into your house and get to know them when everyone knows that this person has a bad reputation why so that's why I say ask first and then get to know and then get to know or do both but at least let that person have a good reputation or generally not a bad reputation now is it permissible to get to know this person you've proposed to someone can you get to know this lady someone has proposed to you is it okay for you to get to know this person sit together have a conversation talk is this permissible of course of course you can sit with this person of course you can have a discussion well as long as it's within within the red lines within the borders of course get to know this person get to know this person's this person's personality get to know them there's nothing wrong with that in fact you don't even need supervision you don't need parental supervision that young man and young woman could sit together and talk yes as long as it's not in a closed area if it's in a public area they don't need any parental supervision as long as there's no it's not a closed area in a closed room that's fine they could sit they could talk as long as it's in a legitimate means they're getting to know each other with the intention of marriage that's fine in fact that is encouraged know the person that you're getting married to don't have surprises don't meet someone and then be shocked or surprised later on now there are some I know this is controversial there are some young men and women that choose in order to get to know each other of course with the consent of the family to have a temporary marriage to get to know each other for a month or two months with the family's consent and approval and blessings they do that that is up to you that is up to the family that is up to the father, the mother and the family whether they choose to do that or not what I'm trying to say is they should get to know each other and if you ask me if you ask for my personal opinion and from experience that I've seen with people you need at least two months a good two months what this person in order to know about this person within two months within two months you know, two months is reasonable it's not too little and it's not too much it's perfect within two months you will get to see the good side of this person and at the same time you'll get to see the bad side of this person within two months you'll be able to discover is this person angry or does this person not have temporary issues and stingy as this person you know religious or not religious you'll be able to discover these things within the first two months I think that is a reasonable amount of time the fourth step do not be emotional this is a time when you're getting to know a person when you're trying to discover this is a time where you put your emotions aside this is not a time for emotions when making an assessment try to be as logical as possible and as less emotional as possible don't judge things based on looks a lot of people the first judgment that they make is based on looks is this person attractive or not is this person the way you want them to be or not if they're not they won't give this person a chance whatsoever that's being emotional basing things purely on looks or this person has a good sense of humor so he must be a good person or you clicked you clicked within the first session or two sessions that's being emotional do you know that after the 10th session after the first couple of months you're compatible that's being irrational that's being emotional at the same time there's some people that are too picky as well there's people that are too picky as well you know this person he's not tall enough for me I'm tall and I want someone to be tall and he's not tall enough or his nose or his accent that accent is this being is this being logical are you giving this person a chance you're basing a marriage based on that person's nose well I have to say it depends on which kind of nose it is it varies from nose to nose but if it's a good nose that will do you don't ruin a marriage you don't base a marriage based on these small things look deeper in that person look deeper in that person don't be artificial try to look deeper in that person look deeper in that person's qualities that person's traits perhaps you will find something that you're looking for yes attraction is necessary I'm not saying do not marry someone that you're not attracted to you have to marry someone that you're attracted to is attraction the most important quality no but is it a necessary quality yes yes there has to be attraction in the marriage if you marry someone that you're not attracted to that marriage will not last in fact marry someone that's attractive marry someone that has long hair marry someone that has qualities that have been mentioned definitely let there be attraction but is that the most important thing that you should look for should that be number one no there are certain things that are more important the fifth step now that you're getting to know this person ask the right questions you see a lot of people they have 2 months they have a year and you thought that you got to know that this person in that year but you didn't because you didn't ask the right questions you didn't make the right investigations I see a lot of a lot of young people a lot of couples when they get together they ask each other questions they ask the most silly questions what is your favorite movie for real you're going to base your marriage on what's your favorite movie well obviously he's going to say titanic you expect him to say something he's going to choose the movie that you like and when you happen to both like the same movie you think you're compatible because you like the same movie or what's your favorite food obviously he's going to tell you anything that comes out of your kitchen will be my favorite food and he's going to tell you what you'd like to hear what's your favorite don't ask what's your favorite this what's your favorite that ask real questions ask questions that will tell you something about their personality about that person's religiosity ask him or her do they pray on time when is it that you pray right at the top of the Azzan or an hour after or 5 hours after ask ask them a tricky question what time is it and make it a bit tricky give them multiple choice is it 5.48 or 5.58 make it difficult if a person that you're marrying does not know the time of Fajr that's up to you do you want to marry that person that tells you a lot knowing the times of prayers among the questions how often do you go to Majalis ask this question do you go just 10 times out of the year 10 nights out of the year then you disappear all year long or are you a regular you attend Majalis you attend your local Islamic center you attend these are the questions that you should ask for among the questions what is your view on hijab and makeup ask ask your prospective spouse ask your future wife ask her what do you think of hijab don't ask her do you wear hijab obviously you'll be able to tell ask her what is your view on hijab what is your view on makeup you want to see are your views compatible or not something that's very important is that there's intellectual compatibility as well you want to be on the same level you don't want to be up here and the person that you're marrying is over here on the intellectuality you know there's no intellectual compatibility you want to be on the same level I'm not saying you have to share all of your views it's okay to disagree but is there some intellectual compatibility there has to be so ask for example do you listen to music hopefully you'll get an honest answer and that they don't listen to music do you go to mixed gatherings mixed parties mixed weddings this is a very important question do you go to these mixed events where people are wearing flashy clothing and there's no hijab and there's lots of makeup and there's lots of interaction and there's lots of chatting and jokes if you care about these things at least you should ask you should ask your future spouse among the questions you should ask your future husband or wife do you have friends from the opposite gender and do you keep in touch with them and do you mix with them this is a very important question if you're the type that doesn't want your wife to have male friends all of a sudden you discover that most of her friends are male well at least you know this ahead of time not that after the marriage you discover that she doesn't have any girlfriends all of her friends are male or the opposite you marry a guy and he didn't know this all of a sudden we discover that he has a lot of female friends and you're not comfortable with that at least discover this before the marriage before the wedding and one of the most important questions that you should ask your future spouses what do you think of sayر حسين القزوني's lectures this is a very important question make them list at least 5 of his lectures صلى الله عليه وسلم صلى الله عليه وسلم صلى الله عليه وسلم the sixth step and this is you know this is quite contemporary we wouldn't be talking about this 10 years ago right now we have to the sixth step is go and see that person's social networking sites accounts that person's facebook that person's twitter that person's instagram that person's i don't know there's new sites every day i can't keep up with them whenever there's a new site they have an account whatever site they have to have that account go and see what are they posting what kind of pictures is she posting you know you can tell a lot about a young lady or a young man just by their facebook accounts or their twitter accounts or their instagram accounts go see the pictures are they good pictures are they flashy pictures lots of makeup the hijab is you know go see go see what is on their account go look at their statuses their tweets do they use foul language you know i see a lot of youth on these social networking sites they use foul language that tells a lot about their character what kind of language are they using what do they tweet about anything islamic what are their interests you can discover so much just by looking at a person's facebook or a person's twitter or a person's instagram go see you might be surprised you might discover things that you did not know about about this person again these are certain things you know when i got married this didn't exist i'm sure many of the elders here this didn't exist at their time but today these things exist and these things say a lot about a person's character a lot about a person's personality a lot about a person's religiosity go and see take a look at that person's accounts and perhaps you'll discover a lot of things the seventh step the test give that person a test i'm not talking about the hiv test or anything nothing like that give a person a personality test or a religious test how how do you give a person a test you give them a question a hypothetical question for example the lady for example she tests her prospective husband she tells him what would you do after we got married if i took off my hijab what would you do what would be your reaction are you okay with that or would you divorce me would you go marry another wife what would you do what is your reaction that's a very important question that will tell you a lot about that person or the man can ask the lady what would you do hypothetically speaking what would you do after the marriage if i ask you to take off your hijab now hopefully this lady is wearing a hijab when you married her she's not wearing a hijab that's another problem if you're assuming that she wears hijab you tell her what would you do if i ask you to take off your hijab look at her reaction look at her answer her answer will say a lot about her about her religiousness about her religion will say a lot about her أخلاق about her personality does she have a strong personality does she have a strong character or not ask hypothetical questions what would you do what if i started drinking what would you do hypothetical scenarios hypothetical questions this will tell you a lot about that person what if i lost my job ask her what if i lost my job and i went bankrupt would you still marry me you'll be able to discover did she marry you for your money or no you'll be able to discover did she marry you for your Canadian passport or she married you for you you tell him what if i got in a car accident and my face got disfigured and i'm not as beautiful would you still live with me you know that could happen have you seen the movie vanilla sky yes no i don't know some people can get in a car accident and their face is disfigured and they lose their beauty well if you married her for your beauty guess what it's over now the reason why you married her now it's gone are you going to stick with her or not ask him that question what would you do these are hypothetical questions that will tell you a lot about that person about that person's personality about that person's attitude about the way that person thinks give her a question give her a test this is a test إمام علي عليه السلام he gave his wife a test he gave her a test he discovered that she's extremely beautiful the outer appearance but he wanted to see which beauty the inner beauty as well he wanted to see if the outer beauty now he asked her a question he told her do you have any request for me on a night like this she thought about it for a moment she said yes I want you to change my name her name was Fatima why? she said because you're children you're orphan children when they hear the name of Fatima they will remember their mother and their hearts will be broken so change my name she passed the test and that day he called her this was a test إمام علي عليه السلام gave his wife a test why don't you give your future spouse a test so that you'll get to know them you'll discover them the 8th step get your parents opinion let your father know what he knows ask your father what he thinks ask your mother what she thinks you know our parents they care for us they care for us more than we care for ourselves they want what's best for us what's best for us they'll always keep an eye out for you throughout your life ever since you're a child they kept their eye out for you now that you're getting married they're overprotective they want the best wife or husband for you ask for their opinion this is one they want the best for you two, your parents will give you rational advice logical advice not emotional advice because when you get to know a person emotions will be involved and sometimes emotions will not let you think clearly they'll start thinking emotional but your parents they're not thinking emotionally they're thinking logically and logically they will see aspects of that person which you're not seeing things that you're not seeing in this person your parents are saying your mother will catch things that you did not catch your father will catch things that you did not catch get their opinion this is why in Islam that the father the father has willaya over his daughter he has authority he has the final say he could decide whether this person is good for my daughter or not because the father thinks logically and rationally he's not thinking emotionally you know if his daughter is thinking emotionally she's in love with this guy because he told her he's a sweet talker he told her that he loves her this is not going to this is not going to work that magical spell that the young man has on their girl is not going to work on her father he's not going to think emotionally he's going to think rationally that's why the father has the final say ask for their opinion their opinion matters your parents want the best for you ask your siblings as well ask for their opinion their opinion also matters you're looking for a young lady to marry as a wife ask what your sister thinks now your sister is going to do her own investigation she's going to look at angles and aspects that you're not paying attention to ask a person comes and proposes to you a young man comes and proposes to you ask what your brother thinks of him ask for his opinion what matters because he will look at angles that you did not look at the ninth step get away from everyone put your phone aside put your computer aside and think think rationally think logically don't let anyone persuade you don't have anyone influence you this is your chance to think about it and take your time there's no rush you don't want to make a mistake marry someone and then you discover that you married the wrong person think clearly think rationally think logically take as much time as you want and make your decision the tenth step is this necessary is it necessary for a person to make an istikhara if you're confident about a person you've done your research you've asked you've asked about that person's reputation you got to know that person you feel confident about that person there's no need for an istikhara if you're confident if you're sure whether yes or no if you're confident that you want to marry this person that's enough if you're confident that this person is not good for you that's enough why take an istikhara you don't need to take an istikhara istikhara should be your last resort when you're absolutely indecisive you've asked you've done your research you've lucked at that person's social networking sites you've gotten to know the person you've asked from his ex-wife from her ex-husband you've done enough research you've done enough thinking and still you can't make up your mind then and only then your resort to istikhara then you go to a seyyid or a sheikh or if you know how to make an istikhara and you're making an istikhara there's some people as soon as someone comes and proposes to them for marriage or they meet someone for marriage immediately they go and make an istikhara this is not right this is not how istikhara works istikhara is when you've thought about it you've asked for advice you've done your research you've done research from all sort of angles and you still can't make a decision then you resort to istikhara not the first day and there are others who are the opposite extreme that refuse to make an istikhara and they take it lightly why? if you've done your work and you still can't make a decision ask Allah SWT to help you make a decision don't be too extreme there's two extremes when it comes to istikhara those who jump and want to make an istikhara right away and there are those that reject now remember that there's something called if you've done all that you can about a person and this person seems to have the right qualities توكل على الله وعلى الله فلية توكل المؤمنون don't be paranoid don't go into this marriage with lots of fears and paranoia that I'm not sure if you've done all that you can توكل على الله depend on Allah and hope for the best and also remember the power of du'a the power of du'a when you've done all that you can you've done your research you've asked for advice and you've decided to marry this person ask Allah for help say a du'a ask Allah to make this marriage a blessed marriage and this person the most rightful person for you the best person for you the perfect person for you ask Allah there's some people that forget this they forget to ask Allah for help and to make this person the right person for them du'a du'a has immense power let's not forget the power of du'a du'a