Hello everyone, I'm Jor Aruraj. I'm a final year PhD candidate working with Andy on the Peloton project for Andy today. Today I'll be presenting a few frequently used sorting and join algorithms used in database systems. Feel free to interrupt me at any point during the lecture. I would prefer to make it more interactive, if you will. So before we get started, a couple of logistical announcements. So homework number three is due today and homework number four has been released and will be due on Wednesday next week. So today we'll be continuing our discussion on how the database system executes queries. And we will focus on a couple of relational operators, the sorting and join operators. So these are frequently used in queries and we'll look at how the database system actually executes these relational operators. So today's agenda, it's sorting and join algorithms and I will start with the sorting algorithms. So first of all, why do we need to sort? Why exactly do we need to sort the tuples? So the relational model that we covered in the earlier lectures, it does not specify order for the tuples. However, users often want to retrieve the tuples from the table in a specific order. And it is also trivial to eliminate duplicate tuples by making use of the sorted order. So you can use it to execute queries with distinct clause. And lastly, you can also bulk load a lot sort of sorted tuples quickly inside a B plus tree as opposed to doing repeated inserts. So there's a clever algorithm for bulk loading tuples that are in sorted order. So these are some reasons why sorting is actually used in practice. And there is also the join operation which I will cover in the later part of this lecture, which can also be simplified significantly if the input tables are actually in sorted order. So that's the general motivation for sorting in database systems. And this has to do with the relational model explicitly specifying that the tuples need not be in any order. So there are two cases. What if the data actually fits in main memory? The entire table fits in main memory. So this is a simple case. In this case, you could just employ a standard sorting algorithm like QuickSort. However, the more interesting and challenging scenario is when the data actually doesn't entirely fit in main memory. In this case, we refer to this problem of sorting data that doesn't fit in main memory as external sorting. And in this lecture, I'll be presenting an commonly used external sorting algorithm called the external merge sort. So the high level idea of external merge sort is very simple. It employs a hybrid sort and merge strategy. So during the sorting phase, what you do is you try to sort small chunks of data after bringing them into main memory. And then you write back the sorted data to a file on disk. And the merge phase is going to do recursive combination of smaller sorted sub files into a larger file. So it's a two-phase approach. So let's start with a simple example to illustrate the algorithm. So we will start with the two-way external merge sort. And we will later generalize this design to a K-way external merge sort algorithm. So to give some context, let's assume that files are broken up into N pages. And that our database system has B fixed size buffers for processing the operation. So that's the problem set up. So in a two-way external merge sort, in the zeroth pass or the preprocessing step, what you do is you first read in every B pages of the table. You then sort it and then you basically write it back to disk. So each sorted collection of B pages is referred to as a so-called run. Now in the subsequent passes of the sort merge algorithm, we recursively merge pairs of runs into runs that are twice as long. So we do recursion. So this is illustrated here. We basically sort it in memory, and then we create runs, and then we merge them back into larger runs. So you can see that at each step of this algorithm, we make use of three buffer pages, two for scanning the input pages, the input tables, and then one for the output. So this picture illustrates the two-way external merge sort algorithm. So you can see that in the zeroth pass, we sort each page. We sort the contents of each page. In this example, each page, you have two entries. So you sort them, and that's referred to as a one-page run. In the second phase, you basically pick two one-page runs, and you use a merge algorithm to merge it into a larger two-page run. And then in the second pass, you combine, you merge a couple of two-page runs into a larger four-page run. You repeatedly do this process until you end up with a single sorted run that comprises of all the tuples present in the table. So you can see that in each pass, we are reading and writing each page in the file once. So because we are shrinking the factor, the number of runs by a factor of two in each step, it's a log n to the base two. Those are the number of passes you will need. And then you still have the pre-processing pass. So the total number of passes is one plus log n. And the total IO cost, because in each pass you are reading and writing a page once, it's basically two times, and there are, let's say, n pages in the table. So you have two n into the number of passes. That's the total IO cost of the operation. So this is a high-level idea of a two-way external merge sort operation. Any questions? Yes. That's a good question. So you can actually stream in the input runs and write out the output run in a streaming faction. But at any given point of time, you only need three active buffers. So that's a good question, yes. Exactly, exactly. So you only need three buffers at any point in time. So in this case, B is equal to three. Yes, that is correct. The number of buffers is three. And because of that, you shrink by a factor of two in each step. So we will next generalize this design to make use of all the buffers we will have. In memory. So the high-level design of this algorithm is to employ a divide and conquer strategy, and you recursively sort the sub files and merge them back, and you repeat this process until you converge. So this algorithm only requires three buffer pages, as Aaron just mentioned. But if we can generalize this to a scenario where we have more than three buffers, so B greater than three, how can we actually make use of these extra buffers? Can we actually use them to reduce the number of disk access? Because we want to reduce the number of iOS in the algorithm. So that's the target of the general extra merge sort algorithm. So in this case, in the zeroth pass, we basically make use of all the B buffers. So what do we do? We basically produce a bunch of sorted runs of size B, because the number of pages in the table is n. You will have n by B ceiling sorted runs. So you have bigger runs to start with, compared to when B is three. And in the later passes, we basically use a k-way merge algorithm, which generalizes a two-way merge algorithm, a binary merge algorithm. So instead of merging just two runs, you're trying to merge k-runs simultaneously. So because of that, at the time you actually merge B minus one runs, so the shrinking factor is higher here, as opposed to two. In this case, it's B minus one. So it's a generalization of the two-way algorithm. We make use of all the buffers, and we try to make use of a k-way merge algorithm here. So you can see that the number of passes in this case, instead of dropping by two, it drops by B minus one. So the log base is B minus one instead of two. So in the previous case, B was three buffers, so three minus one is two. So that's why it was log base two. So this generalizes the design. And again, in this case, the total I will cost is 2n into the number of passes. So any questions about the general external merge sort algorithm? Cool. Let's move forward. So I mentioned briefly that we make use of a k-way merge subroutine in the general external merge sort algorithm. What exactly does this subroutine do? It basically merges a set of k-runs. So that's what you are doing here. So the input to this algorithm is k-sorted subarrays. And we wanted to merge them all into a single sorted array. So what do we need to do? We need to efficiently compute the minimum element of all these k-subarrays. So that we can, at any point in time, we find the minimum, and we output these elements to the output array. When you keep doing this, you end up with a sorted array. So this algorithm is called a k-way merge algorithm. And the idea is to internally maintain a heap or a tournament tree to efficiently compute the minimum element. So the time complexity of this whole algorithm is n log k, where the base is 2. So this is a subroutine that's actually used in the higher level external merge sort algorithm. So let me illustrate this with an example. So let's say we want to sort a file that contains 108 pages. And let's say we have five buffers. So in this case, n is 108, and b is 5. And we want to make use of all the five buffers. So in the 0th pass, we first construct a set of sorted runs where each run contains five pages. So every five pages you rewritten, you sort it and write it back. So we construct 22 sorted runs, each containing five pages. Ignore the last run. And in the next step, we do a merge. We merge these sorted runs. And because we merge them, the number of runs decreases by a factor of b minus 1. And you keep doing this, you end up with a sorted file that has all the 180 pages sorted in order. So if you use the number of passes, the formula that we constructed for the number of passes, you can just plug in the number of pages and the number of buffers and you will end up with four passes. So that's the design of the external merge sort algorithm. And database systems typically use this algorithm for sorting tables that don't entirely fit in main memory. So what if you're trying to sort a table and let's say there already exists an index on that table that is also constructed on the actual attributes you want to sort the table on? What if there is already a table that does the sorting for you? So can we actually accelerate the sorting step by making use of this index? Any suggestions from anyone? How can we actually accelerate the sorting step if we already have an index? Then we don't have to do the scan, right? We don't have to scan the entire table and do this merge sort. If you already have an index, how can we actually make use of the index to accelerate the sorting step? Yes? In the scan to the label? Exactly, that's correct. So the idea here is to simply retrieve the tuples in the desired sort order by simply traversing the leaf pages of the tree because the tree has already sorted all the tuples in the desired sort order, right? So there are two cases to consider here. The B plus 3 can be either clustered or unclustered. In a clustered B plus 3, the order of the data is same as the order of the sort order. So let me illustrate it here. So in the case of a clustered B plus 3, it's trivial. We simply have to traverse to the leftmost leaf page. We traverse to the leftmost leaf page and we then retrieve tuples from all the leaf pages and this essentially gives you all the tuples in the desired sort order. So if there is already an index in the database system, you can just make use of it instead of doing external sorting. So it's always a good idea to use a clustered B plus 3. However, if the B plus 3 is actually unclustered, in this case the ordering is not the same as the one that we need. Here if we actually chase each pointer to the actual page containing the data, we will be doing a bunch of random axes and in general we will be doing one IO per data record and this is a bad idea because you will be doing a lot of random disk axes. So if you have an unclustered B plus 3, it might not be a good idea to use it for the sorting operation. So that's the takeaway. Other than any alternatives to sorting, what if you actually don't need the data to be ordered? Let's say we want to form groups to satisfy a group by clause, but we don't actually need the ordering. Another example might be you want to remove the duplicates in the table, but you don't care about ordering. All you want to do is get rid of the duplicates. Can you actually remove duplicates without sorting the entire table? Yep, you learned about hashing in the earlier lectures. Just use hashing. If you don't care about ordering, you can just use a hash table to do unordered data access. So the idea here is that just construct a hash table and then remove duplicates by hashing into the table and making sure that we only return one, one tuple for each entity instead of returning multiple duplicates. And as we covered in the earlier lectures, hashing can actually be computationally cheaper than doing an explicit sorting operation. So hashing is an alternative to sorting if the query actually doesn't care about the ordering. In that case, you can just simply make use of a hash table instead of actually sorting the data. If you wanted to use hash group 5, if your groups were really, really large, wouldn't you end up having lots of chain in your hash table? That is correct. So it depends on the query. The query optimizer, which we will look into in the next week's lecture, actually decides when to actually do sorting or hashing depending on the query. It's a very good question. So to summarize sorting, we just show that the goal of external merge sort is to minimize disk IO. And in the first pass, we produce sorted runs of size B and then in the later passes, we do a recursive merge to construct the sorted table. So it's a recursive dividend-conquer strategy. And in the next week, we will be illustrating how the query optimizer actually picks a sorting or hashing operator depending on the ordering requirements in the query plan. So if the nodes higher up in the query plan actually require the data, the input data to be ordered, then it makes sense to do sorting. If not, then just do hashing. And again, you have to take into consideration other factors, like the number of attributes that you want to group by and so on and so forth. Cool. So we just covered sorting algorithms. So in posters, for instance, they use quicksort and external merge sort. If the data completely fits in main memory, they just do quicksort. And if it doesn't fit in memory, then they make use of external merge sort. So external merge sort is a frequently used algorithm in most database systems. Any questions on sorting? Soaring algorithms so far. Cool. So let's then move on to the next part of this lecture. So in the next part, we will be focusing on join algorithms. Join is another frequently observed operator in queries. And again, this has to do with the relational model. The relational model inherently avoids unnecessary repetition of information if possible. We don't want to store the same data several times. That's why you try to make use of higher normal forms. And the high level idea here is that we decompose tables using the normalization theory in order to avoid unnecessary repetition of information. But you still may want to construct the original tables that you essentially decomposed. So you might do a join on these decomposed tables in order to reconstruct the original table. So join operator is commonly used to reconstruct the original table and we don't want to lose any information by doing the join. So join is frequently used in database systems and in this part of the lecture, I'll be presenting a whole set of join algorithms that can be done to support this operator. And for this, we will be using a siling database to illustrate these algorithms. So anybody here into siling? If not, you should consider joining a siling club. It's a fun activity. This is from the Hoofer siling club at Wisconsin. I guess Andy was a member of this club. So we use this data set for presenting the join algorithms. So we have two tables here, a list of silos and a list of reserves. Ignore the actual attributes. All you need to focus on is the first attribute, the SID, which we will use to join these two tables.  So each tuple has some size and there are, let's say, some number of tuples in the page because the page size is fixed to be 4 kilobytes. So the main point here is that in the first table, there are 500 pages in the silos table. So we will represent this by n. So n is 500. And then because there are 80 tuples per page, we represent this by PS. PS is 80. In case of the second table, this is a bigger table. So it has 1,000 pages. So here m is equal to 1,000 and the number of tuples per page is 100 for the reserves table. So we have these two tables and we want to do a join. I will first try to distinguish between joins and cross products so that we make sure that the problem is very clear. So R join, yes, is a very commonly seen operation and it therefore must be carefully optimized. You often do joins. However, sometimes people actually do R cross S, which is the cross product or the Cartesian product, where you basically end up joining every pair of tuples in the two tables and this cross product can be really, really large because it grows exponentially. So therefore, a very trivial algorithm to do joins might be to first construct the cross product and then do filtering.  That's one feasible algorithm but it's very expensive because constructing the cross product is computationally inefficient. So that's why we need better algorithms. I will show that there are actually many different algorithms that you can use for supporting this join operator and also that there is no particular algorithm that actually works well under all scenarios. So you need to pick different join algorithms for different scenarios and this is where the database system technology gets very sophisticated. So in today's lecture, we'll be covering four join algorithms. The hash join algorithm will be covered in the next lecture by Andy. So the first three algorithms are different variants of the nested loop join algorithm. That's the simple one, that's the block one and then that's the index-based one. And then we will also cover the sort merge join algorithm which is very closely related to the external merge sort algorithm. Before we move on to the algorithms, I will set up the problem so that we can do some kind of Ivo cost analysis. So assume that the first table contains m pages. So the m pages in the reserves table and then PR represents the number of tuples per page and m is the total number of tuples in that table. In the same way, there are n pages in the silos table and then PS tuples per page in that table and totally there are n tuples, n tuples in the silos table. So to keep things simple, we will focus on the simple natural join and we will consider more complex join conditions later. The cost metric in all these algorithms is the number of disk IOs and if you minimize this, you can speed up the algorithm significantly because that's the high pole in the 10th. We will also ignore output cost, the outputting the actual joined table for simplicity. So we only focus on constructing the join table in memory. So here's an example for a join query, the simplest one probably. So you just join these two tables on their common attribute, the cellular ID attribute. So assume that we don't know anything about the actual tables and let's also assume that there are no indices on the tables, particularly on the SID attribute. So how can we execute this join operation? So we will start with a strawman algorithm, the simple nested loop join. This is the simplest thing you can do. So you have these two tables R and S. What you can do is you just have a couple of loops, you iterate over each tuple in the first table in R and then you also iterate over each tuple in the second table S. And then for each pair of tuples, you evaluate the join condition. In this case, it's basically you check the SID of the first tuple R and then SID in the second tuple S. You check if these two values are same and if it's same, then that's a match and then you output a joined tuple. And this I will note that this algorithm is really slow and we will see why it's slow in a moment. So we refer to the table used in the outer loop as the outer table and the one used in the inner loop as the inner table. So in this case, R is the outer table and S is the inner table. So why is this algorithm bad? Exactly. So you end up scanning the two tables for every single tuple combination. So you end up doing a lot of scans. So in this case for every tuple in R, you end up scanning the entire S table once. So the number of disk access in this case is M for the outer table and M, the small M represents the total number of tuples. So for each tuple, you are scanning the entire S table. So S table has N pages. That's why it's small M into N. So you have those many disk access and that's really slow because you are doing it for every single tuple. So let's try to get some ballpark estimates. So we just plug in the numbers. There's M and then small M and N. So roughly we will be doing like 50 million IO operations when we run this simple nested loop join algorithm and even if you use an SSD, you will get the result of your query after like one hour. So that's not so good. It's very slow. So even if you end up swapping the tables, like using yes as the outer table, we change the equation a little bit because in this case N is the first table. So you swap M and N but still you end up with 40 million IO operations. So that's slightly better but still it will take probably an hour. It doesn't improve things significantly. So what assumptions are being made here? We essentially use a couple of buffers for streaming the two tables, R and yes, and we use one buffer for storing the output. So we basically again just make use of three buffers but in reality you will probably have more buffers. So anyone has any ideas for how to speed up this simple nested loop join algorithm? That's correct. You can optimize for page performance. How would you optimize for page performance? Okay. Yes, yes. Yep. So the idea is to actually amortize the IO cost by doing block-oriented access instead of tuple-oriented access. Because in this case for every tuple in R, you are scanning the entire table yes once. What if you amortize the cost by doing it for a group of tuples in R? So this is the basic idea in the next algorithm. The block nested loop join algorithm. So here instead of reading a particular tuple, you read an entire block. So you read a block from R and you also read a block from yes. And then if a particular pair of tuples match, then you output the join tuple. And you can see that unlike the simple nested loop join algorithm in the block nested loop join algorithm, we are scanning the table once only for every block in R, right? Not for every tuple.  You are cutting down the overall IO cost. So in this case, the number of disk access is m plus m into n where instead of small m, we are using capital M. So small m represents the number of tuples. Capital M represents the number of blocks. So this is a significant drop in IO access. Any questions about the block? Yes. So we will generalize this in a moment. That's a very good question. It could be a page. It could actually be multiple pages if you can actually make use of all the buffers, right? So in general, you can see this pattern, right? Instead of using just three buffers because main memory typically has more buffers, you are trying to make use of all the buffers. But we will start with a very simple algorithm and then generalize it to make use of all the buffers. Not really. It's just that we still were using three buffers earlier, but we just used a single tuple from the block we pitched instead of using all the tuples in the block. So the question is, do we actually require only space for storing the tuples in the previous algorithm as opposed to storing blocks in this algorithm? The answer is no. Even in the previous algorithm, because blocks are defined by the operating system to be like four kilobytes in size, you end up fetching the full block. Instead of that, you're doing something not so smart by just using the single tuple after going through all the effort to fetch that block from disk. Yes, you end up doing extra IOs in the previous algorithm because you're not making use of all the tuples you fetch efficiently. So the exception is the same block? Yeah.  That's what we are trying to cut down here, right? In the previous case, for each tuple in a particular block, you are fetching the block once. Here, you are doing it only once for the entire set of tuples. That's correct. So that's why you cut down the multiplication factor by the block size, essentially. In the previous case, it was small m into n. In this case, it's capital M into n. And capital M is like small m by 4k or something like that. So it's very significant. It's a very good question, and that's a nice segue for the next variant of this particular algorithm. What if the entire table fits in memory, right? What if one of the tables actually fits in memory? Things can be accelerated, right? So before we go to that, I think we should probably think about a particular optimization. So which table should be the outer table for this algorithm? Yeah. That's correct. So in general, the smaller table, it makes sense because you end up reducing the number of IOs by making this choice. So we will move on to a variant of this algorithm. So if you just plug in the numbers, right? If you make use of capital M instead of small m, you will see that we have significantly shrunk the total number of IOs. Earlier it was more like 50 million IOs. Now it's only like 0.5 million IOs. So you reduce the total time from 1 hour to 50 seconds. Okay? By just making use of the block abstraction, okay? So what if you have more buffers? What if you have B buffers and you want to actually make use of all of them while running this operation? So here one thing you can do is you can use just one buffer for scanning the inner table and one buffer for storing the output and use all the remaining buffers. In this case, it's B minus 2. So you use those B minus 2 buffers for scanning the outer table of the relation, okay? Outer table of the join, yeah? So that's... So instead of reading just one block from table R, you read B minus 2 blocks every single time because you have those many buffers, right? And then you use all that information in memory to do the join, okay? So because this algorithm uses B minus 2 buffers for scanning the table M, you significantly reduce the number of disc axes. Instead of doing M, capital M into N, you end up doing M by B minus 2 into N, okay? So in the prior case, B was 3, okay? So because B was 3, 3 minus 2 is 1. So earlier it was M plus M into N, okay? So in the prior example, we just made use of 3 buffers. Here we are trying to make use of all the B buffers. So you are using B minus 2 buffers for streaming the outer table. So you cut down M by B minus 2, okay? That makes sense. Any questions? Cool. Yeah? So you can actually do that, but I don't think it's any different from this particular algorithm. This is equivalent to that. You can actually use multiple buffers for scanning the inner table as well as the outer table, but it's equivalent to doing this. So an extreme case would be when the entire outer table actually fits in memory, right? If you can actually pull in the entire table into memory. So in this case, B is greater than M plus 2, okay? So if the entire table fits in memory, you end up with just M plus N, okay? Because you can fetch in everything in a single axis. You can fetch the entire outer table into memory with a single axis. So you can see that if it's basically B minus 2 is equal to M, right? So M by M is 1. So that just disappears. That term disappears. So you will end up with just M plus 1 into N, which is basically M plus N. So this is an extreme scenario where the entire outer table completely fits in memory as he just mentioned. So in that case, you can finish the entire operation with just like M plus N IOs, okay? So in this case, it will be like 1,500 IO operations. Yes. Can you repeat the question, please? That's the B is more than M plus N. Okay. So if every B is really, really big, if you have both tables in memory, I think you can, yes, you can further simplify the algorithm, but you will still need to bring it into memory, okay? So you will still need to bring that into memory. So this is like a lower bound. Unless if it's already in memory from a prior operation, this is the lower bound. This is as good as it gets, okay? That's a good question. You will need M IOs to fetch the entire table because in a particular IO, you can only fetch one page, right? So you will need M IOs to fetch the outer table and then N IOs to fetch the inner table and then everything is in memory. So you don't need to do any more IOs. We are not worried about like outputting it to disk, okay? So everything can be done in memory. So the number of IOs would be M plus N. So this is like the best case scenario, okay? And this is basically a special case of this general algorithm, general equation, okay? Yes, yes. To simplify things, as I mentioned earlier in the lecture, for all the algorithms, we are not focusing on the cost to output the join table back to disk. We are only focusing on the input cost. To keep our analysis simple, yes. So that's the general idea of block nested loop join. So we basically make use of the block abstraction in this algorithm and we try to make use of all the buffers in memory to speed up the join operation, right? So this all makes sense. These are two commonly used algorithms of the nested loop join, okay? But the thing is these algorithms can actually perform bad compared to like if you have an index, right? If you actually have an index, then doing a sequential scan is very expensive, right? Why not just make use of an index? So the reason why these basic nested loop join algorithms are not so good is because for each tuple in the output table, you end up doing a scan over the inner table, right? And you can improve things by making use of the block abstraction or using multiple buffers. But at the end of the day, you still do a scan. And this is a sequential scan. But what if you actually have an index? If you actually have an index, how can you accelerate the join operation? Any suggestions for how we can accelerate this operation using an index on exactly, exactly? So you can replace the sequential scan over the inner table by an index scan over the inner table, right? If you have an index over the inner table. So you replace the inner loop which was doing a scan. Instead, you just do a single index probe to find if this table actually has a match, okay? So here you use an index to find the inner table matches instead of doing a scan. And we can actually make use of an existing index if it already exists or even build one on the fly to accelerate the join operation. And this is the reason why you might actually want to use an index nested loop join. Yes, exactly. So to keep things simple, we will not focus on the uniqueness aspect. But in general, you can just do a single index probe and you will be guided to the exact tuple if you have an index. So that's why indices can greatly accelerate the nested loop join algorithm, okay? Because you are getting rid of the sequential scan over the inner table. And instead, you are just doing an index lookup on the inner table, okay? So in this case, as I just mentioned, for each tuple in years, you are trying to figure out if the i-th tuple in R is equal to the j-th tuple in years and this is an index probe, okay? So instead of actually doing a scan, you are just doing an index lookup and that's logarithmic as opposed to linear in time, right? So the number of disk accesses in this scenario is capital M plus small m into C where C is basically the index lookup cost and C tends to be log M, okay? Log capital M, okay? So this is much smaller than doing a sequential scan over the entire inner table. So here's a summary of all the different variants of the nested loop join algorithm. So you have the simple one, the log one and the index based one, right? So the general idea here is to pick the smaller table as the outer table, okay? And then you try to buffer as much of the outer table as possible in memory and lastly, you try to loop over the inner table or you try to leverage an existing index to figure out matches in the inner table, okay? So that's a general design of the nested loop join algorithms. So any questions on nested loop join algorithms? Yes, sure. Okay, I will repeat it, yeah. So why is this cost M, capital M plus small m into C? So C is basically just the index lookup cost. So it's basically log N, capital N, okay? N is the number of pages in the inner table. So for each tuple in the outer table, which is M, you are doing an index lookup. So that's C. Yes, yes, yes, yes. You assume that the index is residing on disk and you still have to do logarithmic access as you traverse down the B plus tree to figure out the index match. But instead of capital N, it's basically log N, okay? That's a good question. Any other questions? So the question is, this assumes that you already have an index, right? Or you build it on the fly. So that's another interesting choice for the optimizer, the query optimizer, whether to build an index to ascertain this operation or only use this algorithm if an index already exists. So sometimes it actually makes sense to build an index just for ascertaining this operation. But oftentimes you probably use this join algorithm, this index nested loop join algorithm only if an index already exists, okay? Because otherwise you will also have to factor in the cost for building the index in this equation, right? And that can be like high as well. So index nested loop join is often used when you already have an index on the inner table on the join attributes. Good question. Yeah, I think you assume that M is already like, you are doing a scan over the outer table, right? So that's what is factored by capital M. And then once you have the outer table pages, you are doing an index lookup for each such tuple. So that's why it's M times C. So that's factored in. So this represents all the IOC will need to finish the join operation. Cool. So that summarizes the three different variants of nested loop join algorithms, okay? So we will next move on to the last join algorithm that I will present in this lecture. So it's the sort merge join algorithm. So the sort merge algorithm is actually very related to the external merge sort algorithm that I presented earlier in the lecture. So here also you have like a couple of phases. So in the sort phase what you do is you first sort both input tables, okay? You sort capital R and capital S and then you scan the two sorted tables in parallel in a merge phase and you end up constructing the output table, okay? So you have two different phases, the sort phase and the merge phase and this algorithm also makes sense only when the tables are already sorted on the join attributes, okay? So if it's already sorted on the join attribute then this is a nice algorithm that you can leverage instead of doing nested loop join, okay? So I will present that in the next few slides. So when is this algorithm actually very useful? So it's generally used when one or both the tables are already sorted on the join attributes or if the output of this operation should be sorted for operations higher up in the query plan. If you need ordering in the output or if one of the two input tables is ordered then it makes sense to use the sort merge algorithm, okay? So those are the scenarios where you will often end up using this as opposed to a regular nested loop join, okay? So you're leveraging the ordering in the input tables. If it's already ordered then it makes sense to use this algorithm as opposed to blindly just doing a nested loop join, okay? So sorting can be achieved either by doing an explicit sort step, okay? You actually use the sorting algorithm that we just presented earlier in the lecture like external merge sort or oftentimes you just use an index on the join attribute so that you get the tuples in the desired sort order, okay? So let me illustrate this algorithm with an example. So again we are going back to our sailor database example. So you have these two input tables, okay? The sailor table and then the reserve table. The first thing you do in sort merge join is you sort it, okay? So here the join attribute is r.sid equals s.sid. So that's the join predicate.  So you do the sorting in the first step, okay? So now if both input tables are actually sorted on the join key then you want to do a join, right? Because these tables are already sorted, you can just do a merge, right? You can do a merge of these two sorted tables to construct the join table, okay? To construct the output table, okay? So you are making use of this ordering property in this algorithm, okay? So I will note that you don't have to do this explicit sort step if there is already an index on one of these tables on the join attributes, okay? So the next phase of this algorithm is to do the merge, okay? So you have these two input sorted tables. How do you merge them to construct the output table? So you start with scanning these two tables in parallel, okay? And you output them whenever their sid matches, okay? So you basically see that for each pair if it doesn't match you just keep moving the two pointers forward in the two tables and you eventually end up merging both tables, okay? So you basically scan these two tables in parallel and you output these tuples that match so there is no multiplication involved here so it's not m into n in this algorithm it's basically m plus n because you are just doing two iterations one over the first input table and another one over the second input table, right? So that's the nice thing about this algorithm.  So as I mentioned in this case the number of disk access is basically two parts the sorting cost is basically m log m at a high level basically m log n and then there is n log n we also add in log b to account for the log aspect but in general it's m log m and n log n and then there is the merge cost m plus n because you are just scanning these two tables in parallel, okay? So there is the sort cost and then there is the merge cost oftentimes the tables are already sorted, right? If it's already sorted or if you have an index then you can skip the sort cost and then you only have the merge cost remaining, right? and that's as small as it gets as I discussed earlier in the lecture m plus n is as small as it gets you are just scanning the two tables, okay? So that's why this algorithm is often used when the input tables are already sorted so any questions about the cost equation or the algorithm in general? Cool So let me illustrate the sort merge join algorithm with actual numbers so let's say we have like 100 buffer pages, okay? So you can essentially sort both the tables using these buffer pages in just two passes so the overall cost is like 7,500 I.O. operations and if you take a regular off-the-shelf SSD if you have like 0.1 milliseconds per I.O. then the total time taken to run this algorithm is basically 0.75 seconds, okay? So it's really, really fast, okay? compared to, especially compared to the simple nested loop join algorithm, okay? So what's the worst case scenario for the merging phase of the sort merge join algorithm? The worst case happens when the join attribute of all the tables in both relations contain the same value, okay? In which case you are basically in a bad shape because you will end up doing M into N instead of M plus N because you keep like iterating over things for every pair, right? So this is a worst case scenario and I think at this point I should channel Andy here so Andy wants to like clarify that you don't have to worry about this case this is very unlikely in practice so don't worry about this, yeah? So I would actually characterize it as just being an aspect of the data distribution in general you tend to not have the same value for the entire and yes it also has a relationship with normalization theory you would probably then have a layer of indirection if most of the values are the same, right? You will probably have a layer of indirection so that also helps What is the extra? So the extra sort cost is the cost of actually sorting these two tables if they are not already in sorted order, right? So this M into N just represents the merge cost, yeah? So in the worst case this can get as bad as nested loop joint that's correct, yes but in the common case it's much better in practice especially if you already have a layer of indirection especially if you already don't have the sort cost Yes, so basically the underlying assumption is that these tables are already sorted so in which case this is a much better algorithm in the common case as opposed to vanilla nested loop joint algorithms, yes? Good question, yep? Yeah, in this scenario you can't do anything better I guess Yeah, this is like a worst case scenario and this is like a trick question this normally won't happen in practice that's why I was telling Andy, right? So you don't have to worry about this this is unlikely, okay? So that brings us to the conclusion for all the joint algorithms so this table summarizes all the joint algorithms that we have focused on so far the three different variants of the nested loop joint so you have the simple nested loop joint algorithm and then the block nested loop joint algorithm and then the index nested loop joint algorithm and we also covered the sort measure algorithm which can be used when the input tables are sorted so the IO cost is, as you can see it gets better and better across these different algorithms so we started with a really really slow algorithm which could take an hour to join this joint query when you use the simple nested loop joint algorithm but when we make use of the block abstraction we significantly drop down the time from one hour to like 50 seconds and that makes a big difference for the users of the database system, right? because now they don't have to wait an entire hour it's more interactive, okay? and then things get even better with if you have an index, right? you actually have an index on the input table you can do index probes as instead of doing a sequential scan over the inner table so then you get log n instead of capital N, okay? so that again shrinks it further to like 20 seconds and then eventually we have the sort merge joint algorithm if the tables are actually already sorted then this is a really nice algorithm that we can actually use to again construct an output table that's also sorted which can be used by other operators higher up in the query plan so with the sort merge joint algorithm you can run it in like 0.75 seconds I will note that all these numbers are basically just orders of approximation so it depends on the actual technology and the size of the tables but this will give you a rough idea of why it is actually important to optimize the design of the algorithm, right? you are decreasing the query execution time from an hour to like less than a second and that can make a big difference for the end users of the database system where would you want to give this goal into the query optimization part somewhere so in what case would the change in time from a block nested loop joint to an index nested loop joint not be worth it like currently it's just halving the time but if you look at it from like the birth value I see so your question let me paraphrase it so you can clearly appreciate the drop in complexity between the simple and block nested loop joint algorithms because you are cutting down time significantly but both the block and index nested loop joint algorithms are like 50 seconds, 20 seconds does it matter as much as the gap between 1 hour and 50 seconds, right? so I guess the answer to that question is it depends on the actual size of the data set so these numbers make sense for like small tables but if you are joining really really big tables then it does matter and especially if you already have an index why not make use of it, right? so in those settings the gap between these two different algorithms is more accentuated it's more amplified, right? but in general I think optimizing the joint algorithm is very important because it's very often used in actual queries that we see in the real world and I have heard that some queries can even like end up joining like 100 tables or 1000 tables in the real world so optimizing the joint algorithm can go a long way in improving the performance of the application especially as the number of joint tables increases any other questions? cool, so this summarizes all the different joint algorithms that we covered in this lecture I will quickly move on to a very different idea this is not related to joint algorithms this is related to joint types so SQL actually supports different kinds of joint types there is outer join, inner join semi join, anti join, lateral join and etc so these are different kinds of joint types so this picture actually represents some SQL join types so for instance in the first example on the top left you are doing a left outer join of two tables A and B and then in the middle one you are doing an inner join so you only get the set of tuples that are present that have matches in both A and B so we have these very different kinds of join types but we refer to them as join types these are different join operators that the user can actually make use of we implement these join types using different join algorithms so make sure that you can grasp the difference between join algorithms that are used inside the database system and join types that are used by the end user so let's do a case study so if you look in the Postgres database system it actually employs internally a state machine if you will to track the state of the join algorithm so we narrow down on one specific join algorithm the merged join in Postgres this also holds for other kinds of join algorithms like hash join so in each join algorithm you have a state machine to keep track of the state of the algorithm so that you can support all these different kinds of SQL join types so the state transitions in these algorithms depend on the join type so in this picture you can see that on the right side we have a bunch of states and these states have got to do with whether it's an inner join or outer join et cetera so this is how it's done in the real world and you do this for each join algorithm so to conclude there are many join algorithms that are often used in database systems and this actually is a nice illustration of the sophistication of the technology underlying database systems as a developer if you are asked to join two different tables you would probably start with the simple nested loop join it took several years of refinement to come up with better join algorithms like the sort much join algorithm or the index nested loop join algorithm and reimplementing all these algorithms in a new software system can take several many years so this is where it makes sense to actually use the database technology as an end user instead of reinventing the wheel inside your own custom user application so join algorithms illustrates this sophistication of the database technology and as we discussed during the lecture picking a join algorithm is actually very challenging for the query optimizer and at a high level a few heuristics are often used one such heuristic is that when the selectivity is actually very small that means that the number of tuples matching the join predicate in the outer table is very small you end up doing an index nested loop join and similarly when you are joining whole tables you often end up doing a sort much join but I am kind of waving my hands here things are actually tricky because you have so many different join algorithms that can be used for running a particular SQL query so the query optimizer has to make all these decisions it has to decide which join algorithm that should be used for executing this particular query and that's a very very interesting problem and this whole area of research is called query optimization so stay tuned for more details in next week's query optimization lecture where we will illustrate how the query optimizer makes these choices about which algorithm to use which join algorithm to use whether to use sorting or hashing and so on and so forth okay so to recap in this class I presented a couple of sorting algorithms and a bunch of join algorithms so sorting and join operators are often observed in SQL queries and these algorithms are often used inside a database system to support these relational operators in the next class Andy will actually cover another join algorithm the hash join algorithm and he will move on to more exotic join types like the semi-join, anti-join and lateral join that are actually often used in real applications so these are other interesting join types and we will cover them in the next class and we will also move on to aggregation algorithms and I will note that the sorting algorithm that I presented in the first part of this lecture often simplifies things for all these other operators like join and aggregation because if the input table is already sorted you can actually come up with more efficient algorithms for supporting these more advanced relational operators yep thanks


Welcome to All Hands Update, here's what's happening around your Navy today. Commander Task Force 70, Destroyer Squadron 15, and assets from the Japan Maritime Self-Defense Force, participated in Fleet Synthetic Training Joint Exercise 1671 at Commander Fleet Activities at Coastgate, Japan. FSTJ is a computer-based synthetic training that allows geographically separated units to integrate in a demanding virtual environment. The Boxer Amphibious Ready Group and Marines from the 13th Marine Expedition Air Unit arrived to the U.S. 7th Fleet Area of Operations. While Ford deployed to 7th Fleet, the team will participate in exercise Sangyang 2016 and conduct operations in support of the Asia Pacific Maritime Security Strategy. The Department of Defense announced the Military Spouse Employment Partnership Program has partnerships with more than 300 employers. The program connects military spouses, department organizations, and companies that have committed to recruit, hire, promote, and retain military spouses and portable careers. You can read more about what's happening around your Navy at Navy.mil.


in the Cascade Mountains of Washington, the Navy opens the world's largest radio transmitter. Its one million watts can flash a message around the world in a tenth of a second. A ship or installation anywhere on the globe can be instantly reached from Radio Jim Creek as the station is known. Built at a cost of $14 million, the station embodies the last word in radio transmission and is a giant step forward in the Navy's network of communications. The biggest thing that Jim Creek does is we provide operational command and control communications for Pacific Submarine Fleet. Jim Creek was established and built in the late 40s, early 50s. It was one of several sites across the U.S. that was surveyed for this specific type of an array between two mountains. It was the best spot out of about 13 other spots, so they chose it to build it here. At the time, it was the world's largest transmitter. Where we're at right now, we get coverage across the entire Pacific Ocean into the Indian Ocean and back across the continental United States and into the Atlantic. So because of where we're located at and the footprint that we get across the world, it's one of the reasons why Jim Creek is still around. The primary mission of Jim Creek is obviously the transmitter and the VLF broadcast, but it's also used as a secondary recreational facility also for active duty, retired and dependents. You can tour that the Interpretive Center has a diorama of the facility, has equipment that was originally installed here that has been taken out and preserved. It's a really interesting perspective if you wanna learn more, we'll be actually in here.


Director and staff of the Central Statistics Office appealed for public cooperation as they worked towards completing the enumeration process for the national population and housing census 2022. Sean Mathre, director at the Central Statistics Office, expressed that those certain challenges were anticipated. The magnitude of these challenges were unprecedented. Some of those challenges relate to the enumerator and supervisor dropouts. This is something that we did anticipate, but not to the extent that it has played out. This round of censuses, the extent to which the enumerator and supervisors have dropped out have been unprecedented. Mathre noted that the challenges faced by St. Lucia are not unique, as a recent meeting of Caricum chief statisticians and directors revealed similar issues in other territories, with some ascribing the socio-economic fallout of the COVID-19 pandemic as a key contributor. The situation with regards to the cooperation from the public seems to be one that is unparalleled. There is almost some sort of mistrust, negativity, some hostility out there exhibited by some of the respondents towards the enumerators. This is nothing new to pass censuses, but this time around, again, is the extent and magnitude to which this thing has been played out this time around. We all have a green shirt, and we all carry an ID card, and some of us have bags too. District manager for Grozely 3, Stephanie Marshall Constantine, said to minimize resistance and build public trust that enumerators can be easily identified via t-shirts and IDs bearing the census logo. She lamented some of the difficulties experienced by some enumerators who reported insults and threats of violence directed at them, and even dogs being let loose in their direction. So I just want to encourage people to take part in the census. It's very important, very vital for St. Lucia. It's like the framework for all other surveys. That is the framework that will be used for other surveys. So I just want people to come out. At least if you don't want us to interview, you can just tell us the number of persons in the house so we can have a con for the population. Despite the challenges faced on the ground, the Central Statistics Office is projecting completion of the enumeration process by September 2023, as they employ agile methods to address the constantly changing variables. So we have been in a position or situation where we have to continuously adapt our measures and our approaches to attending to those challenging issues and to ensure that at the end of the day that the implementation of the census is a successful initiative and that the data that is collected, the integrity is intact, it's sound and reliable. Mathryn added that many of the districts including Viewfort have been completed. He had high praise for members of the public who have made themselves available to provide the information necessary to make the national population and housing census a success. I would also like to appeal to those persons who have not been enumerated or visited by our enumerators that when the respective households are visited by the enumerators that they cooperate and provide the information that is required. Individuals who have not been enumerated and would like to be part of the census are invited to contact the Central Statistics Office at 4681577 or WhatsApp 2873295. You can also visit the Central Statistics Office on Facebook and Instagram. For the National Competitiveness and Productivity Council, Glenn Simon reporting.


Our altcoin of the week this week is FunFair, because math. GM minus M squared equals MA. So let's break that down. What is the best way to get people engaged in something? Make it a game. How do you get even more people engaged? Give them an opportunity to win money. How do you get them even more engaged in that? Give them an opportunity to lose money. So game times money minus money squared equals mass adoption. The FunFair formula. FunFair is our altcoin of the week because it targets everyone. And I mean everyone. It is our altcoin of the week because of casino floors at four o'clock on a Tuesday. It is our altcoin of the week because sometimes mommy needs a break. And it is our altcoin of the week because of boredom, because of public transportation, and because everybody poops. FunFair claims to revolutionize the casino industry by harnessing the power of blockchain technology. This is the best idea since someone decided to revolutionize the peanut butter industry by harnessing the power of chocolate. The blockchain ensures that online casinos are not able to rig games in any way and players can then be certain that they are solely responsible for losing their own money and no one will ever again have to listen to their uncle claim that a game cheated him. In their introduction video on the website they claim to not only bring the technology to the table but also the games of their own. Jeremy Longley, the CTO, says that they will bring games that are fun, exciting, fast, and engaging. The company claims that their games will be fair and lower cost so players will be able to win more money. Our advice would be to hold, because whether you agree with gambling or not FunFair is hitting a target about the size of Vegas. FunFair's token is called fun. They are literally giving you the opportunity to win or lose fun.  How could you lose all that money? I don't know, I just had so much fun. But don't worry, you can always wild away your woes by playing some games on FunFair.


Well, in our session on country experiences, we had presentations from very different countries, from Asia, from Africa, from Latin America and Europe. And many of the countries had made plans already and strategies to pursue forest landscape restoration, reforestation and rehabilitation of degraded lands. The commonalities between these countries are that many actually struggle with stakeholder engagement with motivating stakeholders to join these activities, to provide incentives so that one can change from a more degrading land use to a better land use, to a more sustainable and also more diverse land use. So these kind of funding issues have been very, very important in all of the countries. The success of landscape restoration would be if today degraded forest would be productive in future, would produce more trees, would have a higher diversity or in terms of soil, the soil productivity would increase and science can measure that in terms of soil nutrients, in terms of surface run of water quality. Experts are interested in very specific ecosystem services and one can analyze them. Therefore it is so important to decide and to see whether landscape restoration is successful or not to do a proper monitoring and really to assess how the situation looked at the beginning and how it looks now after a couple of years of rehabilitation activities. These kind of monitoring systems are difficult to establish and they are not in place. So overall we know a few examples where landscape is now in a better stage compared to the past. Exmining areas have been rehabilitated, sites which have been degraded because of storms of even natural causes, they have been restocked with trees. This one can easily measure but soil quality, soil productivity, water quality is not so easy and takes a long time. So science is actually advocating for good monitoring, scientific monitoring to see whether restoration activities are successful or a failure.


What's up everyone? It's your girl Kenzie here from late in sports guards and I rip and last personal in the daytime here for Jeff And we've got two twenty twenty-two Panini impeccable football hobbies Good luck Jeff 2022 impeccable. Here we go. It's here up first. We've got to 50 Russell Wilson Broncos Christian McCaffrey 57 of 75 and And a to all that is 21 to 25 on the bronze First hit up. We've got Jets rookie Otto 57 of 75 Jeremy Rocker rocket rookie Otto Eagles rookie Otto to 35 Jordan Davis 26 of 35 and we've got rookie landscape Otto for the Lions 21 of 49 Jameson Williams such cool cards Jameson Williams We've got Niners Steve young 15 of 25 Nice Steve young. It's on card to and last sit here for you Jeff. We got Steelers dual relic Otto 58 of 75 Kenny Pickett Nice hit on the Jersey helmet relic To 75 Kenny P We'll mag that one. All right solid first box there box number two Good luck. Do you have a bar nice? Save the bar for last that card on the bottom there All right, we've got Elijah more 50 out of 75 and CD lamb one of one Here you go Jeff nice one-on-one parallel Cowboys Get this one in the mag for you as well extravagance patch Otto 12 to 75 Mason Crosby Just kick her card there Second kicker card Otto of the day. We pulled a Justin Tucker out of honors photo Got Texans rookie Otto 14 to 35 Derek Stingley Jr. It's one stick to and vaunted greats. We've got 30 of 50 Drew Brees Otto Nice breeze 30 of 50 We'll make the breeze soon making Dylan work for this last last one Raiders to 25 Delicious ink and loose the illustrious ink Bo Jackson We've pulled a total of three Bo Jackson autos today. This is the first football one the first two were both baseball Nice bow nose to 25 and Chargers we've got wow three of five Isaiah spiller Partial helmet logo and gloves. I believe That's pretty sick. That is really nice three of five spiller There you go, I'm not like that one, too All right last pack of the day and it's gonna be the bar card good luck Jeff Thanks again for hopping in It's gonna be a saw rookie on one tryouts fine silver six of 25 There you go nice sauce to 25 that will do it for your personal congrats again on all those hits very nice We'll get these right out to you


What's good y'all is your boros back again with another video So we're gonna check out why Shawn Michaels was the 90s most hated man in wrestling man We all know the stories of how Shawn was before he found God before he changed his ways or whatnot We know Shawn The stories of how he was pretty much he was he was difficult To deal with back in an in the early 90s or whatnot because he knew he was the guy at that time he knew he was one of the main attractions and He had a lot of pullback stage, but he also had a lot of controversy in beefs and issues So we're gonna check out why he was the most hated in the 90s, man This should be an interesting one. This is by stun by wrestling. I'm subscribed to him So make sure you guys go check him out. Appreciate all love support man. Let's get right into this one, man Should be good when y'all go subscribe to him, man Shawn Michaels was such an incredible performer. Nobody had provided WWE fans with so much entertainment in so many ways a devastating back injury in 1998 seemingly cut Michaels career short It was everyone's belief that Michaels time in the ring was over and the fans thanked him for his service Nobody ever expected Michaels to come back and yet he did when Michaels returned four years later He was like a brand new man. Yeah Incredibly Michaels would go on to provide almost another decade of career defining moments including WrestleMania matches and moments that will be remembered forever facts facts However, Michael's peers in the locker room have provided a different perspective on the man over the years By all accounts Michaels was like the devil behind the scenes during his first run in the 90s Causing trouble for everyone around him in this video We will take a closer look at why Shawn Michaels was one of the 90s most hated men in wrestling This this definitely sounds about right with many people's All right, man, man to check out some more of wrestling's most hated man It sounds like after he's wrestling debut Shawn Michaels was quickly paired with Marty genetic as the midnight rockers in 1985 The team absolutely thrilled at that look at the guns providing them with in-ring action They'd never seen before the tag team Synchronicity that the midnight rockers displayed in their AWA matches was Revolutionary and they set the pace for many of the tag teams that followed them in the years to come The boys were just a team on screen They would ride together and bunk together always talking about the business and how they could continue to innovate Matches reports were abound that the midnight rockers really did rock all night long And in the words of vergonia became tough to handle as their newfound fame Quickly went to their heads by day Michaels was thrilling in the ring and had an incredible mind for the business But by night he became a party animal alongside genetic He was out all night hunting for women and often getting kicked out of bars and totally trashing hotel rooms Michael's first visited Las Vegas in 1986 His cocaine habit also started that night Moment that he would later describe as the start of his downfall It would be an addiction spanning over a decade in an interview Marty genetic Estimated that he and Michaels were on the road for 300 days a year and they parted for 290 of those days Pat Patterson skated the rockers for WWE in 1987 their reputation was well ahead of them and according to Michaels They were treated accordingly by the boys already in the Federation's locker room We came into a pretty cold reception people had heard about us They thought we were young punks all of it was true for the first few weeks Michaels and genetic tried to keep a low profile backstage But once they were comfortable they joined Davey Boy Smith and Jim Powers in the bar after a show Late into the evening Jimmy Jack funk appeared and started literally chewing glass and getting in the boys faces Eventually Michaels broke a bottle over his own head to try and shut bunk up It later transpired that genetic had been hitting on a girl in the bar that Powers was Later claimed that Michaels and genetic ended up tearing the bar up and caused a huge ruckus Whether that was true or not It was enough for Vince McMahon to backtrack on his decision to hire the rockers in the first place Yeah, it was a devastating moment for both men having so briefly reached the top of the mountain They ended up getting fired and going back down to the bottom this time Wrestling in front of just 50 people for the continental territory in Knoxville, Tennessee Yeah, to know money Michael says that losing his job with WWE was devastating to him and he immediately entered a downward spiral of drug-taking and suicidal thoughts He said that the only thing stopping him from taking his life was the devastation that it would have caused his parents After the experience of being at the top of the mountain employed by Vince McMahon Ron Fuller's continental wrestling promotion was like being in the gutter each booking provided the boys with no more than $300 and genetic claimed that often they would accept payments in cocaine to top up their fee Eventually new continental armstrong reached the end of his rope with Michaels and genetic and gave them their two weeks notice The boys decided to quit right away and walked out the door leaving armstrong with a hole in his booking sheet Luckily for the Rockers They still had some name value and so were invited into the Memphis wrestling territory who had connections back with the AWA As they started to pick their game back up in the ring Michaels could see that the writing was on the wall for the AWA as the industry was rapidly changing and they needed to get back To WWE Vince McMahon was feeling generous in 1988 and gave Michaels and Ginetti a second chance But they were told that this was their final opportunity in the ring The Rockers got back to their highly motivated best and the fans immediately took to the young babyface tag team The Rockers never reached the top of the tag team Maintain however and truth be told Ginetti was starting to irritate Michaels Most of the fans at the time didn't realize that tensions had been brewing between the men for months That bad sentiment came to a head one night in a hotel lobby blood was spilled very publicly Exactly how the fight started between Michaels and Ginetti is still a mystery However, the fight was brutal enough for the police to be called Yeah, I didn't know they had it luckily Randy Savage arrived at the hotel just in time to vage for the pair and To sign some autographs for the cops Otherwise, it would have been a night in the cells for both men on screen The fans were shocked and disgusted by Michaels as he threw Ginetti through the barbershop window Classic moment. He'll Michaels was adamant that he wanted to be a single Superstar and to not have to share the spotlight with anybody else in hindsight I see the beginnings of these negative traits in Michaels There would be evidence all the way through his career in the 90s He was a wrestler who was a selfish and arrogant as he was gifted a drug addict Who still managed to deliver gold every time he laced up his boots night after night while Michaels was rid of Ginetti He still recognized the importance of having someone's back and more importantly Having somebody to watch his back in a business that was notoriously volatile Damn that was crazy the clique were a notorious backstage faction formed of Michaels Kevin Nash Scott Hall And Paul Levesque aka Triple H Michaels was glad when Hall was hired to play the part of Razor Ramon as they'd both been buddies while working in the AWA Kevin Nash walked through the door next on the recommendation of Michaels to play his on-screen bodyguard Hall and Nash had been friends since their days as lower midcarders in WCW and the trio hit the road together Sean Walkman joined the group after Nash witnessed Walkman playing pranks on other members of the locker room backstage Nash immediately knew that Walkman was the kind of guy that could hang with the clique The guys were fans of Triple H ever since they saw him wrestle on WCW Saturday night And they became even bigger fans when they found out he didn't drink or do drugs So he could be the designated driver and a bank carrier number one Sean Michaels needed and it was even better because he was the clear leader of the pack The man had strength in numbers backstage and began leaning on Vince McMahon to influence booking decisions in their favour But it cost the card McMahon had little choice but to acquiesce to their many demands Shane Douglas said it wasn't just the guys on the bottom rung I know that there was this very strong reaction among some of the bigger names in the dressing room They didn't like what was going on with the clique They saw it as a threat to their ability to make money to their ability to have talent to wrestle and oppose off of Shane Douglas himself was affected by the clique during his brief run with the company in 1995 Sean Michaels and Douglas were supposed to wrestle for the Intercontinental Champion shipper in your house for however Michaels had to vacate the title before the match due to a Supposed injury Douglas would receive the Intercontinental title via forfeit however He would be embarrassed shortly thereafter when Scott Hall would force him to defend the title Hall went on to defeat Douglas for the belt making him the shortest ever After just 20 minutes Indications of the clique using their power backstage to get what they wanted The Douglas wasn't good enough to compete in the federation very early on this Humiliation on screen was the final nail in the coffin two months of backstage pressure and Politicking to get rid of Douglas on behalf of the clique and sure enough It wouldn't be long before he was out the door and back in ECW Douglas would see on this treatment by the clique for years to come even using the experience to feed back into his character In ECW the real reason that Michaels had to forfeit the Intercontinental title to Douglas was due to him being needy beaten Unconscious during a bar fight in October 1995 on television An angle was concocted whereby Michaels had suffered a concussion The clique had been momentarily split up due to half the roster going on a European tour and the other half staying in the US Michaels and Waltman decided to hit the bar one night and Michael started flirting with a girl on the dance floor So the girl turned out to be the girlfriend of a soldier who took offense to Michaels hitting on his woman Of course things quickly escalated and Michaels ended up getting knocked out after getting his head Slammed in a car door damn The age would later say a million times where we'd been out places and sure would do the exact same thing But I was there Kev was there Scott was there, you know, it's different Nobody in the locker room at the time was surprised that Michaels ended up in this situation Hotel next door neighbor Sunni didn't ask any questions as she escorted Michaels to the local emergency room Sunny whose real name was Tamara Sitch was WWE's number one woman in the mid 90s now considered to be the Federation's first real diva She debuted alongside real-life teenage sweetheart Chris Candido as fitness freaks the Body Donners in 1995 Sitch's act of kindness in taking Michaels to the hospital led to them getting closer and closer She would check on him every day Michaels didn't care about the feelings of Sitch's actual part of Chris Candido as the couple made their affair very Oh, I'll be honest with you Shawn Michaels was Was the original Wrestling city boy, bro Shawn Michaels was the first I'm pointing it out. I'm claiming this now Shawn Michaels was the very first Wrestler that was a city boy before city boys was a thing as exactly was Shawn Michaels was bro He didn't care if you had a chick. He didn't care, you know If you was married and it didn't matter if Shawn wanted you he was gonna shoot for you You know what Shawn actually deserved this hoodie right here. I'm wearing Let me let me move the screen so y'all can see it move the mic You see this hoodie you see this hoodie. It's a toxic hoodie. He was the embodiment of toxic He should have been wearing this for sure Let's get back to it man. Vs to everyone around them They even went on vacation together with Sitch showing off the photos of them both in Jamaica right under Candido's nose according to Sitch she and Michael's would even have sex multiple times and at WWE shows While click members acted as a lookout in the interim many wrestlers have stated how distraught Candido was during that time period and her Michael's didn't care what anyone else thought about the situation In a later interview just credible said the click didn't take any prisoners They didn't hold anything back. I know Shawn Michael's was notorious, you know And I love Shawn to death, but he was and he'll tell you himself He was a prick due to his attitude Michael's was almost impossible to deal with face-to-face Nobody could judge Michael's mood on a day-to-day basis as his alcohol and drug consumption reached almost out of control levels He's life being ruled by uppers and donors not even Vince McMahon himself was safe Vince Russo would later say Shawn Disrespected everybody. I never saw anybody getting Vince's face more than Shawn not even close I think Shawn to this day is the only person. I've seen like scream at Vince and get away with it Yeah, confirmed Kevin Nash Michael's just did not care yet Friends around him Shawn out of his ass nobody could doubt that he was the best wrestler on the roster if not in the entire world Well, that's apart from arch nemesis Brett According to Hart always became a member of the clique back in 1994 when WWE was on the tour of Germany Hart was still on good terms with Michael's haul and Nash at the time and the men were drinking in the hotel bar The men broached the idea of a wrestler's union to Hart whereby he would be the leader and would take the group's ideas to Vince McMahon This was before Michael's had an established relationship of his own with the boss And they knew that Hart already had that relationship in place and therefore could influence McMahon's decisions at the time Hart declined to the offer He was well aware that all three men's social habits didn't align with his own and he personally felt that he would get no Benefits from unionizing with them as far as Brett Hart was concerned in 1994 He was the golden child. Yeah, and his push had already begun Yeah, as the 90s wore on animosity between the men would grow into complete Hatred for one another in many ways their career had ran in parallel Both debuting as smaller guys Wrestling a more technical style in their respective tag teams in the 1980s Hart was always slightly ahead of Michael's in his career winning the WWE Championship first The WWE had long been known as the land of the Giants thanks to the likes of Hulk Hogan and Andre the Giants in the 1980s Yet the decision to put the belt on Hart proved that Vince McMahon was willing to look at smaller More athletic guys to main event the company going forward. It was a big thing right there This was a breakthrough ever since he arrived in the company He'd been concerned that his height would hold him back So he certainly knew he had the opportunity to be world champion himself going forward at WrestleMania 11 Michael's faced his buddy diesel for the world Championship in the main event Michael's went into the match as the heel challenger But he actually emerged as a baby face and a losing effort thanks to his craved pleasing antics As far as Hart was concerned Michael's had gone into business for himself and decided to turn himself face Now they were both on the exact same trajectory They were both baby faces fighting for the same spot as number one in the company Bret Hart would go on to defeat diesel for the world title for a third time at the 1995 Survivor Series Hart felt really Unappreciated as the champion and he believed he's reigned to be transitional He believed that they were just waiting for the right time to give Michael's the belt and he was right WrestleMania 12 the two fan favorites clashed in an Iron Man match, which was won by Michael's After the match Michael's barked at heart to get the fuck out of his ring while he celebrated Wow later would say that he felt like yesterday's news to Vince McMahon and other members of management and Michael's and the rest of the clique had directly influenced them at the time Michael's gloated to heart that he was going to defend the belt against Scott Hall and Triple H and try and get one more main event between himself and Kevin Nash in an interview Michael said in my mind I can certainly believe that I have the arrogance at that time to think that I could dictate who I was gonna work with and In my mind, I can even believe me at that time Rationalizing that I'm taking care of my buddies and working with my buddies and not thinking in February 1997 Michael's vacated the championship on a special Thursday night episode of raw Michael's told the viewing audience that he'd suffered a career threatening knee injury and more Importantly, he'd lost his smile in reality Michael's has always insisted that he was both injured and at the peak of his drug addiction at the time But others have always maintained that he didn't really want to drop the belt to Bret Hart directly Which is why the injury storyline was put on the table instead That makes my heart would continue to oppose each other in terms of mindset right up until Hart's last day in WWE a comment in 1997 on an episode of raw in which Michael's Ironically insinuated that Hart had been having an affair with Sonny even resulted in a real backstage fight between the men I can see that been written and videos have been made about the Montreal Scrooge over the 1997 survivor series However, Michael's denied all knowledge of the plan to screw Hart out of the title for years It took a long time for the truth to finally come out about the events that occurred that night and Michael's was fully in on the job Yeah, Michael's would later say that he considered that night to be the lowest point of his entire career Yeah, Mark Calloway aka the Undertaker Recognized the danger that the clique represented very early on he could see the many ways that the clique were causing damage to WWE locker room morale was at an all-time low in the early 90s as the clique squashed everyone They didn't approve of desk. The fear of being picked on was Monumental among the wrestlers in the locker room as the clique didn't just rib the other talents They targeted them and ensured that their careers in WWE were short and totally miserable Calloway saw that the clique were trying to monopolize the best spots on the card for themselves and They were mostly succeeding Calloway formed a group of his own known as the Bone Street crew alongside Yoko Zuna And they carefully selected other wrestlers to join them based off their similar mindset While it has been later claimed that the group didn't ever come to blows with the clique Calloway's boys did try and keep them in check by the middle of 1996 However, there wouldn't be a clique left to deal with The curtain call was an incident that took place at a Madison Square Garden house show in April 1996 With Kevin Nash and Scott Hall leaving for WCW It was the end of the clique as they knew it to the fans the men in the ring were enemies when they decided to hug each other In front of the fans at MSG. They were breaking kayfabe. Yeah, Vince McMahon was totally furious Sean Michaels managed to avoid punishment for his actions considering he was one of WWE's biggest stars and got off the hook As Scott Hall and Kevin Nash were out the door all of the punishment fell to Triple H Vader was an exciting prospect in 1996 The agile big man had proved his worth by providing thrilling Contests in WCW and Japan and he held multiple world championships on his way to the federation Vader seemed like the remedy to WWE's lacking main event scene and surely his violent style would Compliment Michael's high-flying agility perfectly a match at Sub-aslam 1996 between Michaels and Vader for the title was the end goal and the feud was set up at the Royal Rumble in January of that year So far so good only problem was Michaels didn't like Vader and didn't want to work with him One rumor suggests that Michael thought that Vader didn't wash his wrestling gear often enough and was Unhygienic and thought that Vader worked too rough and believed that Vader would cause him an injury Yeah in ring style So the writing was on the wall for Vader's main event push during the match at Sub-aslam When Vader forgot to execute a planned spot coming off the top rope Michael's angrily yelled at Vader during the match due to the discrepancy After the match Michaels complained to McMahon as was so often the case at the time and McMahon agreed Vader would end up sliding down the card until he was finally released in 1998 His true potential was never seen in Wow Michael's politics towards his fellow wrestlers finally came to bite him in the ass for literally decades to come When he decided to pick on the rock during his formative years The Rock's massive rise to prominence came after the one first retirement at WrestleMania 14 in 1998 But that didn't stop Michaels from having a go in 1997 Michaels had originally Disrespected the Rock's grandmother during a show in Hawaii when he was a teenager and the Rock never forgave him for that Matters were made even worse when the Rock arrived in the company Brett Hart claimed in his autobiography That both Michaels and Triple H despised the Rock when he debuted despite being hampered by a Terrible gimmick the young Rocky my via was obviously gifted in the ring and Michaels was getting scared believing that he would take the place of his best friend at Triple H And so Michaels tried to force Vince McMahon to get the intercontinental title off my of here But the idea was nixed the Rockers never truly forgiven Michaels for his behavior And so that dream match has never come to fruition For sure Michaels there must have been a lot of soul searching done between his retirement in 1998 and he's returned to the ring in 2002 By all accounts he was a different man apparently having famed God and becoming clean from drinking drugs The man will always be a legend in the wrestling business in the truest sense of the word However, we shouldn't forget all of those he trampled over in order to get there This was a good one man. This was a good one man. I love this. This is this was This was fantastic. I love this video. Make sure y'all go subscribe to the homie stunned by wrestling I'm subscribed to I'm gonna link down the original video down below in the description So you guys can go check them out. This was very dope I didn't want to pause it too much cuz I wanted to you know, don't want to make this video too too long But this was very enjoyable very informative and it's crazy how much pool Shawn Michaels had back then where he could tell Vince. I don't think he's he got it in events And then he just slides down the card until eventually he's let go.  Let me know man. What's your favorite? We're not gonna focus on the negative. What's your favorite? Shawn Michaels HPK match of all time. Let me know down below But I appreciate all the love and support you guys have shown on the channel road to 150k. Appreciate your kicking me See you next one. Peace


In September of the same year, 1944, came Operation Market Garden, the largest airdrop ever attempted. An entire airborne army, British and American, was dropped into Holland just ahead of General Montgomery's steadily advancing British Army. Its mission was to capture certain strategic bridges over the great rivers of the area before those bridges could be destroyed by the retreating Germans. In the bitterest fighting of the entire war, the mission was only partly successful. However, the airborne divisions of World War II were not limited to an airborne role. They were also employed as regular infantry divisions. The 101st, in fact, achieved its greatest fame fighting as an infantry division at Bastogne. But fighting as an infantry division required somewhat different weapons and equipment than fighting as a strictly airborne unit. And this equipment had to be left behind whenever the division took to the air. Still, this concept of the airborne, with one foot in the air and the other on the ground, persisted until a very few years ago, until the advent of tactical atomic weapons. These weapons, of course, changed a good many concepts of warfare. Because of their tremendous power, doctrines which had been honored for generations were suddenly outmoded. No longer was it possible to mass large numbers of troops for an attack on a narrow front. Without were the great motorized columns of troops and weapons and supplies which clogged the roads of Europe during World War II. Out also were the large ammunition dumps. The elaborate harbor installations. All of the concepts involving these facilities would have to be modified or discarded. In short, tactical atomic weapons have forced modern armies to disperse. Battles henceforth will have to be fought by relatively small units having great mobility and firepower. That's why our airborne divisions recently have undergone a complete reorganization. The old airborne division, which had about 17,200 men, was referred to as a triangular division. That is, it had three regiments, each of which had three battalions, each of which had three rifle companies. The new division, which has only 11,500 men, is what we call a pentomic division. Instead of three regiments, it has five battle groups, each having five rifle companies, but no battalions. Each battle group is a self-contained unit with its own signal troops, engineers, and artillery. This means it is capable of operating entirely on its own whenever the occasion requires it. In other words, the new airborne division is designed to provide the greatest possible flexibility on an atomic battlefield. It is also designed to produce the greatest possible firepower. Even with one-third fewer men, the new division has much more firepower than the old, more firepower in fact than any division in the whole history of warfare. To achieve this added firepower, more men have been placed in combat positions and have been given better weapons, more automatic weapons, harder hitting weapons. And for a Sunday punch, there is the Little John, which is capable of delivering an atomic warhead to any spot in the local combat zone. And this tremendous firepower is always available. Every weapon in the division, as well as every other item of equipment, can be transported by air. Where the division goes, its weapons go with it. To make this increased air mobility possible, the Air Force has developed several new cargo planes. There is the C-123 Provider, for instance, which can carry either troops or heavy equipment. There is the C-130 Hercules, a turboprop plane, which is capable of carrying heavy loads for long distances. If there are no suitable landing fields at the destination, it can deliver its cargo by air drop, as mobility on the ground been neglected. While the large cargo trucks have been discarded, more of the smaller trucks and jeeps have been included. And a new and ingenious little vehicle has been added, the mechanical mule, which can make itself useful in a hundred different ways. All weapons in the division can be moved rapidly, wherever they are needed. And if the ground is too rough for them to be moved overland, they can be moved by air. In short, the new airborne has the firepower to overwhelm the enemy and the mobility to use its firepower to the best possible advantage. And in the roles which have been marked out for it, the airborne will need all of the mobility and firepower it can muster. In any large-scale war of the future, it can expect to do most of its fighting behind enemy lines. It may be dropped there to seize a key bridge or road junction and to hang on until the advancing ground forces can affect a link-up. It may be set down far back in the enemy rear areas with a mission of setting up an airhead. The airhead can then be built up by the serial delivery of troops and supplies until it is capable of supporting an entirely new offensive, just like a beachhead. The airborne may also be used to conduct a raid on a vital factory or airfield and then get out. Or it may be dropped to work with local guerrillas and to harass the enemy in whatever way is possible. But unless and until there is a large-scale war, the airborne's principal job will be to help prevent one. And this is how it might be asked to perform. Let us suppose that one of the smaller free nations is suddenly invaded by a neighbor known to be dominated by the Communists. This is an explosive situation. Allowed to develop, it threatens the peace of the entire world. It's flashed around the globe. To Paris. To London. To Washington. All branches of our government are alerted, including the Department of Defense. After a conference at the very highest level and consultation with the United Nations, it is decided that immediate action must be taken. The brush fire must be snuffed out quickly before it has a chance to spread. The airborne is selected to serve as the fire brigade. At Fort Bragg, North Carolina, the 18th Airborne Corps receives an urgent message from the Department of the Army. Two battle groups are ordered to prepare for immediate departure to the trouble zone. Key personnel must be alerted. A lot of things to be done and not much time in which to do them. The first transport planes are due to arrive at 0900. Since the Air Force has been on a standby alert for several days, it has already collected a number of planes at various spaces and has been getting them ready to take off at a moment's notice. At Fort Bragg, the airborne units waste no time. Both weapons and equipment must be ready so they can be loaded aboard the planes along with the troops. Soldiers living off post use every available means of transportation to reach their units. As the takeoff hour approaches, the troops fall in for roll call. As soon as the roll has been called, the various units start for the airport. Hours after receiving the alert, the first flight of Air Force planes arrives. Each plane is loaded according to a carefully worked out loading plan. Weapons and equipment go right along with the troops, often in the same planes. As soon as they are loaded, they take off. The fire brigade is on the way. In the country which has been attacked, American officers are conferring with the local authorities. They must first learn the exact military situation, then they must work out a plan of action. By the time the first airborne troops reach the arrival airfield in the objective area, the plan of operations has been completely worked out and is ready to be put into effect. The leading elements prepare immediately to make a parachute drop into the combat zone. They are loaded aboard planes which have been flown in from nearby bases, especially for this purpose. And their weapons and equipment are prepared so that they can be dropped along with the troops. Planes take off for the local trouble spot. As they near their destination, they drop down to the designated altitude and the men get ready to jump. Once on the ground, the paratroopers promptly roll up the stake. That is, they reassemble as quickly as possible. At the same time, they gather up their equipment. From now on, everything goes according to a prearranged battle plan. Each unit has been assigned a specific mission. One unit hurries off to seize a key road junction. Its mission is to block any further advance of the invading force. It's have been assigned to prepare landing strips which will accommodate the planes bringing in the main body of troops. With the small graders and bulldozers which were dropped with them, the airborne engineers begin carving out narrow runways. Not fancy, but good enough to handle the planes which bring in the rest of the troops and equipment. Strategic road junction, the invading force has been stopped temporarily. Fog is sent aloft to spot targets. The information is relayed and the artillery immediately takes these new targets under fire. Reports that enemy forces are beginning to retreat. Doctors move up. Their mission is to gather up men and their weapons and leapfrog them forward. With the helicopters, the troops can be set down behind the retreating enemy. Countrymen soon turn the enemy retreat into a route. The invaders army is scattered. Quantities of his weapons and equipment are destroyed. He ceases to be an effective fighting force. Battle sputters out. The war is over. A brush fire which was burning so briskly only a few hours ago has been snuffed out. What is even more important, it has been snuffed out quickly before it has had a chance to spread. Before it has had a chance to menace the peace of the entire world. The fire brigade has done its job. You have seen what today's airborne is and how it could be expected to perform in an emergency. However, since any potential aggressor is well aware of its firefighting capability, the chances of such an emergency arising are greatly reduced. In other words, our airborne has a strong deterrent effect. And that suits everybody fine. The airborne has no desire to race halfway around the world to put out a fire. But if one should break out, it's nice to know that the airborne is ready, willing, and able to take care of it. Now this is Sergeant Stewart Queen, your host for The Big Picture. The Big Picture is an official television report for the armed forces and the American people. Produced by the Army Pictorial Center. Presented by the United States Army in cooperation with this station.