Here we are in our QuickBooks Online test company file using the Accountant view as opposed to the Business view. You can toggle between the two views by going to the cog up top and switch the view on down below. We're going to duplicate some tabs to put reports in, which is a custom we'll do basically every time. We're going to go to the tab up top, right-click on it to duplicate it. Right-clicking on the Duplicate tab because we want two main reports, Balance Sheet Income Statement, Duplicate it. As the tab to the right is thinking, I'm going to go to the tab to the middle, down to the reports on the left-hand side, and we want the Balance Sheet, one of the favorite reports here, of course. Back to the tab to the right, let's go down to the reports. Again, this time the Profit and Loss, which can also be called the Income Statement. The P&L, I'm going to close the hamburger up top, and I'm going to change the range to the end of 2025. Let's go to 120125 to 123125, and this is where we will place our data. Running it shouldn't be anything in there. We have a nice clean report to start out with. Let's go to the tab to the left, close the hamburger, do the same thing on the Balance Sheet. We're going to go from 120125 to 123125, and I'm going to run that report. Nothing is there so far. Obviously, the financial statements are basically our end result, what we are trying to build from an accounting standpoint for our business. Let's go to the tab to the left. Now we want to think about the purchasing cycle. We're going to think about it as though we're on an on-ground kind of store, and run the perpetual inventory system that QuickBooks has designed, and then think about where the deviations are going to have to happen when we try to integrate data that's coming from like a Shopify or an Amazon type of situation. If we had an on-ground kind of system, let's follow the inventory. If I was to purchase the inventory, we may or may not use a purchase order. We might just be paying for the inventory with an electronic transfer, a bill, but let's go through the whole electronic transfer, which would be a check or an expense form. But let's go through the whole process here. First, when we request the inventory, we want to be able to track not only the dollar amount, but also the items. The way QuickBooks does that is we use the items. These would be like the inventory items similar to what you would put in like a Shopify store when your products into a Shopify store. Again, note that I'm not saying that we're necessarily going to put these items in our final system into QuickBooks online because that might be redundant. We might not need to do that, but we want to see how QuickBooks would normally do a perpetual inventory system so that we can see kind of the deviations we might make for a Shopify integration type of thing or any other kind of platform. So here's where the items are located. It's under the sales tabs. We're going to go to the products and the services on the right. Now, some of these you might have like hours in labor in here. Don't worry about any of these other ones right now. We're just going to add generic product one, two, and three. So if I go up top and I say I want a new and we're going to make inventory items. If there were inventory items, that means we're going to actually track the physical units in QuickBooks online. If it's non inventory, we're not going to be tracking the units. Now, if we were doing a full service perpetual inventory system, we would want to be taking an inventory item allowing us to have inventory reports. I'm going to give it a generic name, product, product number one. And we could have the SKU number. If we have one, we can add that. We can add a picture of the inventory and so forth if we wanted to track it, which is actually kind of cool if you're doing on ground kind of stuff. We can add it to a category so we can have groups of inventory and so on. I won't get into that in detail. Class tracking if we need quantity on hand. Now I'm going to start with zero on hand because we're going to purchase them and I'm going to say this happens at the beginning of of let's just say the beginning of 2023. Just so I make sure it's at the beginning point reorder point. This will be the point that we want to remind us to order more inventory. So then we've got the inventory assets. This is going to be the account that it's going to go to when we purchase the inventory. The description is going to be product number one and then the sales price. So this is going to be the amount that goes on our sales form. The the invoice type of form. Let's say that we sell it for $100. The income account is going to be a sale of product. So when we make an invoice or sales receipt, that's the account it's going to go to. And when we purchase the inventory, let's say we purchase them for $75. It's going to go to inventory when we purchase it and then cost of goods sold when we sell it in a perpetual method kind of similar to what would happen if you're checking something out of the grocery store. So we'll take a look at that in future presentation. And then we might have a preferred vendor who we buy the inventory from. All right, let's go ahead and save that. Let's and let's make a couple other ones kind of to match our little thing over here. So we might have another one product number two inventory product number two product two. And I'm just going to say that this is going to be quantity on hand to zero beginning of the year that we've got this other product. Number two and description sales price. Let's say this is a $200 product and we we purchased it at a cost of 150. Let's say cost to goods sold the expense account. Same thing looks good. Let's say save and new this time. Save and new and this this is going to be product product number three product number three. And we're going to say quantity on hand is zero. We're going to purchase these January okay reorder point description. Let's say this is for $300 and we're going to purchase it for 210. So we try to sell those ones. You pull them into the store with $100 item and then you upsell them. Okay. No, we won't get into sales strategies here, but we'll say save and close that one boom. So now you've got your items here. Now when I purchase when I purchase the item, it will be tracking not only the dollar amount we spent on it, but also the units. That's what we would be trying to do within QuickBooks something that we might not be doing if using an e-commerce kind of stores. We're looking here to full service accounting system within QuickBooks that we will deviate from with Shopify. So let's say that I would then go up and I say that we're going to buy the inventory. We would buy it possibly starting out with a purchase order of requests for the inventory and or we might just buy it basically with an expense form and electronic transfer or something like that. But let's do the full thing. Let's say we're going to say it's a purchase order requesting the inventory. Let's say it's going to be requested from vendor number one, which I just made up event. Let's just make up a vendor vendor vendor for P one for product one. So we're buying it from vendor P one. Okay. Generic. Can't you come up with names? No, I'm not that creative. It's going to ship to you. It's going to come to our store. If we were doing it internally, that would be the idea here. So we're going to say the purchase order. Let's say this happens on 010 low. No, we're on 12 1231 to five.


Hello everyone and welcome to another episode of Code Emporium as we continue our journey to build transformers from scratch. Now in the last few videos we've taken a look at different components of building out the encoder of the transformer neural network and in the last video specifically we also took a look at the architecture that was involved in building this network. So in this video we're going to look at the code which is used to build the encoder and if you look here the code is only 100 lines long. So I'm going to be explaining as much of this code as I possibly can and we'll also try to make it catered for people who are just starting out with Python, PyTorch and also just the concept of deep learning too. So let's get to it. To help me kind of explain all of this alongside the code I also have a runnable Google colab notebook which I've executed and have run on just some sample input just to get an idea of which layers are run and how large the shapes are for every tensor as they pass through them. But don't worry I'll probably be explaining these bit by bit. So in order to get started let's just start with some basic parameters that we're seeing here. D model is going to be the size of every single vector throughout the encoder architecture. And what I mean by that is we have for example in this transform architecture we have my name is a J let's say that we want to translate from English to some language French will be passing all of these simultaneously through the transformer encoder and eventually we're going to get different vectors here for every single word more technically this would be a word piece or it could be characters but I'm just saying words as just an example here. Now the size or the number of numbers for every single one of these vectors is going to be 512 and not only that but if we kind of blow up this architecture we're going to see that let's say we have my name is a J each of these is represented by vectors which will inevitably become 512 dimensions and we also kind of see this throughout that all of these vectors you'll see whatever we do will add eventually something called positional encodings it'll also be 512 dimensions and this is going to be like three times 512 so everything every vector that you'll see kind of throughout the encoder architecture will be somewhat related to 512 dimensions and this is the reason why I've just defined it as a parameter here. The next is a number of heads so the number of heads specifically is used in the concept of multi-headed attention. Now in the transformer neural network architecture we see these multi-headed attention units well essentially when we're performing the concept of attention we are actually going to perform it eight times in parallel and this can be represented in this figure over here where you'll see that we have we'll construct like let's say query vectors key vectors and value vectors over here and we'll get all of these vectors and then eventually we will perform there's going to be eight operations that we perform simultaneously where we take the softmax and perform all of that good stuff with respect to attention so essentially you can see consider this as the number of parallelized operations that we perform within the encoder. Next is drop prop now throughout this encoder we're going to be performing something called a dropout where we're going to randomly turn off certain neurons and what this allows the neural network to do is it forces it to learn along different pads over here and thus make weight updates accordingly this will help the neural network be better able to generalize data instead of accidentally memorizing specific data and effectively this acts as a regularizer if you've heard the concept of regularization for neural networks and it's pretty useful when it comes to very deep networks with a lot of connections and parameters now I've set this probability to 0.1 which basically means that there is a 10% chance that a given neuron will be turned off on a given stage we can adjust this value to be anywhere between 0 and 1 now we have batch size which I've just set to as 30 so when we're dealing with neural networks and we want to perform some training typically we would pass in multiple examples at the same time those multiple examples constitute a batch and there's actually a couple of reasons why we do this so first of all it's faster training and second it's also more stable training now let's take a look at this diagram over here where this is the contours of a loss function that we see as the eccentric circles and we want to get down to the red point so you can imagine these outer contours are higher we want to kind of get to the lower point now if we had performed you know looked at one example at a time pass it through the network perform back propagation with the weights update then that would just be one update and we would need to do this for every single example and that's where you get this purple curve which is stochastic gradient descent where some examples may be good and so they'll lead you to the loss but then the next example may be bad and lead you away from the loss and it becomes this very noisy gradient update and we need a lot of them in order to converge to the eventual red point now on the other hand if we had taken batches of data like if we had basically batched our entire all of our data together put it through the network and be like first look at all of these examples and then you perform your back propagation then we would have led to this blue curve right here now that could be a lot of data to process at once so typically a good middle ground for many of these machine learning and deep learning problems is to use mini batches so do somewhere along the middle where we will batch some some arbitrary number of examples together in order to just learn effectively quickly and also in a way that's more stable so that you every time that you know you will see examples you will see that the loss on average will always decrease and so in this case for now going back to our screen we're going to be it's saying that we are going to look at 30 examples of let's say 30 sentences in some English language and it's only then that it's going to propagate through the entire network that is go through the encoder and the decoder then we have a loss function which is going to be computed and then the gradients are going to be calculated in the reverse direction and will have gradient updates all the parameters will be updated only after seeing 30 examples and so we have like a mini batch gradient descent that we're going to be performing next is max sequence length which is this is the largest number of words in our case words let's just say that we're translating from English to French so it's the largest number of words that we can be passing in at a time through the encoder now in reality this is always going to be the number of words that we pass in through the encoder because if we actually look at the blown up architecture if I just go all the way to the beginning over here you'll notice that if we pass in my name is a j as the input English sentence to the transformer encoder we're going to be also passing a lot of padding tokens here and this padding token will be let's say you know if there's the maximum sequence length is like 200 words and the sentence is only four words and there'll be 196 padding tokens here that are just going to be passed in and this is always going to be the case where we'll have a sentence and then we will add some padding tokens such that the maximum sequence length is achieved so there's always a fixed length input for any kind of sentence that we decide to input to our transformer encoder the next is ffn hidden now when we actually look at the architecture here we have like a feed forward network while most of the cases like throughout the entire encoder even like through almost all the entire decoder we will have 512 dimensional vectors like I mentioned before it's only at this step over here that I'm going to be expanding the number of neurons at some point to be 2048 before making it eventually back down to 512 and this is simply to just learn additional information if and while we can like any other feed forward layer is designed to do now num layers over here is the number of transformer encoder units that we want to include in our architecture and if you look here at our diagram you'll see that there's this n times whatever this is so this is the n is the number of transformer layers because these are typically repeated so we have inputs going in we'll have it pass through the encoder layer and then we'll have it pass through another encoder layer and then another encoder layer and then another maybe as many times as you want to and then it's pass into the decoder which is also repeated in this case n times so we will repeat it again like if in this case it's five times we will repeat the decoder five times before eventually coming out with an output and so in our case I've defined n the number of layers as five and this can vary depending on complexity so you can change this to a higher number if you have a lot more data and also it can pick up more complex patterns otherwise you can also keep it pretty low now we'll take all of these values and we're going to create an encoder object by passing it into the class defined as encoder which I'll scroll up to and passing all of these values so let me just take a look now at that same code over here because I think it's just easier and cleaner so we have this class called encoder and in this case we have a this is the constructor and this is a forward method now every class that we consider here as a part of a network typically will derive its subclass from module so module is like the super class the class in which it inherits from now the reason why we will inherit from this module class is that it allows us to perform many operations behind the scene that are required for learning so for example for one it actually provides its own forward method and it also helps us you know take our tensors for example and it will we can add it to for example cpu and CUDA where we can do like cpu where we might need to take our tensors put it on cpu if we're doing inference or we we need to put it on CUDA for if we wanted to do some model training and it also helps us access like parameters a lot better where we can take for example let's say let me go here and see we can get our parameters and also even modify them for initialization purposes so module basically provides a lot of that bootstrap code for us that we don't really need to worry about it also helps us you know if we want to save checkpoints it's easy to do so using these modules where we've trained let's say for like a hundred iterations we want to save the model modules defining modules or extending classes from modules makes it very easy to save the state and for those reasons we would want to use modules and so we extend modules here and because we extend modules we want to override the forward method now in our constructor though right here i've defined this sequential here so this sequential unit is essentially going to take all the comma separated values here and it's going to execute them in the forward pass one at a time one after the other now the way i've defined this is well for the encoder the encoder contains multiple encoder layers in this case it contains like five encoder layers because here in num layers is going to be five and so i will call encoder layers which i will explain very soon and i'm going to execute this five times and here you'll see i'll have a star in front of a python list which will basically take the list and it will deconstruct it into its component five elements and then we're passing it into sequential so now we have a sequence of five encoder layer objects now we are overriding the forward method and when we override a modules forward method what happens is that it will take an input and it will propagate that input for the forward propagation step where we will pass it into our layers which we've defined here so essentially it's passing it through all of the five encoder layers and then we'll have an output over here which i've just overridden as x and we're returning that value and so that will be the overall output of the entire transformer encoder and so i hope now you understand like what's going on now at a high level we can now dig deeper and deeper and go with the lower level layers so for example what is this transformer encoder layer that we see here so to see that we'll scroll up now in our encoder layer we have a constructor again and a forward method which we use for the forward pass now in the constructor we'll first start every constructor out by calling the superclasses constructor so that it defines everything in the module so all the module components are now set up now multi-head attention is going to be performing the multi-head self-attention of the encoder we then have layer normalization which i will explain shortly we have dropout and then we'll have the feed forward layer then one more layer normalization which i am explained you know which i have defined individually and then another dropout i've arranged this forward pass and i've arranged all of them in a way that it's kind of very similar to looking at this diagram so you can just see this rectangle z encoder layer where if this is x this input is x you see that first we will take a residual value we'll just save it because we need it later we'll then pass it into the multi-head attention followed by add normalization so we'll take the residual x then we will then pass it into the multi-head attention in this case i'm passing a value called mask to the attention layer and set it as none this is because like the process of attention requires paying in self-attention specifically like let's say that you have all the words in the english sentence it's going to pay to all pay attention to all of the other words since all of these words in the encoder are passed in simultaneously we don't need to pass in any masking there's no need to say oh we can't look at some words in the future because we kind of already have all the words of our english sentence if we wanted to translate it to some other language and so i'll explain this in a bit but for now the mask actually is just set to none because we don't really need a mask here it's optional and then we'll perform dropout which is randomly turning off the neurons we will then add the residual connection to this current value and then we'll perform layer normalization so this is the equivalent of add and norm that we see over here we'll then take this output whatever this is x again we'll pass it through a feed forward and do some add and normalization so let's take the residual pass it through a feed forward do some dropout just to randomly turn off the neurons and then we perform add and norm and eventually we'll now get the output of our encoder layer and so i hope that looking at just this is a good representation of what this image here looks like now that we have a little bit of a higher intuition of like the encoder layer let's actually pick these individual units apart and see what they're really doing and let's start with the biggest multi-head attention class over here now in order to understand multi-head attention i think it's actually best to start with the scale dot product operation which is like the single head attention or the core or crux of the attention mechanism so let's get started with this now if you look at the original paper here and i'm going to scroll down to this section you can see that the core attention matrix can be represented mathematically as a softmax of let's say this q vector which is a query vector times the key vector we'll scale it and then we'll perform a softmax and multiply it by a value vector so this attention matrix essentially when we talk about query key and value essentially every single word is going to be broken down into three vectors that is a query vector a key vector and a value vector now the query vector is what am i looking for the key vector is what do i have to offer and then the value vector can be considered as what i actually offer on attention so i think the distinction between the key and the value it seems a little fuzzy but i think if we look at the code it's going to be a lot more clear now this here is the attention function where it takes in a query key and value and an optional mask like i mentioned before now in this case our query vector for one attention head is going to be 64 dimensions and so this will be like 64 i'm going to write this out entirely when we're just doing a pass for just looking at the layer sizes themselves but for now just say that this is just a constant and this dk is like some value called 64 we then now perform the matrix multiplication of the query and the key now this key vector is actually not just for one word but it's for all words for this one head so this might be a tensor instead so this is going to be like it's going to have like a batch followed by sequence length and then some encoding value and we need to hear only perform the transpose by only flipping the last two dimensions to understand this operation let's say that we have like some tensor that's 30 cross 200 cross 512 now in our case this is the batch dimension cross the maximum sequence length cross for every single word what is the embedding dimension length now if we just do a transpose of this we'll notice that all of these values are just inverted all of these shapes are completely inverted but if we use a transpose negative one comma negative two it's only going to take the last two dimensions over here and flip them and that's why we have a 30 cross 512 cross 200 instead of a 30 cross 200 cross 512 and so that's is exactly what's happening to the key vector over here and now we multiply both of these in order to get the attention matrix so essentially for your reference right now this is going to be a max sequence length cross max sequence length tensor along with an additional batch dimension and we aren't scaling this value with respect to a constant which is the square root of decay now the reason why we're doing this we can kind of see in this collab notebook so let's just say we have a query a key vector and then the query times the key vector and for each we're going to determine the variance of the examples that are within them so in the query vector it's like 0.68 it's kind of close to one kind of close to one this one's well above one though the query times key that variance is pretty large but and also the mean is closer to zero for all of them well you know just keep them keep in mind this is negative 0.04 now if you scale these values though you'll notice that well the query and key nothing changes but the val but this query times key now also becomes has a variance of the order of one and its mean also actually has a variance that's slightly it's a four times smaller than this it's closer to zero and so what scaling allows us to do over here is to ensure that the variance of values within these matrices have kind of like a mean zero standard deviation of one and this allows us for easier and stable training now what I mean by this is let's say that we perform back propagation right so we did a forward propagation we have some values we now do some back propagation and there are going to be if there are extremely large values here that can affect the size of our gradient step and so what we want to do is make sure that these values are as nominal and normalized as possible so that we have stable steps that we take during training and we won't have like obscenely larger obscenely small values that are propagated throughout the network that affect these gradients negatively and so it's just stable learning next we have an optional mask which you pass through a scale tensor note that when we're passing a mask we don't need it for the encoder but in the case of a decoder there are situations where you know we have an input where we pass in simultaneously but technically we don't know what's going to be the output for the next word because we're generating them one at a time and in order to deal with this what we'll do is we'll create a mass that kind of looks like this where we'll have a mass that has like it'll have like let's say this is for the output of a French sentence where let's say that it only has four words so this would be you know some output French sentence here and this would be four words of the output French sentence of self-attention now in this case though in in training we have access to all of the data so we can see everything but we shouldn't be using all of that data during training because that's considered cheating because during inference time we don't know what the next word is going to be generated when I'm generating the first word I only know about it when I'm generating the second word I only know about them and that's why you can see that when I know the first word I can only perform this like attention for that first word for the second word I can perform for the first I can pay attention to the first two words and then I can pay attention to the first three words for the third word and pay attention to the first four words for the fourth word and so this is going to be more relevant when I talk about the decoder in a separate video. But for this video, you don't need to worry about this mask at all. It's also considered an optional mask if we wanted to pass it through the encoder. Next is that we are going to then perform attention. So when I say attention here is that we'll take all of the values of this scaled, and then we are going to perform a softmax so that we get probability values of how much should we focus. And the way that looks here is that we'll have initially this let's say this matrix right here. This is the scaled matrix of with applied mask in this case. But if we apply the softmax operation, you'll see that all of the rows will add up to one. And so this is basically saying the first word should focus 100% on this word. The second word should focus 51% of the first, and 48% of the second, and so on for the third word and the fourth word. So now these are like interpretable probabilities of how much we should be focusing on. And then we're going to multiply all of these attention values, which are now like probabilities with the value matrices. So this is now going to give you a new set of tensors for every single word. Now what these new values do is for every input word will now have a new value vector. And that value vector will actually have all the information associated with context, it will know how much attention it needs to pay to all of the other words in that sentence. And so you can consider this value vector for the words to just have more context awareness and hence be of higher quality than just the inputs. Now this here is the code for one attention head. But when we're actually executing the forward pass for multi head attention, you can see that that only happens at like one stage here. And we kind of have to prepare the entire input in order to split it up into these multi heads. So what are we doing and how are we doing that? So I'm just going to put some numbers here because I think it's going to be useful where we have this D model is going to be 512 dimensions. The number of heads, this is going to be eight. Now that heading dimension here is going to be 512 divided by eight, which is 64. Then we'll have the KQV layer, which is going to be a linear layer of D model cross three times the model, the linear layer is basically like a feed forward layer that is used for propagation. And in this case, typically in theory, let's go back to our encoder architecture over here. In theory, we would have like all of the input vectors and we will split them up individually each 512 dimension vector, we would split it up into query, key and value vectors and then perform their operations like so. However, in code, what we would do is we would kind of like perform all these operations in parallel, but within the same tensor itself. So they're not like three individual vectors, but they're all just three, like stacked tensors. And so what's going to happen is that this is going to be like 512 cross whatever three times 512, which is 1,536. And then this is also going to be just a feed forward layer of 512 x 512. And so when we have like a forward pass where we're going to have batch size cross sequence length cross D model, this is the input. So in our case, it'll be what is the batch size? This is going to be like 30 cross 200 cross 512. We then pass it through the query, key and value layer, which is up here. And so we're going to get for every single one of these, we're going to get for every single word that we have, instead of we're going to have three vectors of 512 dimensions each. And so that'll be 3 at 30 cross 200 cross, well instead of three times 512, it'll be 1,536. Then we're going to reshape it though to be batch size, which is 30 sequence length. Now we want to break out this last dimension into these parts because right now we have the query, key and value vector, but we want to break it up into eight heads. Number of heads is eight times the heading dimension over here is three, it's going to be 64. So it's three times 64 for every query, key and value vector, which is 192. So we've now broken up the query, key and value into eight heads. And now I'm just going to switch up basically the dimensions over here. So it's 30 cross eight cross 200 cross 192. And that's how it's going to look because we're switching these two dimensions here. Now what this chunk will do here is it's going to basically break this entire tensor of this shape into three parts. And the way that it's going to break it up is in accordance to the last dimension. So we'll have now QKV vectors where let's just say each are 30 cross eight cross 200. Let's just put a cross over here cross instead of 192, it'll be 192 divided by three, which is 64. That's QKV, each of them are this. Now let's actually take these values over here and if we pass it now into our scale dot product attention, let's revisit this just to see how what we're actually looking at. So just to see this would say QKV and they are going to be 30 cross 200 cross, what did I say here that would be it would be 30 cross eight cross 200 cross 64. So let's just copy and paste that right over here. Now what we're doing here is this last size, we're going to say, we're going to get the shape of Q, which will be returning this value. And it will return the last part of that shape, which is 64. So DK is 64. Now when we scale this, this is going to basically take the square root of 64, which is eight. And we're going to perform a matrix multiplication of these two operations here. Now when we perform a matrix multiplication, it's going to be a 30 cross eight cross 200 cross 64. And then we're transposing just these two layers over here. So it'll be 64 cross 200. And so we are going to end up with a 30 cross eight cross 200 cross 200 dimensional tensor. Now this is the batch size, the number of heads. And now this is going to be our little precursor to our self attention matrix, because it's number of sequence that's a max sequence length cross the max sequence length. So this is the precursor to our self attention matrix. Now this mask here I've said broadcasting to add because in PyTorch, you actually don't need the exact same dimensions that you're adding when you're adding tensors. So for example, this mask is actually going to just be probably a 200 cross 200. But because we're adding in this way, we are actually going to be PyTorch is pretty smart in that it's going to just add this 200 cross 200 matrix to every batch and every head. And so it'll apply it everywhere in parallel that we need to. So essentially, we're going to end up eventually after this with the same matrix, even if we were to do, you know, some masking will will still end up with the same shape. Now attention here, it's just applying a softmax operation. And like we saw before, if we kind of scroll to the self attention here, you see that this was the before shape. And then after you apply attention, you'll see that this is the after shape. And nothing has changed, no shape has changed at least, except the values themselves have changed. So we'll come back here. And it's essentially going to be just the same shape. Now the value is going to be a matrix multiplication of the attention matrix cross these this value tensor. Now we know that this attention matrix is this shape, and the value tensor is of this shape. So what we're going to do is we have it for every single batch for every single head. We are now also going to have it's going to be attention that's 20 200 cross 200. And then we multiply that with 200 cross 64, we're going to get a 200 cross 64. So what this is is for every batch for every head for every word, we now have a 64 dimensional embedding that is the value tensor. And we're going to return both of these two back. So let's now go back to the logic where let me copy this over. So now we have an attention and value matrices. So we'll have attention is going to be this. And then the value is going to be let's just copy this, which essentially the same thing, but 64 just for our reference. Now we're going to reshape the value tensor to just be this dimension. So let's see if the math works out here. So that's batch size is 30. The sequence length, this should actually be the max sequence length over here, which is this is going to be 200. And then the number of heads is going to be eight times the heading dimension, which is going to be 100 and which is going to be 64. So eight times 64, that's 512. And if we compare to our old value tensor over here, that's 30 cross. Well, what we're doing is we are essentially taking this 30, we're taking this 200, but we're multiplying eight times 64 to be 512. So a little bit of a rearrangement is going on. But essentially, we now essentially have what we actually input to our to our function for multi head attention. So whatever goes in the shape also comes out. And so this entire out vector over here is just going to be kind of like this X, but just way better in terms of contextual awareness. And it's because that they have like the same kind of inputs and output shapes that we can cascade them one after the other as many times without worrying about like disrupting some code logic here. Now multi head attention is now going to return all the way to the encoder layer right here. So we're we input, I should have said this to over here, where it's going to be 30 cross 200 cross 512. But essentially, even after self attention, we're going to have the same 30 cross 200 cross 512.  And when you randomly turn off neurons, it doesn't really affect the output. And then we're going to perform layer normalization. So let's actually get into layer normalization. As with any kind of module, layer normalization has a constructor, as well as a forward pass for forward propagation. Now parameters shape will actually tell us along which dimensions we want to perform the layer normalization on. And typically, this is going to be your embedding dimension. So in this case, it's going to be like 512. That will be like the dimension itself. Now EPS is just a very small epsilon value, because we're going to perform some division over here, like we have a standard deviation, where we have this epsilon, and this is going to be a denominator. So in case that the standard deviation becomes zero at some point, it's to prevent these these infinite values that might occur because of division by zero, gamma and beta. Now these are two learnable parameters that we are going to, as we you know, as a network learns, it's going to be updated continuously. In this case, these are going to be in this case, they're going to be 512 dimensional tensors, both of them are 512 dimensions that we'll just define over here. Now this gamma is effectively going to represent like a standard deviation of values, whereas beta is going to represent the mean of values that will be applying, you know, that will be, you know, learn continuously during, you know, as we get more and more examples into the network. So now when we take our forward pass, we'll provide some inputs. Now recall that the inputs have a shape of batch size cross max sequence length cross 512, which is D model. Now this dims over here is going to be just the length of parameter shape. In this case, it's going to be just like negative one, I believe, that we're just basically saying this is the last dimension along which we want to perform layer normalization. And that is true, we want to perform layer normalization on the layer dimension. So we're going to do is we're essentially just going to take the mean of all of the values, let's say we have a word vector, right, 512 dimensions, we'll take the mean of all of those values, and we'll get only one value, right. So that's exactly what we're doing in this step. And keep dimensions true would be that, well, instead of getting a 30 cross 200, we will get 30 cross 200 cross one, because we want to keep all three dimensions here. And that's why we set it to true. And now we compute the variance, which is basically taking every input value, subtracting the mean for that layer. And then we're going to take the overall mean, and that will be our variance here. And this here is also going to be another 300, it's going to be 30 cross 200 cross one. And so our standard deviation is just going to be the square root of the variance, and which is also hence going to be just the same shape because we're just scaling it. Now what we're going to do is take every single input in that dimension, like 512 dimension, we'll subtract the mean of that 512 dimension and divide it by the standard deviation of that same dimension. And what this allows us to do is now come up with an output that is again, 30 cross 200 cross 512. But every single one of these, you know, word vectors is now going to be layer normalized. So we'll have values such that, you know, the mean is zero and variance is going to be one. And this is the same concept as like normalizing your your data, and hence the name layer normalization, because we're normalizing data by the layer. Now, we when we're applying layer normalization, though, this is only applied to like a single sample or at most a batch of samples. But we want to make sure that these numbers are applicable for across the training set. And so that's kind of why we have learnable parameters, gamma and beta, that will kind of help us in making sure that we are scaling these values why appropriately, so that the the eventual output tensor that we get is going to be comparable throughout every single example. And so when we apply it, like let's say self dot gamma, we basically multiplied with every single value of why we are going to get the same 512 dimensional output. And I can put a multiplication with like even though gamma is not the same shape, it's like 512, it's effectively going to say we have one value of gamma that is one learnable parameter for an entire batch and for like an entire sequence of of values. And so for this entire thing, because we have 512 dimensions, we are going to have 512 learnable parameters in gamma and another 512 learnable parameters in beta. So this concept here, because you know there are different shapes, it's the same as broadcasting that I mentioned way up here when discussing like how we're adding how we're able to add two different vectors of different dimensions. So I hope this makes sense. And now what whatever was input, the same shape is output. And we now end up with a layer normalized output. So let's take this. And now what we're doing here is we would have added the residuals which wouldn't have changed the shape. And we now perform layer normalization, which we know also just doesn't change the shape. And again, now we're taking a residual connection. Just to repeat, if we go to our case over here, you'll see that let's see go over to the figure, we're now at this stage over here where we're taking a residual connection, we'll pass it through a feed forward layer, and then add and normalize. And that's kind of exactly what we're doing in this step over here. So we now pass it through a feed forward layer, which is essentially going to be these point wise feed forward. Now this point wise feed forward ticks in D model, and also that hidden dimension of 2048. So let's actually go to this really quick over here. So we have a constructor, we have a forward method. Now, this is going to be a linear layer, which is 512 512 x 512. It's how it's going to transform it. And this is another linear layer, which is going to be a rather 512 cross 2048. And this is another layer, which is going to be 2048 cross 512. We're going to apply ReLU here. No, ReLU here is an activation function. And there are different kinds of activation functions. So activation functions in general, they help neural networks learn more complex patterns. So typically, when we say that there is no activation function that is, you know, without ReLU, what a function might have looked like a straight line that just basically passes through. And you know, with straight lines, you're not able to capture as much information. However, if you're able to do, you know, ReLU is a is an example of a piece wise linear function. So that means it's made up of multiple straight lines. And so it's better able to kind of capture information. And also, it kind of in its own way also acts as a regularizer where it where it will turn off certain neurons activation. Now, because this tends to lead to a problem in some implementations, you might see something called leaky ReLU that is adopted that allows some information to be passed through here. And we can also have used like the TANSH operation, which would have, you know, constrained the values between negative one and positive one. And then we have like the sigmoid activation, which we typically use when we're interpreting probability values on the out or like the last layer of your neural network. So depending on where your use case is, we would typically use these activation functions. And a commonly used one is ReLU. And so I'm using it here. So we have ReLU and then drop out, which don't really change dimensions. And so we have our input, which is again, 30 cross 200 cross 512. We then now take it, pass it through the linear layer. This linear one is 2048. So it's going to now be expanded to 2048. Pass through ReLU, which means that it doesn't change its shape, but the values will change because it's activated. Because it's activated.  And so the input shape is the same as the output shape. And that is what we are returning back. And so that's our point wise feed forward layer. And if we go back here, you'll see what comes in also comes out. So essentially this FFN or fat feed forward network is just going to return the same dimensions. Dropout, like I mentioned, will not change anything. And also layer normalization, which we did previously will not change anything. And so even here for this entire encoder layer, whatever is we input here is what we are outputting as well, at least in terms of shape. Now, because of all these operations, though, like I mentioned before, this new output X is just going to be much more context aware than what was the input over here. And so the control will now return back to self layers, where we're going to execute this same exact process the same forward pass like five times because we have five layers. And in each case, we're just going to not change the shape. The input shape is the output shape. So we can just keep recurring through them. And then once we finish the sequence, we will end up with the final vectors X that happen to be just really good at just a encapsulating context. And when I say really good encapsulating context, this is only after some training has commenced, as it just gets better and better as we look at examples. Now, all of the shapes that you see that I've mentioned here, I think I've covered all the shapes that can possibly happen here, all of them are actually written out in a flow right here when you want to execute it on dummy data here. So you can kind of see the flow controlling going from attention to drop out layer normalization where we start out with the same tensor 30 cross 200 cross 512. And even after like the sea of operations, you can see that we'll end up with an output of still 30 cross 200 cross 512. Yeah, so I hope that all of this kind of just makes sense. I'm going to put the code for this notebook, which is the same as this code essentially. And I'm going to put it all in the description down below. So it's going to be all on GitHub. So please do check it out. And if you think I deserve it, please do give this video a like, do share, do subscribe. And we'll continue in the next video our discussion for building out the decoder. I actually have some code for that. And it looks pretty cool to and we're going to do kind of a similar deep dive. But also I'm also going to do like an architecture deep dive where I go through something like I have been before, like here. So if you kind of like all these explanations which are, you know, either textual to get intuition or, you know, they're very practical with code, please do consider subscribing. It will really help me a lot. Do share. And I will see you in the next one. Bye bye.


Not And I want to send my condolences out to to India, as we know, because we followed you guys, Donnie officially retired. Got a lot of sad messages from folks. Obviously, I think India is probably gonna be in mourning for ever. Yeah, it's just over and over again. We're like, 2020, man. Yeah. But I had, this is an ode to the captain, to Captain Cool, M.S. Donnie. And it was subbed by our subbers. Indian cricket commentator and journalist, Harsha Bogo, and Bogley. I think it's Bogley. Forgive me if I mispronounce that. And M.S. Donnie's various former teammates take you through his role as a captain for the Indian cricket team. Donnie put his favorite song as the background score for his retirement video. The lyrics go, quote, I am a poet only for a moment or two. My story will last only for a moment or two. My existence will last only for a moment or two. My youth will last only for a moment or two. I am a poet only for a moment or two. I'm, quote, that's beautiful. He was the voice of the voiceless. He taught a generation of small town kids to dream big. As captain, he humbly served his nation. He did the people of India proud. This is truly the end of an era. So yeah, this was by our subbers. I don't think this is their video, but they subbed it for us, so. Thank you. As always, here we go. Donnie is, I don't need a sentence, three words will do, the coolest one. I think he's made a lot of selfless choices along the way in his journey. That's who MS is for you. For me, MS Donnie is a very passionate cricketer. Knows his ability and always backs his ability with calmness personified. He became a leader, inspired millions. After that, people started to become like him. We hope that one day, all of us will retire. That his impact on the entire team, on the entire Indian cricket team, is a great honor and it's not easy for him to fill them up. People that someone who came from Bazaar Ranchi, Kharagpur and then strode across the world, seemed like a giant, never seemed to have any trouble adapting to a different level of cricket. That's what differentiates champions from others. I didn't hear of him really for a long time until a producer told me, I've just seen this man bat. He's got long hair. He says he walks like a bull. He bats like he owns the place. Forget everybody. Remember MS Donnie. I don't show much emotion. If I had an expression with two faces, I wouldn't be like this. I would say this to myself. We were actually laughing. David was also with us. David was a great player in the morning. We were talking about what's happening in the real world. Like Mendoza and Donnie are doing. They shot in every corner. They shot big shots. They shot big shots. That was the first international level of a Shwati match. There was a lot of pressure. There were so many big players. They were so eager to become a great player. Anyone who knew MS at that point always knew the fact that he was a very astute cricketer, a very astute person. He was someone who was very sharp at all points. He was on the button. He knew what he wanted to do. He was very clear about what he was trying to achieve for himself as a cricketer and to represent the country. And also, when he was captain of the team, he also felt that he wanted all of us to do well. And he built that kind of an atmosphere. When I went to MS and said, give me the ball, I want to bowl in the bowl out, he said, yeah, okay, do it. I know you'll do it. I think the beauty of that World Cup was that everyone in that ground, no matter where we were in the field, when he made that decision to give Jogi the ball, we just trusted. So every decision that was made, every call that was taken was trusted by each one of us on the field and just off the field, just outside the ropes as well. And I think that's what created that magic of 2007. That World T20 in South Africa changed a lot of things. We know it led to the birth of T20, cricket in its modern form as we know it. But the single biggest gain beyond being the catalyst for IPF was the arrival of MS Goni. As someone who could lead from the front, who in the going work got tough, was really calm as the first we saw of him. And yet who in times of victory was not grandstanding, was not putting himself there in front. Coming in, becoming the lead player for Chennai Super Kings. It started a love story with Chennai Super Kings that has just grown and grown and grown and grown and magnificent. The first IPL in 2010, all those Champions League trophies, but more than anything else, the finishes. I asked Rahul Dravid once many years ago. I said, why Dhoni, why not Dhoni at three? He said, because nobody will finish like him at number six. He's far more valuable at six. We'll find a three, but we won't find someone who finishes like him. And then the legend of Dhoni finishing grew that gave in Dharamshala, where as I've often said, I saw this ball, white ball going against the night sky, like it was a meteor headed somewhere. That game in Sri Lanka, where what, 14, 15 together the last over, refusing singles, wanting to finish it himself. The aura, the jethafina of Dhoni, the finisher, just grew and grew and grew. And to be honest, even from the commentary box, if it was 90 of seven overs, as it once was in the IPL, you didn't think it was over. You said, how is he going to find a way out? Not will he find a way out? 2011 is something which was very important tournament when you're playing in front of your home crowd. World Cup is a big stage. And as far as bowling was concerned for a keeper to be constantly communicating with Bola at times, it's his hard job to do. And he had enough trust in me to take the initiative or take the lead with that regard. That really worked very well throughout my career and MS as captain, you know, was always the case. And Mahendra Singh Dhoni's class was special. He took the team with him. If you're talking about 2011, I'll say after that, till 2019, if you're talking about Virat Kohli, if you're talking about Coach Ravishaswari, then there's a pillar for him too. Viva Singh Dhoni and Mahendra Singh Dhoni. So they left and I don't need to say what they did. They just want to see how much they've taken their mind and will for the sake of their skills. Back in his ability, I think that's been the greatest asset which he had playing for it. He has pushed the game to that stage where the bowlers are feeling the pressure and they're committing those errors which he can really capitalize on. And that is something which shows in MS Dhoni the great finisher. He's only been captain of India for six years. In those six years, MS Dhoni has got IPL titles, Champions League titles, but with India he's got a World T20, he's got a World Cup win. And you could say that all that is because there were all these great players around him. Then he comes to England for the Champions Trophy and he says, I need an opening batsman. So who's my opening batsman? He says, Rohit Sharma, you're my opening batsman. It's changed Rohit Sharma's mind. What a role he has had to play in the evolution of Jehel and Kuldeep. I've been very lucky that I had the stump mic turned on and I've heard some very, very interesting instructions. There's almost as if someone's providing the voice and the bowlers are following. I think he's paid a magnificent role in the development of people. And I think the smooth transition of Dhoni to Kohli is one of his biggest moments. I think he's got an uncanny ability to just live in the moment and back a player when he asks him to do difficult things. His biggest legacy has been the fact that every time he's there at the crease, you have hope. It doesn't matter what the situation is. He has played that kind of innings pretty much throughout his business career. I, as a cricketer, was a friend. I'm grateful to him for all that he's been to me and the leader that he's been to me. He was very capable. He was very talented. He explored at the best, highest level. So thank you, Dhoni. Great captain or great team-man or Indian captain who has been taken so far I can't believe it. Thank you very much. What endeared him to me even more was once he gave up the captaincy, there was no song and dance about it. There was no grandstanding. Come, look at me. Me, captain, greatest. I am abdicating. Look at me. No. That's it for me. Thank you. Yeah, I'm getting emotional. He seems like a special player. Obviously, we never had the pleasure. We've seen clips of him. Obviously, but that's not the same. Right. We didn't grow up with him. But you can equate it to the greats of the people like we've known, like of our favorite teams. I'm sure you've had some good players in the Yankees at some point, right? Yep. One or two, maybe. One or two. But it's like when Peyton Manning retires. Everybody in the world of football knows how big of a deal that is. Exactly, yeah. And that's American football, if you don't know what that is. But he was one of the debatably greatest to top three at Worst. Right. Greatest quarterback of all time. All the time, yeah. Just retiring, and it's a big deal. There's players on personal teams. You know, if you're Cowboys or I'm a Falcons fan, so like when Julio Jones retires, it's just gonna be a really sad day for me. Yeah. Because he's like my favorite player. Yeah, and I, you know, and comparatively, like when you have a team you root for, like when we lived in Texas, I loved the San Antonio Spurs. And so when David Robinson and Tim Duncan, both of them, when they both retired, that was very emotional. They were tired from the Lakers. Yeah, and Shaq, when Shaq retired. But they had a resonance that was bigger than their sport as well.  But man, anybody who said to me that Kobe was the best basketball, I was like, yeah, you're absolutely right. Kobe's the best. And when Shaq retired, as much as I couldn't stand how much they dominated the Spurs year after year, you couldn't deny the fact that there was no more, no one, everybody in the NBA, no one were dominating. So the whole world of cricket knows this is one of the greatest ever. But you also have the feeling, and that's what a lot of them echoed that, how selfless of a player is. Like that's what always makes the great ones great. Yeah. They are the greatest, but they don't feel the need to gloat about it. Yeah, that's what happened with both, you know, with Robinson and Duncan on the Spurs, and I can name a number of Yankees, but Jeter, all selfish bastards, went and now Derek Jeter, from the day he stepped into Major League Baseball until the day he left, was a gentleman and an upstanding guy, as was still going to hell for playing for the Yankees. Those of you who don't know, this is what they call the Red Sox and Furiority Complex.  Anyways, that was beautiful. Really lovely. I'm sad that we never got to actually see, I mean, we got to see Satchin play. Yes. And hear the Satchin.  And I know just from the, how much we've heard about it in the comments on Twitter, on Instagram, just how big of an impact. And it was an expected retirement too. It was, but it will still be for the majority of Indians, I think, like most, when a legend leaves a team, that first season they come back out, you feel the absence really hard. It's like an end of an era. It's like a part of your life has ended. And when you watch again, do you think is it ever gonna be the same? It won't be, but that's the beauty of a moment like this and that he was right. I think probably the case with all athletes and especially for the impact. We're comparing right now the sports that we grew up with and as much as they're a part of the American fabric, because they really are a part of the fabric, we know what cricket is to India. We don't have any comparable sport comparison and what the athletes on the Indian cricket team mean to you guys. So we compare what we know, but it's not a fair comparison because we know that these guys- But I know you'll still yell at us for it. I know, yeah, I'm sure you will.  But yeah, but thinking of all of you guys, obviously because we know it's not an easy time when you have the greatest of all time, your favorite player retires, it's never fun. But I'm hoping Donnie, I don't know what his plans are because a lot of times when greats retire, they go into commentary, they go into stuff like that. So I don't know what his plan is, but I'm sure he has a wonderful life ahead with his family. And that's all you can wish. Yep, there you go. And I hope y'all can get cricket soon. Yeah, I know a lot of y'all are missing cricket right now.


Now let's discuss the character of Lunato. He is the Duke of Messina and he's the man who invites Benedict, Claudio and Dom Pedro to his home and of course it's because of this invitation that Claudio falls for his daughter Hero. Now he is presented as a very protective father and also a very caring uncle even if he does support the patriarchal structure at the time, patriarchy being a society that's controlled by men in the sense that he does also get a little bit tired and exasperated by Beatrice's strong-headed nature. So on the one hand, he kind of tells Beatrice off, he tells her, you know, the attitude you're never going to marry anybody, so he does try to help her find love and ultimately this is how she manages for another Benedict. However, when he does realise that Hero, when he falsely believes that accusations against Hero for not being a virgin, he also is really horrified about this and he asks for a dagger to kill himself because death really is only thing that can cover up his shame. And even when Hero pretends to die, he is kind of happy when she, you know, falls and she faints because he wants that, it's so shameful to consider that his daughter might actually not be a virgin, okay? And of course by the end, he's presented as also very honourable and forgiving of Claudio when he realises that his daughter, you know, these were all very force slanderous things said against her and he decides that Claudio can still have Hero's hand in marriage. Now, as you see behind me, I've prepared key quotations to remember for his character, particularly if you're writing about him or other key characters in the play as part of your coursework or exam, so let's get started with the quotations and the word-level analysis you need to do for Leonardo's character. Now, the first quotation is when he is showing that he's exasperated by Beatrice, he says, by my trothines, that will never get via husband. So here's basically saying, Beatrice, if you keep on being so headstrong, you'll never get married, okay? And of course that was seen as something that was really important during that time for women to have to be passive. And of course, Beatrice isn't. Now, the word-level analysis here you want to do is the literation of T in troth, vow and thee. This shows that Leonardo is being very emphatic when he's speaking. The second quotation to bear in mind with his character is when he realises or when it's revealed during Claudio and Heria's wedding that Heria might not have been faithful and Claudio is denouncing her and he's so horrified and mortified. He wants to be killed, he wants to die because he asks, hath no man's dagger here a point for me? And this is a rhetorical question, so that's one of the key things to point out. Now here, of course, what this is showing is how highly a woman's virginity was seen as and, you know, a woman not being chased was seen as tantamount to her social death and so he sees this as so shameful. Also, the other word-level analysis you want to do is the essence of A in hath, man and dagger. In addition, within this same scene, he literally says to his daughter, death is the fairest cover for her shame.  You can't believe it, okay? Now here, of course, what this is showing is he has very strong patriarchal views about women's role in society. Women are supposed to be virgins. Whilst men can run around and sleep with all these people, for example, Benedict being a bachelor, he's able to do that. However, if Harrow, God forbid, she sleeps with the man, then that is something so shameful that death is really the best cover for that. Okay? Now here, you want to talk about this notion of this metaphor, this metaphoric language, death being a cover, a way of kind of changing and removing and averting people's gaze from Harrow in order to maybe save her reputation. If she dies, then that's really the only way that her reputation can maybe somewhat be restored. Now, the other quotation for Leonardo's character is when he states, within the same scene, do not live hero, do not open our eyes.  Because of course, he doesn't want her to revive and then, you know, have a renewed sense of shame that he has this living daughter who is not a virgin. Now, the word love analysis you want to do here is the repetition of do not, where he's very emphatic, Leonardo, and also the assonance of E in live hero, oh, thine and eyes. Now, the final quotation, of course, this is when he realizes that his daughter is innocent and he asks, this is one of the main accomplices of Don John, he asks him, are thou the slave that with thy breath has killed my innocent child? Okay, so this is a question, it's not a rhetorical question, it's an actual question. And he's saying, you know, are you one of the people he plotted with Don John? And of course, this is when he realizes that Don John is the person who is at the center of all of these allegations and he realizes that his daughter is actually still a virgin, she's chaste, she's innocent. Okay, so of course, here we can see that he's very angry, but ultimately he forgives Claudio and still lets him marry his daughter because he realizes that it's not Claudio's fault that he believed that, he was also deceived, okay? Now, of course, this is a question and the word of the analysis you can do with this quotation is the alliteration of tea, hair, to show that he's really angered to realize that somebody would tarnish his daughter's reputation. So that's it, when it comes to Leonardo's character, of course, he's one of the key characters within the place that do make sure you try to pay attention to some of these quotations and use them and incorporate them in your writing. Thank you so much for listening.


So that was fun, wasn't it? You all know from previous lectures that a linear system cannot solve the Excel problem and you just made it do it. It's of course possible because a violation of the abstraction relative to what's really going on here. And now you can say, well, does it matter? Isn't this mightily abstract that we have that? Yes, it does. Many deep learning failure modes are failure modes of the abstraction. For example, a small gradient may be exactly a zero and your network will never alarm. Also, e to the power of 1000 is infinity. So if your neural network has, say, an exponential then all of a sudden you have infinity. So all gradients vanish at that case. In fact, they become undefined. Now like a lot of functions that we'll be using during the course will use softmax. The softmax of z is e to the zi divided by sum of j is one to k of e to the zj. Now, what happens if e is large? Well, it shouldn't produce any problems like gradients might become zero or one. There's no reason why anything should go wrong here. But the gradient in practice will be infinity. It will be not a number, sorry. The reason why it will be not a number is because we have infinity divided by infinity plus a couple of sums. It's entirely undefined in that case. It is not undefined mathematically. This is a perfectly well-defined gradient. It's just that in the abstraction that we use with numbers represented by how many digits we have it looks like it's infinity. I have seen students spend weeks debugging this very problem. Oh, I'm getting not the numbers. Well, I don't know where are they coming from? Not the numbers are impossible because we know that the gradient when we write it out by hand is always well-defined. Well, it's not in the abstractions that we have. And when you're debugging artificial neural networks it's often useful to be mindful that the abstraction that we write on paper is not what is actually happening on the machine. Now, let us get back to linear neural networks and do something completely different. What we'll do is we'll now do deep linear artificial neural networks and just see how well that works. We take a 10-layer neural artificial neural network. We will not make it non-linear, not like the previous exercise, not on purpose making it non-linear. We'll leave it in the domain where the abstraction is good. And we know that it can implement the same function as like the one-layer neural network that we used before. Okay, now it's your job to show that it can implement the same function. Please come back in 10 minutes, Max.