ry single vector is context aware and that those vectors are what x constitutes and we'll be passing it in here later now I have like the almost like some similar thing which is y that I've generated over here that is batch size cross max sequence length cross d model so this is essentially also a sense of like the every single word has been encoded into a 512 dimensional vector but this is merely just the input part so why is simply like this stage right over here at this point I've just generated some random values but they would actually come from some you know data set and we would have transformed it for this video it's a little too much code but I will show this in subsequent videos for sure but for now just to get the use to the entire flow I've introduced it here now this mask that I'm creating over here is actually going to be a look ahead mask this is required because during training we pass all the words of like the target sentence at the same time but we don't have information about every single word in that sentence at the time and so we would need to mask some inputs so that we can prevent ourselves from cheating and looking ahead and so we would create a look ahead mask if you're curious about how that mass looks it kind of looks like this where we have for every single this would be a 200 cross 200 tensor where we have you know for the first word it's like saying that we don't need to mask it zeros are don't mask negative infinities are mask so for the first word it's allowed to see itself the second word is allowed to see itself and the word before it the third word is only allowed to see the stuff that happens before it and so on until the very end so this kind of prevents cheating from happening and hence we'll prevent a situation where the decoder performs well during the training time but terrible during the test time because of data leakage and then we now pass in all of these values calling the constructor of the decoder class which we will take a look at and then we call the models forward pass and for the forward pass I've done a lot of print statements over here so you can kind of see exactly how what was the dimension of every single tensor that occurs and I will provide this code all in the description down below but in this video I'm also going to explain why these dimensions are the way that they are so I've pasted all of the decoder code actually right here in the collaboratory notebook so this code over here and this code are the same and so I'll just be using whatever is in the decoder collaboratory notebook let's start with the decoder class here so I've defined a decoder class to have a constructor which takes in all the parameters that we just mentioned before as well as a forward pass now I'm overriding this torch modules because module itself is a PyTorch extension that provides a lot of the underlying boilerplate code that's required for creating neural networks this includes like performing back propagation it performs like memory management and also managing like whether like tensors are supposed to be on GPU during training time versus on a normal CPU machine during testing time and it also facilitates model forward passes too and so because of all of these nice little intricacies that torch provides I use torch modules over here and because we use torch modules we also can now provide a forward pass and we can do things like call this entire method directly as if it were just a method this would instantiate in fact the forward call or the forward function so when we call the forward pass we have the English sentence the Kanada sentence as well as the mask so let's actually write out all of the shapes of these values so that we can kind of walk through its execution so x is going to be the English sentence which is going to be 30 cross max sequence length which is 200 cross 512 and we can say the same thing is going to be for y as well y is also going to be the batch size which is 30 sentences cross 200 which is the max sequence length cross 512 and then the mask itself at least for this simplistic case is going to be 200 cross 200 which is max sequence length cross max sequence length now we're going to be passing this into layers so layers here is a sequence of you know decoder layers so essentially if you look at this figure over here I've this here is one decoder layer but it's repeated n times and so if this is under a class called decoder layer we repeat this n times that will be what self dot layers is essentially now typically you would use something called torch n n sequential that looks kind of like this right over here but here you can see I'm using sequential decoder and this is mostly because like when I want to call the forward pass if I'd use just like sequential on its own you cannot pass in more than one parameter and because I have more than one parameter to pass like I need to pass in a mask for example that's kind of why I've implemented my own sequential decoder because I just can't use this directly so it extends sequential and what it will do is I'm just going to take this input but when I'm calling every single decoder layer I'm going to be passing in all of these parameters but I get the new value of the canada outputs that is the output sentence every single time and I just keep feeding it back into this module but the same value of x that is you know from our encoder that is the English sentence is just passed in regardless of which decoder layer that we are looking at and thus sequential decoder thus becomes an array of these decoder layers let's just actually take a look at what each decoder layer looks like so if you scroll up here you can see also decoder layer it extends you know torch modules and hence it also has a constructor and a forward method that we can call directly now let's again walk through the forward pass and I'll describe each of these as and when they appear so first of all we're assigning this here to like this underscore y and this is going to be used for residual addition so you can kind of see in the normal paper here you see that there are these arrows over here these arrows are essentially skip connections or residual connections and you would typically see these in very deep neural networks this is because as like the deeper the network is you know as you do a back propagation of your gradients they become smaller and smaller and smaller and eventually they might vanish towards like the beginning of your network and if gradients vanish that means that your neural network will no longer learn and so in order to propagate a signal much stronger we use skip connections they propagate like the input signal much better in the forward direction and also like any kind of loss or I should say like the derivative of that loss with respect to the inputs they will propagate it very well in the backward direction as well and so there's always going to be like some gradient update that occurs even for like the earlier layers of the network and that's kind of why like this architecture uses those skip connections what we're going to do is now perform the mass self attention where we take the Kanada sentence and we're also going to have you know the decoder mask which is the same 200 cross 200 mask that I showed before so this self attention actually you can see over here is multi-head attention so let's go over to that module multi-head attention which is right over here in the multi-head attention you can see that because this is the Kanada sentence in this case it's going to be 30 cross 500 and no 30 cross 200 cross 512 that's the dimensions right that's batch size max sequence length and demodel now we're going to have query key and value so this qkv layer if you can see here it's a linear layer that will basically map this 512 dimensions to the three times 512 dimensions which is 1536 I believe that's the multiplication here so looking at the decoder architecture it's kind of like this where we have batch size max sequence length cross 512 and we create query key and value vectors for each and every single word and that's why since one word becomes now three vectors that's why we see 1536 so the query vector for a word is what am I looking for the key vector and the value vectors essentially are like you can consider them as some form of memory or you can also consider them as what do I have to offer so it's kind of like its own information that it already knows so coming back to the code this is now going to be a qkv which would be 30 cross 200 cross 1536 as we've created query key and value tensors which are going to be used for attention but we are also reshaping this because we want to perform a multi-headed self-attention so we're kind of distributing this across eight heads and because of this reshape I'm quite literally going to write this out as 30 cross 200 cross the number of heads which is eight that's how much we've defined over here it'll be eight that we passed through cross if you kind of just do the multiplication I think it's like 192 that's 192 times eight will be 1536 and so we've created eight attention heads and if we look at our decoder what this why we're doing this is that these are essentially going to be eight parallel processes that are going to go on that is each one of these rectangles there's like eight rectangles over here and each one of them is going to perform its own attention and then in the end we're going to be concatenating them together so coming back again to the code this is going to be 30 cross 200 cross 8 cross 192 we're just going to be reshaping this so let's just write this out again so it's 30 cross we have the second dimension which is eight cross the first dimension is 200 cross 192 that's just as an advent of the reshape now we'll be chunking this up and to by the last dimension into three parts over here so that's 30 cross 8 cross 200 cross 64 that's 192 divided by 3 and this is Q but you can also say that the key and the value vectors will also be the same so I can kind of say this is going to be the key as well and this is going to be the value vectors here as well now we perform scale dot product attention so this is like the crux of the attention mechanism over here and we're passing each of these query key and value vectors along with a decoder mask so let's go to our scaled let's copy this because we probably want to know what the sizes are for these in fact we'll do that for everything over here we go over to scale dot product attention which is like this right here and we'll just copy some of these sizes and I know the mask was like a 200 cross 200 just to keep this in mind so DK here is actually just a constant that we are going to be using in a division over here and this is required to scale the multiplication the matrix multiplication of Q times K and we require this because we want all of these values to kind of be with like a mean zero and standard deviation of one and this scaling parameter actually just helps us accomplish that now in fact we can actually see this in action by like you know I'm in my github repository right now where you can kind of check out all of the code for this video and also like the past videos that I've created on transformers over here and you can see that let's we want to perform this operation of self attention where we definitely we have like a Q query vector and the key vector which we do like a matrix multiply over here but now the query and key the variance is like close to one but the variance of like the multiplication of these you could see that it kind of goes off the charts a little bit like it's five to six times greater however if we were to like you know scale these values you can see that the variance now becomes much more tractable when we perform the multiplication here we transpose the last layer and the last and the penultimate layer so that means that we we actually make let's say if this is the key vector we actually transpose just both of these and we can't just do like a k dot t because otherwise that would transpose like the entire thing it would be a 64 cross 200 cross 8 cross 30 tensor which is not what we need we just want to transpose these last two layers and so if you do a matrix multiplication of query and key that would be 30 cross 8 let's do that 30 cross 8 cross and this would be a 200 cross 64 times you know 64 cross 200 which would be a 200 cross 200 scaled tensor in this case it's scaled but this is just a division it doesn't change any shape so this is kind of like the initial parts of how the attention matrix is actually going to look what we do is if mask is not none in this case it is not none we're going to add this mask and we're adding a mask of 200 cross 200 to this tensor so even though it is a different shape PyTorch supports something called broadcasting where the last dimensions here match the last dimensions over here it's still a valid addition and it's just going to add the same exact tensor or apply the same exact tensor to every single one of these elements so there's like 30 times 8 which is like 240 so that's like for every batch for every head we're going to apply the same exact mask and so this mask is now going to end up with it's going to be 30 cross 8 cross 200 cross 200 after the operation of this masking is done just note that in this case this is a look ahead mask but we could have also added some padding to this same mask and we would have still ended up with the same dimensional tensor once the entire you know operation was said and done next is that we perform softmax now softmax is just like taking exponentials and dividing it by the sum of exponentials which essentially does not change the dimension of anything here the attention matrix is still going to be the same dimensional shape it's just going to do the softmax on the last layer this last layer over here so that every single row is now going to add up to one now the values here is going to be a matrix multiplication of the attention and the values vector which is this is the attention and the value tensor looks like this so this is going to end up with the shape of 30 cross 8 cross 200 cross 64 right and this is just for one attention head and we're going to be returning both of these so in the self attention case this attention is going to be how much attention are we paying for every single word on itself whereas the values is going to be well what are the actual context aware final tensors that we are looking at right now for every single word so that's for every batch for every head for every word we have a 64 dimensional vector that represents context of that word and eventually when this logic returns we copy this over the will come back all the way back to multi head attention over here this logical return and both the value this is the values tensor and now we're going to well reshape this such that we flatten the heads dimension as well as you know this 64 over here and so this would be 30 cross 200 that's max sequence length cross 8 times 64 which is 512 and so effectively we are concatenating whatever we found from each and every single one of the eight heads together and then we have this output that is just going to be passing through a linear layer and this linear layer actually doesn't even change the dimensions it's like a feed forward layer that maps like 512 dimensional vector to another 512 dimensional vector itself and so we have the same shape over here so I can just copy and paste that over here and it's this now that is passed out so interestingly here because this multi head attention you can see like it's the same input shape that was input same that was output and so it can also be a repeated layer without like disturbing anything like very jarringly like oh the shapes don't match you won't be getting like such errors here so now once that multi head attention is complete the control is going to now transfer back so self-attention is now complete and the output is this shape now we perform dropout dropout as I mentioned before is the random turning off of neurons so that the neural network can learn a longer more generic generalized path better for generalization and hence it doesn't memorize so it's the same shape it's just an operation though next is add and layer normalization so we can add whatever we saw here which is the same shape essentially this was the same shape up here this underscore y we add these two together and then we just normalize them with layer normalization so let's go to norm one norm one is layer normalization and we pass a parameter shape over here called little parameter shape and we pass 512 so let's actually look at layer normalization for a little bit here going up to layer normalization all right so we have like this dims over here and this dims is going to be what along which layers do we actually want to perform layer normalization the entire goal of layer normalization here is for constraining the values of every single layer to be with like a mean zero standard deviation one and this helps stable training so that we can jump you know the jumps that happen during like every single training phase and gradient update phase are pretty stable they're not too erratic and so this here is going to be I believe dims is going to take on the value of just the last layer so it's just going to be like a list of negative one in this case that's what dims is next we're going to compute the mean along this last layer now the inputs just for clarity are going to be of this shape which is 30 cross 200 cross 512 that's exactly what we're passing in so we're taking the mean of all of these elements here that is all the 512 dimensional vector that represents a single word we're going to take the mean of that and we say keep dims is true because we if you just take the mean of these and you you know you run this operation it's going to just return to 30 cross 200 dimensional tensor but we want a 30 cross 200 cross one dimensional tensor so that we can perform some other operations down here so that there's like a the tensor shape dimensions match so one step at a time we take the inputs which is the shape we'll subtract it for every single mean that means that every single word is going to be having like there's like a single mean that we have computed and we're going to subtract basically that same mean value over here so you're going to end up with the same dimension over here and so this is going to be the same dimension that's the variance and standard deviation is simply the square root of that variance which also doesn't change the shape over here and the output hence also doesn't really change at all the only difference is now this y is going to have much more normalized values where the mean is zero standard deviation is one and hence allows more stable training but what you see here too is like there are some gammas and betas so gammas and betas are a set of learnable parameters in this case there's going to be like 512 essentially of these that's the parameter shape so all of these gammas are initialized to one and all the betas are initialized to zero so there's essentially like 512 gammas and 512 betas that we'll be learning over time and the goal of these learnable parameters is kind of like if you've seen like batch normalization too where we have essentially gammas and betas they help learn along dimensions that are not included in this entire operation of like computing the mean and standard deviation and so even like even though we do like some strange normalization along like this 512 dimension we're trying to make sure that all of these parameters become kind of like comparable to each other and that's kind of why we introduce all these we introduce like so many gammas and betas we have like one for every single batch and word that we have like a specific gamma value and so what we do is we multiply gamma times y that is the value of gamma times every single element in y and just add like beta so gamma is essentially going to be like a standard deviation it'll learn to be like a standard deviation across multiple examples and then beta will also learn to be like a mean across multiple examples so that it can better so that like this out over here is better comparable across these multiple examples and after performing this operation we still end up you know it's still the same shape right now we have the same input same output shape as well so when we return the control now from layer normalization we go back to where we go we go back to decoder layer and the output layer normalization is the same shape right and we just perform the same operation here but instead of massed attention we now perform cross attention which is slightly different now in this cross attention case I'm not passing in any mask since like the encode this is going to be the difference between self attention and cross attention which I will mention actually let's just go over here so it just becomes more apparent as to why we're not passing a mask for now so multi headed yeah this is multi headed cross attention you'll kind of see like the code is very similar to what we saw for self attention however the main difference you'll see is that instead of taking a QKV layer we have a Q layer that's separate from a KV layer so every single element that is in your encoder that is like the the connoisseur words that we pass in the decoder they are going to form query vectors but then we are going to convert every English word that we get as the output of the encoder to key vectors and value vectors so you can see for self attention we created you know from the same connoisseur words we have query key and value vectors but for cross attention if we go all the way here now we're performing cross attention let's say that this is coming from the English encoder this is like the set of English words for the sentences in the encoder each word is now encoded into a key vector and a value vector from the encoder but for the connoisseur side we encode everything just into a query vector and so while self attention everything is in every word is encoded into a key key query and value in cross attention we have the query coming from the decoder but the key and the value coming from the encoder and then we perform very similar operations along with no mask in this case so that's why coming back here we'll see that like the inputs here like x is going to be you know the same dimension 30 cross 200 cross 512 and so y will also be that dimension but now after passing through the key and the value again key I say every word is converted into like key and value but essentially they're going to be stacked on top of each other as a linear layer so we just convert 512 to a 1024 dimensions over here and so because like every word is going to now have to like a key and a value from the from the encoder side this is going to be 30 cross 200 cross 1024 every now for the query side it's going to be 30 cross 200 cross 512 and we're going to now reshape this so that we can perform multi-headed cross attention and so the key and the value let's take this but instead of like 1024 it's going to be eight times this headings dimension is 64 so I believe it's going to be 128 and we're also going to reshape our query vector too where the query vector is essentially also going to be multi-headed so we'll take this we'll paste it here and this is going to be times eight but instead of just two times the head dimension it'll be just the head dimension itself which is 64 now we're going to permute the key and the value over here so let's just take this but we'll switch over the second and the third places so that's 30 cross 8 cross 200 cross 128 and the same kind of goes for for the query itself too which is 30 cross 8 cross 200 cross 64 and then we will chunk the key and the value vectors here into just two parts based on the last dimension so we'll take this and we're breaking into two parts so this will be 64 for the key and then we also have the same thing for the value essentially and so we have query key and value tensors of like the same shape and this is where we pass this in now all through the scaled dot product attention again so if we go back up here we actually end up with the same shapes right 30 cross 8 cross 200 cross 64 they all have the same shapes essentially in the mask in this case we don't have a mask but all the operation shapes remain exactly the same and even if we don't execute this code it's not like the scaled this scaled value it still maintained its original shape anyways so it wouldn't matter I hope that what's clear here is that in the decoder case we actually don't even need for the cross attention part of the decoder case we actually don't even need to execute this unless you put in like a padding tensor as well or like a padding mask so that you can ignore some of the padding tokens which is required because you know some sentences might not be as long as like the total length the maximum sequence length of like 200 but unless if you don't do that you don't really need this this section over here that's not going to be executed and hence it's going to skip over and these shapes are going to be preserved so it's kind of exactly what we want and so what you're going to get is this output tensor over here put in our value tensor and now we are going to now concatenate all the heads together as we did previously and so this is going to be 8 times 64 which is 512 so we'll just reshape that and this is now going to pass through a linear layer which is not going to resize anything it's just for additional information and so we actually now have an out vector that is going to be you know 300 cross 200 cross 5 or 12 which is exactly what we input in here so we copy this this is multi-head cross attention so let's go back to the decoder layer and the output here of y is still going to be the same shape next we perform you know kind of the same procedures as before we perform dropout which doesn't change the shape it just randomly turns off neurons and we'll also do layer addition with layer normalization which does not change the shape again and next we still have this so we pass it through a feed forward connection now this feed forward network ffn I've defined it as position feed forward let's actually go to that just real quick it's actually quite a simple function so it shouldn't take too long so we pass it through the input that we have which is x is going to be like you know it's going to be this shape right that's x we pass it through a linear layer which is going to map it to a hidden dimension right so that's 2048 and then we pass it through relu which doesn't change the shape it just it's an activation function that will allow the network to better understand and model complexity in your data and complexity in patterns typically like relu does have like some issues with like the dying relu problem because you know for any activation value that is lower than zero it will just completely shut it off that's just the nature of the function and so some other functions like elu, gelu, leaky relu are also used which you can experiment with too but for now we have relu and that doesn't change the shape dropout doesn't change the shape and then linear2 which is just going to be the inverse of the first linear layer where we still now get this demodel so let's just write this out and this will be 512 dimensions and this is return and so at the end of the day when we do our position feedforward network we are going to end up with a 512 dimension output here which is the same as everything that we've seen before next we perform dropout like we said that changes no shape layer normalization changes no shape and we return the value that we get at the output of one decoder layer and this output is exactly the same dimensions that we input to the layer itself so we take this value and this is the decoder layer that we will go over here this decoder layer let's just put this in a much more readable way so this is the output of one decoder layer so this module executes once and now we have this y which would have been this shape over here and then we just input that same y you know this has to go num layers we defined it as like we defined it as 5 so this will execute like 5 times so now this new tensor which is slightly more context aware of the Kanada output language is passed back into this module and we keep updating y which is like the Kanada target tensors for the Kanada words now once all of these are complete the control will pass back over here which the output is going to be well the same shape right and this is good so the input now matches the output and from this output I haven't coded it out here but it's essentially going to be mapped into a softmax a linear layer of softmax and from that we can actually compute what the next words could be so everything that I've kind of like discussed over here I actually like put so many print statements so you can kind of see like I've printed it all out in the end you will see that's exactly what the output shape is 300 cross to 30 cross 200 cross 512 and to see exactly like what happens you know in the conceptual architecture we're essentially way at this stage over here we have the batch size cross max sequence length cross 512 that's exactly what's he 30 cross 200 cross 512 where we now can map it to a you know we can map each of these now to a Kanada word that is human interpretable and perform some softmax to determine what is the word that is supposed to come next all right so now that we have a good idea of even how to code each of these components of the encoder and the decoder now I want to kind of focus on the beginning over here like how are we going to take language and encode it into a format that computers can understand so let's talk about that now so in this Google collab notebook I have a couple of files here that consists of the English sentence which is the source sentence as well as the target Kanada sentence which is the sentence we want to translate to now if you want to know how that information looks it kind of looks like this where this is the Kanada file where there's about like I believe like four million records over here that's four million sentences and then we have a train file over here which is the English translation for every single corresponding Kanada translation over here so this is he is a scientist this is the Kanada translation of he is a scientist and we need to kind of like clean this file, process it, create a dataset and then feed it to our transformer neural network so the source of this dataset is Samanantar if you want you can actually kind of read this paper of what it contains but essentially it has the English translation so for every English file there will be 11 files where you have 11 Indian languages and those languages are over here which are short forms of some of these Indian languages I am using the KN file which is the Kanada file and you can just see like some more details of like what are the number of records and our tokens and whatnot and also all these like they have they also had benchmark analyses so I would just recommend that you read this paper for more information but if you actually want to download this dataset you can hop over to this link which all of these links will be in the description below and you can just click like on the downloads and then download everything I will say a big warning is that there is no good way to download just a single part of these datasets like I mentioned there's like four these files are pretty large they're gigabytes of data and in fact I think you need about 20 gigabytes of data to download everything and unfortunately you only have an option to download everything at once and not just like Kanada English or Kanada Hindi so I would just recommend that you have some space on your computer before you try to download these coming back to our file we want you to find a start token padding token and end token so start token is going to be to begin a sentence end tokens to end a sentence and a padding token is typically used because we want to make sure that even though sentences might be of different number of words or characters we want to ensure that they are converted into numbers, vectors, matrices that are of the same length and hence we introduce padding tokens next I'm going to introduce a Kanada vocabulary which is a list of a set of all possible symbols that we can input and output to and from the model and the same is true for the English case it's the number of characters that can be input and output to and from the model let's talk a little bit about these individual languages so that we kind of get an idea and a better understanding of what we can expect moving forward for translation now these English characters here we have a character set that consists of consonants and vowels and we call this entire set an alphabet as each character can represent an individual sound and so on Kanada is kind of much in the similar way of we have consonants and vowels so this first row is a set of vowels and so is this third row so that would be and then we also have a set of consonants so it's I just read the entire thing wonderful even though we call Kanada an alphabet when just speaking normally it is technically more an alpha syllabary where each character here clearly just represents a syllable at least here but we can also combine them together we can combine consonants and vowels to actually create a sound like a unit here so for example this is and that becomes so we're combining a consonant which is and a vowel which is one of these in order to create like different units of characters here and so Kanada is an alpha syllabary and like we did for ka we can do that for every single other consonant and vowel pairing to get a set of these units of consonant vowels and this is what we call in Kanada a ka gonida and so since English is more of a phonetic alphabet language whereas Kanada is more of an alpha syllabary you can kind of see that there would be some form of complications that can occur when you're translating from English to Kanada they're very different kinds of styles of writing so it's fun to keep in mind though now for every single one of these lists of Kanada and English vocabulary we want to create an index so that's a dictionary that maps some integer to a character that you see up here or a character that maps also to an integer and I'm just creating that index over here now we're going to read these files that we mentioned over here entirely from our google drive and we're only going to pick out the top 100,000 sentences for now so that's faster and easier to train we probably don't need four million but if we do need more we can always pull more by just increasing this total sentences value and I'm just going to get rid of these new line characters that are appended at the end of each sentence and then just print it out for you and so you'll see like the English sentences look like this and their Kanada translations correspondingly kind of look like this now as an input to this transformer neural network entirely either the inputs or the outputs we're actually going to convert every single character into some embedding instead of every single word as we have probably discussed in many videos before when we encode every single character into an embedding we want to ideally it should be a little bit smaller so that there's not too many parameters to learn inference becomes faster but over here you can see that the maximum Kanada sentence it has 639 characters while the maximum English sentence has 722 characters now you can go and plot a distribution but I'm typically sure that you'll see a very long tail curve where there might be only a couple of sentences that are very long and I don't really want to just accommodate all of those sentences that are just super long even there's only like a few of them anyways I would rather just accommodate the majority and try to decrease my dimension so that just becomes easier to learn less parameters throughout the network and so what I'm doing is I'm just trying to see the 97th percentile of like the longest word so this is basically saying that there's 172 Kanada sentences in my data set that are less than 172 characters and there's only like 3% that are more and with English we have a very similar size so what I'm going to be doing now is defining a maximum sequence length that is the maximum number of characters in a sentence should be only 200 anything more than that will just get rid of from the data set and so I've just written little helper functions that kind of just check these conditions whether they're actually first of all whether they have valid tokens that is all of the tokens that are present in each of these lines is actually a token of the vocabulary that we've described up here so only if this is true and also it has like a valid sentence length that is less than 200 characters and then those are the ones that we actually use in training the transformer and so we reduced from 100,000 to just 81,900 which is completely fine and why it's like such a huge reduction is mostly because both the English and the Kanada sentence translation have to have these two parts satisfied next we are going to create a data set so PyTorch has a predefined class called a data set which is required in order to feed data and train a PyTorch model this kind of takes a care of a lot of the boilerplate code under the hood so that there's consistency in how we fetch batch and everything else with data and in this case since we're working with text we create a text data set class and I want to override we have to actually override when you're creating a custom data set a get item the length and also a constructor if you require it so in my case this length is going to be used internally by data set to get whatever the length of the current sentence is but whereas get item here is going to take in an index and get the corresponding English and Kanada sentence which we retrieve when we're going to be training so we're iterating over a batch we'll be getting an English sentence and a Kanada sentence and this function is going to be called to fetch that data I'm creating a custom data set because there isn't really this data set class doesn't really satisfy my own needs but you can check PyTorch's repository for data set just to see if what you have is already built in otherwise you can build a custom data set like we are doing here and so when you execute data set and you'll just like get let's say the second element in that data set you'll see that you'll return a tuple of the English sentence as well as the corresponding Kanada translation now for the sake of this entire setup let's just say that the batch size is three now to explain why we were batching in the first place is that let's say that we just take one input sentence and one Kanada translation as just the batch size so batch size one which is essentially no batching whatsoever so if we pass in one Kanada sentence an English sentence during training we'll get some output loss function and then we're going to be performing back propagation update all of these like millions of parameters over here in order to now get a new state and then we will repeat this again with passing another English and Kanada sentence and again all of these parameters have to be updated updating each and every single parameter for just every single example can take a very long time and also your loss steps will also be very jaggedy so in order to speed up this training we kind of parallelize passing in information to this network so in this case I said three so we can put three input sentences in Kanada in English pass it through the network and only after all of these have been read we only generate a single loss and then we just back propagate that loss so it's only the parameters are updated once for every three that are input so we can increase the batch size to decrease the number of times the entire network is going to be updated and this will speed up training and hence we use pretty typically for many machine learning cases we use mini batch gradient descent let's just say that we set the batch number to three so what you'll see is actually two tuples of data so for the first one you're going to see three over here there's three English sentences one that are comma separated two and three and then that's going to be up till here and then you'll see the corresponding Kanada translations in another tuple over here and that's kind of how the data is going to be given and processed during training next is tokenization so we have these sentences but we need to convert these into numbers because computers don't understand text they understand numbers and so I've created this tokenized function over here that'll take a sentence and it'll take also whichever language depending on what you want to other English or Kanada it'll take that character to number embedding and it'll take an optional start token and end token depending on whether we want to pass in a start token or end token and so if we have a start token we'll prepend it to the beginning of the sentence if we have an end token we will append it to the end of the sentence and in other cases we need to have padding token so for the remainder of all of the characters we are going to just introduce a padding token that we discussed previously and then just create a torch tensor so that everything instead of a Python list it passes a tensor as everything in PyTorch is processed typically with tensors so to look at an example let's just say that we have a batch since the batch size is 3 it has 3 English and Kanada sentences over here now I'm going to describe some empty lists that is this is going to be the list of all English tokens in these sentences and Kanada tokens in the corresponding sentences now for every single sentence what we will do is we are going to call the tokenized for the English part but in the English case I don't want to use start tokens and end tokens we are passing them all simultaneously anyways we will have the entire English sentence so there's no need of a start token and end token now for the Kanada case I need to pass in a start token because during the generation phase you're not going to have any Kanada word to start with so you're going to have to inject something into the model and that will be your start token and I'm also going to pass in an end token as well just to indicate this is where the sentence ends and after that it's just padding tokens and so you can see that if you look at the Kanada tokenization so we have like these three Kanada sentences their corresponding like numeric interpretations would look like this this is the first sentence this is the second sentence and this is the third sentence so they've all been mapped to from characters to an index an integer from that character integer dictionary that we just created previously and we can see that these 123s are padding tokens this 124 over here is an end token end of sentence and this zero is a start token we can see something very similar for for English too if we wanted to try this out where 95 in this case is the padding token and then we have yeah there's no start or end token so it's just the padding tokens and everything else is just characters and the sentences that you see now in the last part of this video I just wanted to very quickly just touch on masking now coming to this transformer architecture you'll see that you don't really need a typical type of masking for the encoder part the only kind of masking you need is a padding mask and that's just to say hey do not look at the padding when you are computing this loss function and upgrading gradient weights here so don't look at the padding tokens they mean nothing so we might need to have like a padding mask interjected within the encoder but then that is in this multi-head attention part but with the decoder you need a mass multi-head attention which means that in the decoder since this is the generation phase and during training you have all of your Kannada translation data but during inference you don't have any of that so you shouldn't be looking forward to what tokens you haven't generated that's a form of data leakage and we cannot have that so what we do is instead we would mask all those tokens and say hey any character that comes after this current character we don't want to look at it we have no context to that we only have the context to characters that come before it and on top of this we also have a padding mask that is just masks that like kind of like what you mentioned in the encoder we just should not be using in order to compute back propagation upgrade and updates everything that I've mentioned just now can be kind of converted into look ahead masks and padding masks for both the encoder and the decoder and I've kind of printed all of these things out here now instead of like a I'm using zero that says hey this is not masked and negative and like a negative large number or technically it's like a negative infinity that says hey this is masked because eventually if you look at the code we're going to be passing this through a softmax function right over here and when you do a softmax that's essentially like an exponential function so that's whatever is zero it's e to the power of zero is going to be one which is okay that's passed through but if it's a negative infinity e to the zero is going to be zero which is don't pass through so that's kind of why we use negative infinity zero instead of like zero one and I specifically do not use negative infinity and use like a very low negative number instead because there will eventually be cases where your entire rows are just zeros and if all your rows are zeros during a softmax that means the output of softmax is going to do zero by zero this is going to lead to NANDs or numerical instability and if you get that as a NAND then the output loss is also going to be a NAND not a number and this is just not trainable and so I just instead just inject very little information into it and this is effectively not going to change your model too much and hence I do it here the output of this though and you can try this out is that well you'll get an encoder self-attention mask as zeros means that pass through until the character that you see and then you'll just get negative infinities then the decoder self-attention mask you'll see that it's a look ahead mask so only the first one is zero everything else is going to be you know negative infinity here only the first two in the second row only the first two are zero in the third row only the first three are zero and so on and then we have like a decoder cross-attention mask which is kind of more similar in that it has a padding mask like the encoder self-attention mask here I've put all of this actually together in a very cute class of a function which we call sentence embedding where I have this wonderful set of operations that I'm going to be performing and I also have this batch tokenization so we have a tokenization function everything that I've written out I've encapsulated a function to handle batches of data all right so now that we've taken a look at a good portion of different components of how to code out the encoder the decoder and even parts that come before and after it let's actually take a look at the entire transformer code end to end for the encoder and decoder and in this case though and we'll see that there's like 300 lines of code I'm going to explain all of this although we've already done deep dives in different parts of this code separately this section is just going to marry the two so it's not going to be a super long section but it will give you better context into how everything is connected with code so let's get to it so as a reminder we are currently in the process of building a translator from a language called English to a language called Kannada which is a language that's spoken in a specific state in India and the way that we're going to generate the translation for an English sentence is we're generating one Kannada character at a time and not one word at a time or one byte parent coding at a time as you would see probably some other implementations do so you pass in English sentences over here and for each character we will get a vector that encodes some contextual meaning to it and we'll take these vectors pass it into the decoder architecture along with the Kannada vectors which are transposed like by one word there's going to be an additional start token preceding the sentence and then we'll have during training the entire Kannada sentence but without the start token and so in the end this is a way to parallel train the model so that it's much faster in training and during inference time though we would just use the start token and pad the rest while passing it into the entire decoder architecture to generate the first Kannada word that word is then taken and passed into the output section whatever it says here in order to generate the second word and we do this until we get the end of sentence token to generate the full Kannada sentence one character at a time now with that context out of the way we can actually look at the code so this here is the transformer class that implements or extends the torch modules we have D model which is going to be the dimensions for every single character vector typically in this case we take it as 512 FF hidden is the hidden layers for the feed forward layers that we see throughout the network which is 2048 num heads is the number of heads that we see in the multi-headed self-attention and also multi-headed cross-attention drop probe is used for dropout as the probability that we will switch off neurons to promote generalization through the network this is typically 0.1 num layers is the number of encoder layers as well as the number of decoder layers and we can set this to be like 5 or something max sequence length is the maximum number of characters in a given input sentence which in our case we set it to about 200 kn vocab size is all the possible characters that can occur in the Kannada sentence or translation then we have English to index which is taking a dictionary it's a python dictionary that maps a character to a unique number same thing for Kannada and then we have a start token end token and padding token which would be the start and end and padding tokens respectively now during the transformer call to the forward pass we will be taking in x which is the English sentence or rather more specifically it's the batch of English sentences why is the batch of target Kannada sentences then we have a couple of masks here so we have the encoder self-attention mask which is going to incorporate the padding masks then we have the decoder self-attention mask which is going to incorporate both the padding mask as well as the look ahead masks and then we have the decoder cross-attention mask which will also incorporate the padding mask and these four other parameters are going to determine whether we should include a start token and an end token to our encoder and decoder inputs and outputs in this case false means that we do not include it and true means that we do so we'll first start by taking the batch of sentences x passing it through the encoder in order to get the list of character vectors that are eventually going to be context aware we then take those vectors into the decoder along with the batch of Kannada sentences which is why in order to finally get the output of that decoder and we're going to be eventually passing this into a softmax activation when we're computing the loss function if we take a look at the encoder so this x here as the input is going to be the batch of English sentences we're going to construct the sentence embeddings this is going to eventually take every single sentence we're going to use the character we have to map it to integer values for every single sentence and every single character in the batch and then we're going to pass it through the encoder and the self attention mask is going to be just the padding mask because we in the English sentence are allowed to look forward and look backward so we don't need that look ahead mask here and so we're just going to get a list of contextually aware characters in the form of vectors the decoder on the other hand is going to take these list of vectors x but also the batch of input sentences y we're going to perform the sentence embedding that is we're going to perform like the tokenization of characters and then we're also going to include whether we include like a start token or end token and this depends on whether it's the input to the decoder or the output of the decoder so the input to the decoder is only going to include a start token but the output is not going to include the start token but it will include that end token we then pass it through the decoder passing in the self attention mask and the cross attention mask and then we get the output which is a set of Kanada character vectors so that's the entire transformer neural network code and now we want to use this code in some way so I've put together a trainer notebook which you can use to train a neural network I've already gone ahead and trained it on the task of English to Kanada we will go through that trainer notebook code step by step and we'll also go through some fun inference translations just to see how well the transformer neural network did so let's get started before I get started with this video I wanted to give a quick shout out to this user account Tiger07 where they helped point out a specific issue where in my last training videos for building out the transformer I kind of made some errors they pointed out like two lines of code that I kind of needed to change throughout the network and this is the line itself it was just reshaping up these tensors because I think it was PyTorch when I tried to do I was just doing a values like some reshape which would have completely discombobulated all my tensors I needed to do a permutation of some layers it's a very minor technical issue but it was a major one in the sense that it stalled me for a very long time and I'm really grateful that when I reached out to the community you all responded very well I also wanted to give a shout out to this account PingNG sorry for mispronouncing that if anything who recommended the same exact solution but also detailed why that was the case so thank you so much as well and that just shows how well this community does respond because even Slack nor ChatGPT could really help me out but you all did so thank you so much for now let's just go to the part where we're actually instantiating the transformer and the only thing I really want to point out here is that we are going to use only one encoder and one decoder layer so this is going to be the simplest of the simplest transformer neural networks so that it could just train this faster to see some reasonable results that's the only reason I did this and I did this training for about one and a half hours where I trained it for 10 epochs where I'll scroll here you see that's 10 epochs with the data set of around 200,000 English sentences to translate into a language called Canada and all of this training epoch I've printed out 100 epochs at a time for now let's get to the good part of transformer inference so this here is a character language model where we're generating one character at a time and so you'll see everything is going to be within a for loop where I am generating one character at a time you generate the token and append it to the sentence and just keep doing this until you generate an end token which will signify the end of a sentence on doing so you'll get a few examples so let's just look at a few of them when I say to try to translate what should we do when the day starts the the translation it gives is this sentence over here which says what do we do about this although it doesn't translate exactly to you know the sentence at least we can kind of see that there are some commonalities that are retained like this last two words over here and these last two words over here it is retained in some way so that's these two words that just means what should we do and that just corresponds to this part of the sentence so it kind of gets a part of the sentence right but very clearly it distorts the entire meaning so it's not quite getting everything correctly you can attribute this to a few cases so the first thing is like the model is just too simple it only has like one encoder and one decoder and if you increase the number of encoder and decoder components you probably might be able to pick up on more idiosyncrasies in language that's probably the biggest reason but you can also try increasing your training set or increasing the number of epochs for your training time now the second sentence is how is this the truth the actual translation is idu hege satya whereas this here is the generated translation which says hege idu hege hege so this is not really a meaningful sentence but you can see that there are some commonalities between them again so you can see this idu hege is generated as well idu hege so idu means like this hege is how satya is truth so it translates to how is this the truth so it does generate some part of it and you can see just by looking at these two examples you can kind of see that it is definitely learning something although it might not be complex enough to pick up everything about these sentences now this example is my name is a j which should translate to but it translates to so you can see again there are some commonalities with which means name none is my but it's very close to it but it means me it did not pick up at all on my name at all also although the overall translation is off once again we see some words that are actually common and correct now this one's interesting why do we care about this the actual translation that it gave was with punctuation it'll be like why what's the reason that would be the actual translation of this which is actually very close to the translation of this initial sentence over here so not bad it did pretty good there the next is this is the best thing ever whereas you know here it generates this sentence but actual translation is this sentence which translates to this is very unusual so though the the meanings are kind of off you can see that there are just some some commonalities between them again now this is probably the most interesting example throughout the lot where I wanted to translate I am here the actual translation is the projected translation by the translator was so there are different meanings this means this means I am here whereas this means I've heard something and although there are different meanings though you can see that from a character by character generation that this translator is performing it's actually doing extremely well the only thing in the translator's eyes that it got wrong was this e and this k like these two alphabets are like the only thing that are different in this entire translation boring some small like la this they're both loves but either way just one or two characters are the only things that are wrong so in the translator's eyes this translation is actually a pretty good one but this kind of made me think more about the fact that okay this is a character translation but in general word translations might actually perform better but the the caveat of using word translations is that your transformer will need to have a much larger vocabulary as opposed to what it has now so you can see if I scroll up to see like what is the length of any possible characters that are possibly generator tokens that are generated you can see it's only a small set of values here maybe like a you know maybe a 50 to 100 tokens or something like that but if it was words then it all possible words were in this list this entire list would explodes to the tens of thousands because there can be so many words that need to be generated so there's always this tradeoff between larger vocabulary size but interpretable values so you need like much more complex systems like you need probably a more complex translator and also way more parameters in order to account for you know words themselves but with words though the sentence length will technically decrease while here I have provided like the maximum sentence length to be 200 characters the number of words in a sentence doesn't have to be like 200 characters 200 words it could be just you know a dozen or something like that that we can cap it out here's an interesting one too it says click this and this would be the translation which is either new click Mardi which is like this but the actual translation is kyan click click click click click click click click Mardi so although it does get this last part right click Mardi and it does get it here it just loves click click click click click click so it's just funny but it is once again understanding what the task is at least to an extent now the same thing is here where is the mall the translation is well the translation that it generated was yeli yeli yeli which is where where where so at least you got the where part is but it didn't generate anything else now what should we do the translation is enumarabeku now it generated this correct but it absolutely fumbles on this one here today what should we do this I have no idea why it generates this it says adannumaridara mele marida maridare so it just loves marida mari is the is to do and I guess that's like a very common phrase that you see everywhere in both English and also in this language kanada and that's why you're seeing like all kinds of forms when you see like oh do it probably tries to do mari like every single everywhere it just tries to to create this scenario which again very interesting but it completely fumbles despite you know in English when we see this it's kind of like the very same sentence as what we did before just an extra word so that's just a it's an interesting note but if I phrase it as why did they do this this sentence actually generated almost perfectly well but again this is something to do with doing something so it's a very common common sentence in general or a common phrase so that's probably why it did so well this last part here is also a very interesting one it's what's the word on the street and the generated translation is either a bug game so what is the topic of this or what is this about is the translation which kind of does semantically relate to to what this actually means this little idiom here let's now go through some insights where I'm probably going to give you some information and some tips when building out a transformer on your own with any language so first of all I want us to say create a translator with a language that you understand ideally because it's just so much easier to see where the transformer is doing things correctly and where it's also doing things incorrectly so generating that insight for yourself I think is very important and you can better do so if you understand the language itself so in this case people were saying why did you use canada it's because I can understand and I can properly evaluate it otherwise I wouldn't be able to come up with the insights that I did piggybacking off of that here's a I think a pretty important insight that I haven't really seen anywhere but I'll describe here so when training typically the English character set is known as what we call an alpha bet where every character kind of has a phonetic representation to it in a language like canada it's more of an alpha syllabary so you have individual units to actually be complete like syllables themselves and in doing so that means that even though like for example this word I think I typed it out here so this is a character this is ma ma if you write it out in English should be m a with like a you know like an accent on top of it that would be so it's like multiple characters in English but it's a single character in the canada language however when I was dealing with tokens here the way that I'm tokenizing the data is I'm also treating it as like multiple tokens so it would be ma plus a even though in the current language it's actually supposed to be one character I am treating it as two characters and so what semantically just makes more sense is to create a tokenizer that will not just divvy up the entire you know canada like word into very sub characters but rather divvy the canada word into actual canada characters themselves which may or may not be a combination of two or more of these characters also alpha syllabaries are a type of script that are not confined to just this language canada there are many alpha syllabaries out there and so just understanding the writing style may actually create a translator that is more meaningful and so I highly recommend you try this out now another the another insight that I mentioned is kind of similar to what I described before this is a generating one character at a time so it's a smaller vocabulary but longer sentence length but you can play with generating like word at a time where you'll have a much larger vocabulary but smaller sentence length and a good mix of both worlds is to use something called byte pair tokenizations or byte pair encodings which are like sub words now the issue with this is that it's very hard to create a byte pair encoding for certain languages if they don't you know if they're not really don't have a great researcher online presence so it's hard for me to find one for the language canada and hence I went with character tokenizations for now to illustrate concepts and ideas but if you're able to create like byte pair encodings for your languages input and outputs then I think that might be like a good starting point in fact I think this is exactly what's happening in the main paper and a lot of other research associated with generative models these days another one is to make sure that your training set has a large variety of words in general you could see that above when I illustrated here there's a lot of sentences that are like to do right moddy is there's like a lot of these sentences that that kind of go moddbeco and moddy over here in fact if you look at this data set there's actually 10,000 cases at least of like there's like millions of records here but there's like 10,000 cases at the very least where we just have this entire word called moddbeco which is I have to do and that's a very common phrase so I would suggest you try to plot out every single word and their frequency counts just to get an idea of what kind of data set you're dealing with whether it's very catered to like news government articles politics or if it's catered to just like general and random sentences which ideally would be the case for general translators and the other one is just more technical where you're increasing the number of encoder and decoder units as I've only used one keeping it very simple but you can ideally try with more encoder decoder units to pick up more complexities and intricacies in your languages overall yeah this model has definitely learned something and you can use it for you know other languages instead of kanada as well and that's going to do it for this series so I hope that right now you have a good idea of how the transformer works its code its architecture and all of the code for the entire series along with the diagrams are going to be in this github repository which I'll put the link to for in the description below once again thank you all so much for 100k onwards and upwards as the corporations say and I will see you in another video bye bye


TLO what's poppin? We are on kick KICK.com. We are live by the time you see this We won't be so just leave a like comment subscribe turn on your post notification bells Let's continue to grow the family from Chicago to the UK This is The live channel above me if you happen to miss alive or anything of that nature I'll be posting shorts from the live and things of that nature on here These same shorts be on tiktok as well. So, you know, either way y'all can go. I know I made a new tiktok channel I don't even know if y'all peed Well, it's all right. It's called TLO streams. I ain't gonna advertise it or anything I just made it to put clips and things about me. Anyway, man, don't forget. We do got merch We also got the patreon and all the links to this is down below man. This is the lives of Mae Bukhari tiktok tiktok Influencer convicted of less to sheer double murder tiktok be creating murderers Look comfortable, okay In the Audi Maybe Kari was a rising social media influencer, but when she set up two men and he resulted in them being murdered She turned and everybody. Wow. So she the backdoor queen Somebody got her to backdoor ops. They ops and do that She conspired with to do this crime herself and her mother lied to the police on several occasions And her police interview which has now been released shows to the extent that she believed She was going to get away with the double murder of these two men Always done that so this car's come not close. I've always braked. I was doing 60 It was 70. I think it was 70 on the dual-carriage or 60. I was doing 50 straight away went back Went to 50 to 60 went back. This car didn't do anything to me Did I don't know who ditched the normal car going on the right to the left? The silver car don't know which come I don't know which brand it was model. It's come zooming down I've kept this second this second to three second rule gap and then the blue liuns in front and I'm at the back This silver cars come and cut into the middle. So we're basically all three cars very close I've braked and I said to my mom. What is this car trying to do? And my mom said you may just keep your distance everything's going to be okay It's okay just a reckless few people just record because they know it's all the way down There's any cameras going there's going to be cut. They know that they must know there's no cameras The victims were Sakeeb who said yes, he was on point though. She ain't stuttered really she was she definitely lying But that or maybe that part of the story was true But I'm a Hamid is that you're doing They were both age 21 and they died instantly when their vehicle was rammed after a four six in Leicester During the early hours of the 11th of february last year They crashed into a central reservation barrier and a tree and they died instantly for multiple injuries The car then burst into flames and minutes before this mr Hussein who was the passenger in the car had made a call to the police Stating that he was being chased by people and they were about to ram him off the road and he was going to die A murder investigation was launched and Mae Bukhari's mother answering Bukhari was a Hamid off the road and he was going to die A murder investigation was launched and Mae Bukhari's mother answering Bukhari Is this the mom or meh or or or meh? That ain't the same person that we just seen was it the influencer I know I'm supposed to be talking this seriously, but god Jesus Who is this This the mom. Oh, that's not that that's not it's okay That's the mom, okay Sorry, mom. So okay was arrested alongside Rhys Jamal Rikon Karwan, Sanef Gullah Mustafa, Amir Jamal and Natasha Akhtar from Birmingham alongside Mohammed Patel They were all charged with the murder of mr. Hussein and mr. They bogus. Why though what they do He jazzled in the first trial fell apart and the judge reordered another trial And it is only now after eight weeks at least part and the judge reordered And mr. He jazzled in the first trial fell apart and the judge reordered another trial And it is only now after eight weeks at least the crown court Four of them have been found guilty of murder and four of them So this is the influencer girl And this is her mom And this is the two dudes Guilty of manslaughter Mohammed Patel was found not guilty of manslaughter or murder The police discovered during the investigation that mr. Hussein was having a relationship with answering bukhari This had been going on for around three years But answering was married and she tried to end the affair and mr. Hussein had made threats to expose her to a husband With sexual images and videos that they had shared together Answering had said she would pay money to mr. Hussein that he had spent him in the relationship and also to get rid of the images Mr. Hussein had agreed to meet her in a car park in Leicester Mr. Hussein arrived in the car park in the Skoda Fabia that was driven by his friend Mohammed CCTV showed the Skoda driving into the car park But immediately leaves when they realized that this has been a setup An Audi TT car and a sea lion car followed the Skoda out of the car park two minutes later The cars then immediately started to chase them and this is when mr. Hussein made the phone call to the police The collision itself was never caught on camera But during May's interviews she made a lot of different claims Involving the fact that she knew nothing that was going on Inquiries at the scene by the forensic collision crash investigator established at the Skoda Fabia had hit the essential reservation and collided with a tree The impact was so severe that the car split into two and the engine was detached from the vehicle Mr. Hussein and Mohammed died instantly from multiple injuries before the fire took hold The detailed police investigation retrieved hours of CCTV from different locations And this included the lead-up to the incident that occurred a tracker was fitted to the Audi TT And this was a higher car and it showed did it read so okay, so Her mom was having an affair with a younger gentleman Who then wanted to blackmail her mom? So they agreed to give her money give her money when he came to collect it was a setup And they chased them And they turned the chase turned deadly That's what i'm hearing Each speeds of up to 100 miles an hour during the chase And the speed of the skoda at the time of the collision was estimated to be 80 miles an hour The tracker on the Audi TT showed heavy braking after the crash and reduced its speed to 62 The Audi and the Seat were said to have continued across the A46 before reaching a junction where two people were getting out of the vehicle They inspected the damage to the car before getting back in driving on both the Audi and the Seat Leon Were found to have driven past the crashed vehicle before and later to drive back past the collision site None of them called the police Answering Bukhari and Maheq Bukhari and Akhtar were arrested on the same morning of the collision Reached Jamal and Karwan were arrested days later and Amir Jamal and Gullah Mustafa were arrested several months later Seen the investigating officer in the case Mark Parish said were arrested Karwan and Akhtar So it just seemed like her the mom confided in the daughter and that's the daughter friend, right? The mom confided in the daughter about what was going on And the daughter when it got to get her her male friends It was like, okay. Tell them this this and that we're gonna set them up They're gonna we're gonna set them up because he's trying to blackmail you Is that what happened? That's what happened. Okay In my mind Ah, we're arrested on the same morning of the collision Reached Jamal and Karwan were arrested days later and Amir Jamal and Gullah Mustafa And they the guys were there just to like scare them and maybe jump them maybe But I genuinely don't think they tried was Like wanted to ram them off the road like or maybe they did Arrested several months later. Seen the investigating officer in the case Mark Parish said this was a callous and cold-blooded attack That ultimately cost two men their lives after setting Mr. Hussein and Muhammad up Chasing them at high speed and then ultimately running them off the road None of the defendants made any attempt to help them or call the police Instead they all drove off and drove past the collision site and still offered no help As the defendants found guilty were arrested charged and stood trial Their lives continued and they tried to cover their own tracks and gave no concern for the victims families So they've gone all the way down. They've come caught into my my car. I've kept my desk library I was I think gotten to that is 40 because there was no cars behind me don't have 40 this car was not threatening me or agitating the blue car agitating the blue so the blue car went on to the right lane, which is the overtaking lane And the silver car went on to the right lane with the same car as the blue So then the blue car went on to the left and the silver car kept to the left And it was just harassing like the car the blue car was trying to like trying to let you go Go go go you can go now because I'm letting you but this silver car was not letting the blue car go So she's saying she doesn't know what happened at all and she was just randomly on the highway and seeing this stuff Okay. Well, they're not agitating me. They're not trying to put me off my driving. I'm going on my left lane. I'm further away from these cars So like I said, they were doing their own thing. I've gone past the on the right lane now the blue car's in front always The silver car was behind I've gone past and I've gone further away from this car on the left side And then the blue car has caught up with me on the right side I don't know who they are. I didn't even see the silver car as well And the silver car all I hear is like a Like a you can hear like a bang, but not a big bang.  What is going on? What is wrong with are they are they? I don't if they're drunk are they what are they trying to do to this blue car in general? So mama was made All his hand movement, ma'am You aren't really telling this story. You really want to convey your point I leave it. You're not involved. You don't know what's going on. You just go straight on and leave it I was okay. That's fine. This blue car has gone straight now And this silver car has gone behind And as soon as we're gonna ask his mom as soon as we see a slip road, we're gonna go We're gonna go to a slip road We've gone forward and the silver cars hit the back of the blue car the blue car See it like Leon's gone completely gone As soon as I've turned around the silver cars gone Out from the I didn't see all I've seen is they've hit and all I see is Right like a Silver cars swerved He said that the families of the victims in the past 16 months have been through a lot And the trials that they've had to go through have been heart breaking and he said and the police said their heart felt condolences to the families and that they'd delete her tiktok Thank them for their cooperation throughout the investigation He was made too. Definitely lying mother's affair that led to a catalyst of events that led to the death of two men That didn't deserve to die and were far too young. This has also resulted All of her daughter all of that to hide an affair is crazy. You was cheating just man up or a woman up Facing life behind bars for the choices that they made The moment that may suggest into her mother that she could get people to hurt these men that were told you I knew it. I knew it may suggest that She went to blackmailer was the moment that she decided to become a criminal Was no longer a victim of crime her social media influence is now long put a myth and a dream in the past And I don't may back up back up But buh-buh But car all it pop up A while it's still there D.A. She was going crazy. Like you did all of that and through all of that You ain't been you've been locked up since 2022 That's tough And she threw everything away that she potentially could have gotten the future By making these decisions that have affected the rest of her life and changed other people's lives forever Sakeeps family said that he was a very young man and he was kind and compassionate Caring and sensible and they are struggling to come to terms with their loss They are grateful to the court for everything they did to bring justice through their family And sakeeps death has brought so much sadness to their family and so many people that knew him and cared about him Everyone who knew Hashim the driver of the vehicle that crashed Said his death was a massive loss to the family and the whole community that he was from He was a cheeky young person and he was a very handsome man And he had a beautiful smile and was also good inside Hashim would always put others first and this is what he was doing that day When he agreed to help his friend to go to the meeting to make sure that he was okay So when I said my condolences to the family of also Hamid and just to really remember how easily he is for you to throw your life away and to risk it all For something that really isn't worth it So really appreciate you joining me today and thank you for following this story of the past year And I really want to hear what people have to say and please don't Forget to pay your respects in the comments I'll be back again very shortly with some more news and follow on social media as well at scarcity studios And don't forget the web.   How much they got 25 for this They got sentenced already I don't know. Let me know in the comments if they got sentenced I'm gone


Hey guys, welcome to another episode of RT Trio. Now, you will notice if you've been following RT Trio or my channel for a bit that I haven't done these in one, two, three. This is I, and my husband at around Thanksgiving got the virus, you know, which one I mean, and it's taken until recently for us to feel more like ourselves. We're not a hundred percent, but pretty close. So anyway, rather than just skipping the last two that I missed and going to the next one, number 30, you know, I thought I'd just play catch-up and why not just do all three in one video? Now, we will probably fast-forward through most of it, but if you have any questions, comments, or concerns, please do leave them down below. If you want copies of these mood boards and any others that we've done already, join the RT Trio Facebook group. I'll leave the link down in the video description along with the links for the other, sorry, that's my phone, along with the links for the other artists that are creating this month. We do have a guest artist that helped out quite a bit while I was ill. Thank you so much, Jamie. I so appreciate that, and if you want to see what she's done, again, go join the Facebook group because it's all over there. Now, normally I do mine mood boards in a journal like this where we do more like art experiments, where I play with color and shape. But, you know, I'm really feeling like, and if I don't do this kind of art journaling work, I do more art therapy work. And I do use my creativity, whether it's painting, drawing, journaling, needlework, sewing, crochet, whatever it is, I use it as therapy. I have anxiety disorder and depression, and it's a constant thing I'm dealing with. And the holidays between being sick and other stuff that's happened, they've been a little challenging. So I'm thinking we are going to go to an old journal I've had sitting on the shelf for a bit. This little mini-composition notebook, and this is really something I've used for therapy quite a bit in the past. Now, all three of these mood boards have something in common, and that would be their sort of wintry, blue, gray, little bit of tan colorway. So I happen to have in the back of this some pages that already have some blue stuff on them. There's actually quite a few of them. So I've got my paints. I've got some bits and pieces. I've got some pens and some glue and my heat gun, and we are going to work on our mood boards here. I'm going to actually do like I would do in the big book, and I'm going to glue the mood board to the opposite page, and then we're going to work here. So I'm going to do that for all three, and I'll see you back when we're done. Okay, when I went to go grab for the stapler, I think I forgot to turn the camera back on, but you caught most of it. I stapled on the paint swatch that I drew on, and the leaf and the flower, and then I found this little tea bag quote. I like to drink yogi brand tea because I like the way they taste, but also all the tea bags have a little quote of some kind on them. I added some butterfly washi tape. I don't think you saw that. I have lots of washi tape. And I added a little bit of paint highlights to our face. I am going to touch up a couple of the black lines. Just a little bit. It doesn't have to be perfect. It's not about perfection. This is about expressing myself and my creativity and working through things that, you know, set off my anxiety and stuff, and making me happy, as it should be, with most things in life. So this is my take on this month's mood board. I really like the way it turned out. This is the month before. I love that one. I'm not sure where all of that blue, orange and yellow came from, from my mind. It was, of course, inspired by the mood board, but yeah, I love the way it turned out. And then this is the one from around Thanksgiving when I got sick. So they are all a lot of fun. Sorry, my phone keeps going off. They are all a lot of fun. And it's a great way to get prompted for art journaling when or any artwork, honestly, you could use these if you want to, you know, maybe you want to crochet in Afghan and you're like, I have no idea what colors to use. You could use a mood board like this and pick some blue, some tan, some gray. It would make actually a very pretty Afghan. But they're intended for art journaling and you can do something more fine art and painting as we've done in the past where we've done sort of just painting experiments in a book like this. This one here is pretty abstract or you can do something more, you know, layered and thicker and more interesting. I like this a lot. There is a YouTuber called Berber Van Gorp. That's the name of her channel anyway. And I really like the way she does her art journaling. It's very abstracted and layers and all about just how she's feeling. And I'll link her channel down below, along with my other co-teachers. So I'd love to see what you do. Don't forget to tag me in your post, whether you do a YouTube video or you join the Facebook group and you put a post over there, tag me in the post so that I can see it and think about joining the group because it is a lot of fun. We have a lot of fun over there and create some art, have fun with it, express yourself. You don't have to be crazy like me and show the whole planet. Express yourself in your journal, close it up, stick it on a shelf. Let whatever is rumbling around in your mind, keeping you from focusing on what you need to focus on out in your journal. And then you can close it up, put it on the shelf, put it away, at least for the moment and focus on those other things in life. I think it's a good way to go. Anyway, that's it for today. Don't forget to go out and have a good day. Do something nice for yourself because you deserve it. I'll see you next month. Bye, guys.