والتواصل العام is brought to you by Caltech. Welcome back. Last time we introduced the third linear model, which is logistic regression. It has the same structure as the linear models where you have the inputs combined linearly using weights summed up into a signal and then the signal passes through something. In this case it passes through what we refer to as a soft threshold, we label it theta. And the model is meant to implement a probability that has a genuine probability interpretation. And because of that, the error measure we derived was based on likelihood measure, which has a probabilistic connotation, in which case we maximized the probability that we would get the data set that we got the outputs given the inputs based on the hypothesis that is represented by the logistic regression being assumed to be the target function identically. And this makes us able to express the probability in terms of the parameters that define the hypothesis, which are the weights w. And therefore, we have this quantity that we want to maximize. And then we derived an error measure that very much parallels the error measures that we had before in terms of the in-sample error for logistic regression that we will minimize. So this is a useful model, and it complements the other models. One of them was for classification, one of them for real-valued function regression, and this one is for bounded real-valued function that is interpreted as a probability. One of the key issues about logistic regression is that because the function, the error measure is a little bit more complicated than we had, for example, in linear regression, we were unable to optimize it directly. And therefore, we introduced a method that is meant to minimize an arbitrary non-linear function that is smooth enough, twice differentiable. And in the case of logistic regression, although we don't have a closed-form solution, the error measure actually has a very nice behavior. It's a convex function. And therefore, when you apply a method like gradient descent or other methods, it is fairly easy to optimize because you just fall into that minimum and stay there rather than have problems with local minima that we talked about briefly. So the algorithm for gradient descent, regardless of the error measure that you are trying to minimize, first you initialize. And in the case of logistic regression, initializing to all zeroes was fine. We will find out that today in neural networks that will not be fine and will make the point y. And then you keep iterating until termination. And what you do is you update your weight gradually by going along the negative of the gradient. That would be the steepest descent in the error. The biggest gain you would get for a fixed-size step. And in this case, we adjusted the fixed-size step so that it's a fixed-learning rate that is proportional to the gradient at that point. We keep doing this. And then when we arrive at termination, we report that as our final hypothesis. And we talked a little bit in the Q&A session about the criteria for termination and also about local minima that will become an issue for today. So today, when I modify the gradient descent into the more practical version which calls stochastic gradient descent, we will talk a little bit about initialization and we will talk about other aspects that have to do with local minima and whatnot. So today's topic is neural networks. And historically, neural networks are responsible for the revival of interest in machine learning. They have a biological link that got people very excited and people pursued them. And they were very easy to implement because of the algorithm that I'm going to describe today. And they met a lot of success in practical applications and got people going. Now, it is not necessarily the model of choice nowadays. Probably people will opt for support vector machines or other models. Yet, every now and then, the neural networks would do the job as well as the other models. And many industries use it as a standard. For example, in banking and graded approval, neural networks are often used. So the outline for today is very simple. First, I'm going to extend gradient descent into the special case of stochastic gradient descent that is used in neural networks. And then I'm going to talk to the neural network as a model. What is the hypothesis that it is implementing? And I'll motivate it from a biological point of view and relate it to perceptrons and whatnot. And then we'll talk about the back propagation algorithm, the efficient algorithm that goes with neural networks that actually made that model particularly practical. So let's start with stochastic gradient descent. What do we have? We have gradient descent. And gradient descent minimizes an error function that is function of w, minimizes it with respect to w. And that happens to be an in-sample error in our mind. And it is the in-sample error. And the only thing I would notice here that is particular to the derivation of stochastic gradient descent is that in order for you to compute the error or the gradient of the error, which you need in order to implement gradient descent, you need to evaluate the hypothesis at every point in your sample. So from n equals 1 to capital N, you need to evaluate those or you evaluate their gradient. And that will tell you what is the error is or what is the direction you would go to. Which is normal because this is the error we are minimizing. You'd better compute it. So you take the case of logistic regression and we had a very particular form for that. And now you can see that it's an analytic form and in this case, friendly and smooth. And indeed, you can get the gradient with respect to that vector and go down the error surface along the direction suggested by gradient descent. Okay. Now, the steps were iterative. And so we take one step at a time. And one step is a full epoch in the sense we call something epoch when you have considered all the examples at once, which is the only choice we have so far. Okay. And we had this formula that we have seen. And now, the difference we are going to do now is that instead of having the direction or the movement in the W-space based on all the examples, we are going to try to do it based on one example at a time. That will make stochastic gradient descent. So because now we are going to have another method, we are going to label the standard gradient descent as being batch gradient descent. It takes a batch of all the examples and doesn't move at once as opposed to the other mode. Okay. So the stochastic aspect is as follows. You pick one example at a time. Okay. Think of it as you pick it at random. You have capital N examples. Each of them is equiprobial to be picked. You pick one of them at random. Okay. Now, you apply gradient descent not to the in-sample error for all the examples, but the in-sample error on that point. Okay. That looks like a very meager thing to do because the other examples are not involved at all. But I think you have seen something like that before. When we take one example at a time and worry about it and not worry about what other guys are doing even if we are interfering with them. Remember the perceptron learning algorithm? That's exactly what it did. And it worked. And in this case, it will also work. Okay. Now, to argue that it will work, think of the average direction that you are going to descend along. Okay. So what does that mean? If you take the gradient of the error measure that you are going to minimize, which in this case, just for one example. And you take the expected value under the experiment that you pick the example from the entire training set at random. Okay. So in that case, if you want to get the expected value with respect to the red N, which is now a random variable, this is what you get. And if you evaluate it, it's pretty easy. You simply take this value. For every example, it has a probability 1 over N. And the expected value would be 1 over N summation of that. Okay. So this would be the average direction. So you think that every step I am going along this direction plus noise. So this is the expected value, but because it's one example or another, there is some stochastic aspect. And if you look at the quantity on the right-hand side, this happens to be identically minus the gradient of the total in sample error. So it's as if, at least in expected value, we are actually going along the direction we want, except that we now involve one example in the computation, which is a big advantage. And we have a stochastic aspect to the game. Okay. So this is the idea. And then you keep repeating. And as you repeat, always you get the expected value in that direction. And you get different noises depending on which example. Okay. So the hope now is that by the time you did it a lot of time, the noise will average out. And you actually will be going along the ideal direction. Okay. So it's a randomized version of gradient descent. And it's called stochastic gradient descent, SGD for short. Okay. Now let's look at the benefits of having that stochastic aspect. The main benefit by far, that is the motivation for having this, is that it's a cheaper computation. Think of one step that you are going to do using stochastic gradient descent. What do you need? You take one example, you put the input and you get the output, and then you compute whatever the gradient is for one example. If you were doing the batch gradient descent, you will do this for all the examples before you can declare a single move. Nevertheless, the expected value of your move in the cheaper version is the same as the other one. So there's a little bit of cheating here. On the other hand, it looks attractive. If this actually works on average, this is an attractive proposition. So this is number one advantage. The second advantage is randomization. So there is an aspect of optimization that makes randomization advantageous. So you don't want to be extremely deterministic. You want to have an element of chance. Why would I want an element of chance if I know my goal exactly? Well, because optimization is not exact. It's not like you are going for the minimum for sure after that. There are all kinds of traps that you can go through. Like local minimum or not. So let's look at cases where randomization would help. This is an error surface.  The one you encountered in logistic regression was simply like this. That was a lucky one. The convex one. In general and in neural networks for sure, you are going to get lots of hills and valleys in your error surface. So depending on where you start, you may end up in one local minimum or another. You may not get the best one. You may get one or the other. Now, this is inevitable and there is really no full-proof cure for it as we discussed in the Q&A session. On the other hand this small fellow because it's really just like a shallow local minimum but according to gradient descent you go here. The gradient is 0. Everybody's happy and you stop there. So you would love to have an added element that will make you escape at least shallow valleys like that. And the idea now is that because you are not going in a direction that is deterministic in this case there is a random. There is some fluctuation here. So there is a chance as you go here from the local minimum. Now this is a practical observation that in reality stochastic gradient descent does help with this. It doesn't definitely cure it far from curing it. On the other hand it does take care of some aspect of escaping silly local minimum. So this is an advantage that basically is a side benefit. We did it for the cheap computation and we are getting this for free. The other one we also talked about a little bit in the Q&A session which was the flat region. So you could be very, very, very very flat. And then finally going down.  Every now and then when you do the random things the fluctuation take you up and down and the the algorithm is still alive. Still termination is a tricky criteria because for termination you need to consider all the examples in order to know exactly where you stand. But for some of the flat regions just the stochastic aspect also helps it so there are basically annoying artifacts of the optimization of a surface that that gradient descent will help a little bit if you use the stochastic version. Okay. Now the third advantage show that the third advantage you have is that it's very simple. It is the simplest possible optimization you can think of. You take one example you do something and you are ready to go and I will see that applies it. And because it's simple there are lots of rules of thumb for it so people have used it a lot and people use it in different application so you can find rules of thumb that are actually very useful. So I'll give you one rule of thumb that will be helpful in practice. Okay. So remember the learning rate. So the learning rate was telling us how far we go and we talked about that if it's too big then you use the linear approximation if it's too small you are moving too slowly and whatnot. So sometimes what should I use for ATA the learning rate? Okay. And obviously the exact answer depends on the situation and even is it's dependent on scaling the error up and down. Mathematically you can't really pin it down. From a practical point of view if you go for a very wide range of application you take a normal application a normal error function means squared or something and then you take ATA equals 0.1 Okay. So this is the theorem ATA equals 0.1 Okay. Then the proof is end. Okay. Okay. So these are advantages so we are now motivated to look into the gradient descent and let's see it in action. Okay. And I'll take an example far from the sort of the linear models and neural networks and I'll take an example that we looked at before in an informal way and it would be very easy to formalize and implement this way. Remember movie ratings? Oh, what was that? Oh, that was the example where, you know, you want a user to look at a movie and do a rating and you want to look at previous ratings and predict all of that. Okay. Now it looked like this. That is the the proposed solution that we will describe the user by a number of factors which are basically their taste. They like comedy they like action they hate this, etc. So there are some values here describing their taste a profile of the user if you will. And then a movie you describe the content with the same factors. Does it have comedy? Does it have, etc.? And the idea now is that we are going to reverse engineer the ratings the existing ratings in that training set into factors that explain why this rating is and hopefully by the time we do that we will be able to predict future guys. So I do this for the movies that this user saw and then I will take the factors of the user the factors of the movies that they haven't seen and do the same combination that I did here and hopefully get a prediction for the rating. Okay. So all I want to do here is show you this method using stochastic gradient descent which was actually the method that was used in this solution in the million dollar price. Okay. So it's I mean although it's very very simple and whatnot it is actually used and if you are working for something with the sticks that high you probably will try your best to get something right. So the fact that actually stochastic gradient descent survive until that late stage tells you that it's a trivial algorithm. Okay. So in order to put some formality on this we need to give labels for the users and movie. So it will be user i movie j and the rating we will call r sub ij very simple. Now there are factors for the users and factors for the movie. So let's call them something. The factors for the user will be u1, u2, u3, uk so it's a vector of numbers that describe the corresponding factor for a movie would be v1, v2, v3 up to vk which are describe the content of that movie. Okay. So when we said we are going to match the taste of the user to the content of the movie what we were going to do we are going to simply take a coordinate k from k equals 1 to capital K and multiply these two so we are taking an inner product between these two guys the level of matching between the two that is the quantity we are trying to make replicate the rating so we'd like the difference between the rating and this quantity to be small that's the goal okay now in order to be accurate in the notation the factors u1 up uk and v1 up to vk depend on which user and which movie different users have different factors etc so I'm going to add the label of the user okay so it's you know more elaborate notation but it's not a big deal and we also introduce it here in the sum okay so this will be exactly the case and for all the users and all the movies you have you know a shuffle of different users rating different movies so the factors are reused for different ratings that appear in your training set and now your idea error on that particular rating which is the difference between the actual rating and what the factors the current factors suggest so the factors now are your parameters and you are trying to find the value for the parameters that minimize this because you are taking one example at a time if you do descend on this one it will be stochastic gradient descent if you wanted to do batch gradient descent you'll have to take all the ratings add up these terms for all the ratings you have and then descend on those but the stochastic gradient is the one which is used could there be anything simpler you are going to get the partial this by partial every parameter that appears here and remember in the first one we said that all we are doing is we take these factors and try to nudge them a little bit towards creating the rating and now we have a principle way of the nudging the nudging will be proportional to the partial by partial each factor so I have in order to get there so now we have the formula and the formula will be as a vector I am going to move in the space that is now has 2k parameters in this case and I am going to move in that very high dimensional space in a direction that makes me with a certain size of step achieve the biggest drop in the error in estimating the rating so you can implement this and indeed if you implement it you will get a pretty good score for the for the Netflix competition and in this case people started adding terms and obviously regularizing which will be an important issue that will come up with but basically the simple gradient descent with very plain squared error on something as simple as that will get you somewhere okay so now we know that gradient descent now I am going to start with the biological inspiration of neural networks because it's an important factor that's where they got their name and that's how they got the initial excitement that got them to be to have a critical mass of work so biological inspiration is a method really we we use in in generic applications the number of times okay and there is a little bit of a leap of faith there which is to we are interested in we want machines to learn okay so in order to replicate the function our first order is to replicate the structure okay that's what we do so we try to make it look like the biological system hoping that it will perform the same okay it is a legitimate approach because you know something is working there's an existence proof and it has this structure maybe the structure has something to it this is the biological system so we have neurons connected by synapses there are a large number of them each of them does a simple job the job the the the action of a particular neuron depends on the stimuli coming from different synapses synapses have weights and so very much similar if you look at a single neuron what we thought of the perceptron okay except obviously I mean the different quantities and not as exact but in a big network we will be able to achieve the intelligence or the learning that our biological system does and we get to replicate it and get something like that in engineering a network of this source and indeed this was the initial thing and we'll get now I'm going to make a single comment about the use of biological inspiration in this way so I'm going to give you another example where we had biological inspiration okay and we will get a lesson from it if you want to fly we look around birds fly let's try to get inspired by birds okay and after a long chain of events we ended up with this okay now there is no question that the the the structure which is what we are going to use made it okay there are wings there is the tail et cetera but once you got the basic structure going if you are an engineering discipline and biology your goal is to understand why the structure does the function and know it so you want to know how biology does it regardless in engineering you want to do the job you don't care how you do it you are just using biology as an inspiration completely legitimate approach to the problem from different perspectives but once you did the initial thing you are no longer than conformal mapping okay and when you get the solution you get a plane that flies but doesn't flap its wings okay so now imitating biology has a limit you have to get an inspiration for what is relevant and then on your own derive what you need so going back to our model here okay we will get this now if I am a biologist I had better because my job is to explain how the biology system is working so if I tell you that it's doing something that is not biologically plausible I already violated the premise here as long as I get the job done I am okay so it is fine to take the inspiration but let's not get carried away we are actually trying to build something that actually does a job from an engineering point of view and whatever works we will take it and that is okay so knowing that the building block is a perceptron and they were putting perceptrons together in a neural network let us explore what we can do with combinations of perceptrons rather than a single one okay and I am going to do this pictorially okay I will save the mess when we define the neural network itself okay so we will just look at pictures of what perceptrons do and how to combine them and we will get the idea that actually combining this very simple unit does achieve something okay so let's look at the famous problem where perceptrons failed okay remember the four points with you know the diagonal plus and the minus one so if you want something that is plus here and plus here and minus here you are out of luck as far as using a perceptron is concerned okay so now we are exploring can we do this with more than one perceptron arranged in the right way that's the goal okay so we look at this we say okay I can get the first this thing with a perceptron I'm going to call H1 that's easy I'm going to get the second one as this and maybe now I can take the outputs of these perceptrons and combine them in a way that achieves this particular dependency okay and you look at it and you say okay yeah that's actually very plausible and your building blocks for doing that are your old fashioned ores and ands the logical ores and ands okay and you think again let's say that I have two Boolean variables zero or one or in this case plus one or minus one okay can I implement an AND which returns plus one if and only if both are plus one or can I implement an OR which returns a plus one if at least one of them is plus one that would be the AND and OR okay can I implement these using perceptrons why? because I am in the game of trying to use perceptrons to build stuff and I'm seeing where this can take me okay and I realize I already have because of the constant term that has a weight 1.5 I'm already ahead of the zero okay so in order for this to actually go negative both of these guys have to be minus one right okay and therefore this actually does implement the OR because if either of them is plus one I will get the signal plus one for this one I'm resisting a negative bias already okay so I'd better have both of them to be plus one if I'm going to exceed zero on report plus one so this actually one AND using a simple perceptron okay so now you create layers of perceptrons based on what you had so in our case we had each one and each two that implemented the surfaces we wanted in the Euclidean space and we just want to combine them so the combination now if you look at it is that you want the AND of each one and each two bar the negative of this and each two basically you are implementing an XOR an XOR wants one of them to be plus one and the other one to be minus one so this is one you want to implement but that is easy because if this is a variable if I have that already I don't know whether I have that already I know that I have each one and I know that I have each two I don't know that I have this funny quantity with the bar but likely I do okay then all I need to do is combine them this way with the OR function and then I will get the function I want so now you do have each one and each two we already established that these are perceptrons so now what you do when you have a weight of minus one it's as if you are negating and a weight of plus one you are leaving it alone so you have minus one and plus one and then you get the first layer to do the AND but not the AND of the the thing itself but the AND sometimes of the thing or sometimes of its negation in order to implement these guys that I want so you end up with these and these guys these are the functions you want here and now you pass them on to the OR and you get the function you want okay so now let's plot the full multi-layer perceptron that implemented the function we want it looks like this this is your original input space this is X1 a real number X2 a real number in the Euclidean space and this is the X0 the constant one okay this is the perceptron and I can implement music a perceptron after I implement music a perceptron I do the the conjunction of one and the negation of the other in order to get here and then I do the OR and get here okay so this multi-layer perceptron implements the function that a single perceptron failed in okay and we have layers so each layer would be this fellow the perceptron okay and this is the second layer and this is the third layer so in this case we have three layers okay we have strict rules in the construction which is feet forward okay so it's feet forward that is you don't get the output and put it to a previous layer and you also don't jump layers it's very logic before you realize that if you can do the the ands and the ORs and the negation you can do anything okay so I can have a very sophisticated surface and just by having enough of those guys and combining them I can get a very sophisticated surface under the restriction of this hierarchical thing okay so that's pretty good so we now realize that just using perceptrons okay so you say okay definitely doesn't look anything like a line okay and I'm using lines there is no transformation here so what am I going to do let me try eight perceptrons just sort of cornering this if I do this each of them would be plus one somewhere minus one somewhere so I have a pattern plus one then minus one and all I need to do is the logical function that will give me where I am inside and where I am outside okay so I am using eight I can go for sixteen and then I'm getting closer and closer to the circle and I can get as close as I want by having as many perceptrons I want and now I have a bigger task of combining the logical results in order to get the final thing I have and indeed you can prove that okay multi-layer perceptrons with enough neurons can approximate any function and whatnot which is very good and for us you know being powerful these are the flags once I give you this is a great model okay everybody would be excited except people in machine learning which wait a minute I have been there before okay so what are the two red flags one of them is generalization okay so I have a powerful model I have so many you know perceptrons so they have so many weights degrees of freedom VC dimension I'm in trouble well you are in trouble but like this you know it has that VC dimension I need that many examples done deal okay so this is not going to scare us it is just going to make us careful about matching how sophisticated we can go to the resources of of data we have so this is not really a deal breaker the real deal breaker for using multi-layer perceptron is optimization even for a single perceptron we were lucky enough to have this you know perceptron learning algorithm that applies only in cases separable and we said in cases not separable it's a very hairy optimization problem it's a combinatorial optimization and it is very difficult to solve can you imagine now the problem when I take layers upon layers upon layers and combining them and now I'm trying to find what is the combination of weights that matches a function you don't know what the function is here I you know you looked at it et cetera but I'm just giving you examples I'm asking you to match how are you going to adjust the weights the only thing they do they have a way of getting that solution and the way they are going to do it is that instead of having perceptron which are heart threshold they are going to soften the threshold not that they like soft threshold but soft threshold have the advantage of being smooth twice differentiable rings a bell oh maybe we can apply the all general gradient descend in order to find the solution and once you find the solution you can say okay I know the weights soft threshold let me heart threshold the answer and give you that answer so that would be the approach okay so let's look at neural networks okay so the neural network will look like this it has the inputs same as inputs before and it has layers and each layer has a non-linearity I'm referring to the non-linearity generically as theta remember theta was used in logistic regression as very specifically the logistic function I'm using it here generically it turns out the non-linearity we are going to use is very much like the logistic function except it goes from minus 1 to plus 1 in order to replicate the heart threshold which goes from minus 1 to plus 1 in the case of logistic regression we weren't replicating that we were simulating a probability that goes from 0 to 1 so it's very similar to this and in principle when you use a neural network each of these guys could be different you can have your different non-linearity and you will see the non-linearity so I could have a label for each of these depending on where it happens and the most famous different non-linearity that you get to use is actually to make all of them this soft threshold and then when you go to the output make that linear so this would be this part would be as if it was linear regression this would be with a view to implementing a real value function so the intermediate things are all these status to be the same and all of them will be these functions that I'm going to describe mathematically in a moment okay so this is the neural network has the same rules it's fit forward there is no going back there is no jumping forward and the first column is the input X so you are going to apply your input X from an actual example to this follow the rules of derivation from one layer to another until you arrive at the end and then you are going to declare X okay the intermediate values we are going to call hidden layers because they are the user doesn't see them okay you put the input there is a black box and it comes output if you open the box you will find that there are layers and something interesting is having in the layers that I'm going to comment about later on okay but these are the ones and for a notation we are going to consider that we have capital L layers so in this case it would be 3 this is the first the third layer with this input this is not really hidden it's an output layer okay so this is the final layer capital L and this is that okay so the notation here will persist with us for that so now I'm going to take this and I'm going to put the mathematical equations that go with it okay in order to be able to able to implement if you want to code this this will be the next slide okay okay it's a soft threshold and we are going to use the tange the hyperbolic tan hyperbolic tangent and the hyperbolic tangent okay well it looks the formula looks more or less the one we had before for the logistic one it's again based on ES and this one happens to go from minus 1 to plus 1 at 0 it's exactly 0 has a slope 1 has very interesting properties and you can see now why we are using it if you take it this way it looks like a hard threshold okay and in the beginning it looks linear okay so it has you know the combination of both words so if your signal which is what you have here this is the signal and this is the output if your signal which is the sum of the weighted sum of your weights is very small it's as if you are linear if your signal is extremely large it's as if you are hard threshold and you get the benefit of one function that is analytic and very well behave for the optimization so this is the one we are going to use what I'm going to do I'm going to introduce to you the notation of the neural network because it's all notation okay obviously the notation would be more elaborate than a perceptron because I have different layers okay so I have an index for that I have different neurons per layer so I have an index for that and inputs go to the output and then the output becomes the input to the next layer so I just need to get my house in order in order to be able to implement this so although this is mostly a notational view graph it's an important to decide to implement neural network you just print this view graph and code it and you have your neural network okay so the parameters of a neural network are called W weights the weights now happen to belong to any layer to any neuron okay and there are three indices that change one of them the different layers and outputs I get okay I have different inputs and different outputs for every okay so the weight is connecting one input to one output in a single in a certain layer so let's have a notation so I'm going to introduce a notation and then apply it to the W so I'll denote the layer by small L and small L as you see appears as a superscript for W okay and then I have the inputs the inputs we are going to call I as an index and obviously since the weight connects an input to an output the I should appear as an index and the output will be called J okay so now my parameters for the network are W superscript L sub IJ although it's you know more elaborate than we have before we understand where it came from now let's talk about the ranges of values for these three indices for L as we discussed L would be go from 1 to capital L okay so from the first layer to capital L which would be the output layer the final layer the outputs go from 1 to D as if it was the D as in dimension so you have I'm going to the Neuron 1 Neuron 2 Neuron D the dimension of the layer that I'm talking about will have that superscript so D superscript L the number will differ from one layer to another okay and depending on which layer you have you will have different number of output units and therefore the J will depend on that okay now for the inputs they come from the previous layer you take the inputs from the previous the outputs of the previous layer to be the inputs so for the size of the previous layer L-1 okay now I left this out because this will not be 1 this will be 0 anybody knows why yeah yeah yeah it's that constant X0 that we always have every Neuron will have that as an input and therefore we will have a generic one which is sub 0 to take care of that okay so for every value in this array you will have W ij L and these are the parameters you want to determine okay okay so now let's see the function that is being implemented what you do is you get the X's in layer L in terms of the X's in layer L-1 right and our notation we will give this a generic index J so this is the Jth unit in this layer and this was the Ith unit okay what do you do in order to get that we do what perceptrons do or neurons in this case okay you combine them with the weights the weights are connecting the I to the J and they happen to be the weights of this layer so when we talk about the weights the weights correspond to where the output is okay you sum these up you sum them up y equal 0 which is the constant variable up to the maximum which would be the maximum for that layer which happens to be D sub L-1 okay so this is the signal now you pass the signal through a threshold in this case a soft threshold and you are ready to go that would be the function you are implementing okay and indeed this would be your the value of the output X and it happens to be theta of we are going to call this quantity inside we are going to call it the signal again and now the signal corresponds to the output so the signal is layer L and the J signal in that layer you pass it through the non-linearity and what you get is the output of that okay so that wasn't too bad now when you use the network so this is a recursive definition so you do this for the first layer second third etc. every time you use it you get the new output so the first you get the output of the first layer you get the outputs of the second these are the inputs of the third and you keep repeating until you get to the final now how do you start this you start this by applying your input the actual input you have to the input variables of the network the input variables happen to be in layer 0 if you want okay and they happen to be called X1 up to that D0 by definition therefore D0 D okay so this guy matches this so therefore that is how you construct the network so the number of inputs is the same as the number of inputs you have once you leave that it could be anything it could be spanning shrinking whatever it is anything it wants okay and when it arrives it should arrive at the value of your output you have a scalar output okay and therefore after long iteration you will end up and since I have one output the J is only one so this is my output of the network and I'm going to declare that my output of the network is the value of my hypothesis that is the entire operation of a neural network when you so when you tell me what the weights are I am going to be able to compute what the hypothesis does okay now our job is to when I put those inputs and look at the target outputs I find that the network is replicating them well okay so that is the back propagation algorithm so let's do so we are going to apply stochastic gradient descent so you take one example at a time apply it to the network and then adjust all the weights of the network in the direction of the negative of the gradient according to that single example that's what makes it stochastic so let's do it okay now the parameters this array which is funny I mean it's not it's not quite a complete matrix because you have different different number of neurons in different layers so this is just a funny array but it's indexed by ijl it's a legitimate array and this determines h therefore what I am doing here is getting the error on example a single example xnyn okay and my error measure between the value of the hypothesis which is the neural network and the target label okay and this happens to be a function of the weights in the network yn is a constant xn is a constant this is part of the training example h is determined by the w that's why this is w and I'm putting it in purple because this is the active quantity now this quantity okay and what is the gradient of this quantity or the gradient of quantity is a huge vector okay each component is partial the error by partial one of the parameters so we put it down so all you need to do is compute this for every ijl okay that's all you need to do okay and then you take this entire vector of stuff and then you move there's nothing mysterious about this if you never heard of back propagation you will be able to do this as we'll see in a moment now the idea is to just do it efficiently and it makes a big difference when you find an efficient algorithm to do something okay for example those of you who have learned linear systems no FFT the fast Fourier transform okay fast Fourier transform is okay you implement the discrete Fourier transform what's the big deal okay the big deal because it's faster you need the field enormously active just by that algorithm and very similar here back propagation if you look at it I can brute force implement this for every ijl but now I have one thing that will get me basically all of these guys at once so to speak and therefore it's efficient and people get to use it and that's why neural networks became quite popular okay so let's try to compute this okay now let me take part of the network so this is L-1 and this is in the layer L I'm looking at the output of one neuron in this layer feeding through some weight into this guy so it is contributing to the signal going into the next guy and the signal goes into the non-linearity to produce the output okay now this quantity is not mysterious if you look at it we can evaluate those one by one that is for every single weight in the network we can ask ourselves okay what is the error well the error is sitting there at the output I have okay I have the output I went further than I should but the the output is sitting somewhere there therefore there is an error and that error will change if you change W and that will tell you what is E by W so we can do this analytically okay there is nothing mysterious I can get the output as a function of the previous layer of the previous layer of the previous layer until I arrive here so I have this function that has tons of weights in it and I'm focusing on one of them and I can say what is partial E by partial this fellow apply chain rule get a number no but not a big deal okay it's not not your favorite activity but you can do that or even you can do it numerically okay I can take this fellow perturb it just a little bit and see what happens for the error at the output okay and therefore I can get numerical estimate for partial by partial now the problem with those approaches is that I have to do it for each one of them what I'm going to do now I'm going to try to find something that will make me get the entire array which is the full gradient with in almost one shot okay so here is the trick the trick is the following I'm going to express partial E by partial W j is the change in E which is upstairs here with respect to this particular parameter I'm going to get it in terms of partial the same quantity by partial intermediate quantity this signal times partial the intermediate quantity by partial what I want this is just chain rule okay but chain rule with partial derivatives you need to be a little bit careful because there may be different ways your variable is affecting the output and you need to sum up all the effects but here if you are looking for how does W affect the error it's W ij it affects only the sum S j so if I get partial by partial S j this is the only link which W ij affect the output and therefore I'm allowed to do this and there is nothing to sum up with respect to so I have this chain rule okay that's nice I can probably look at this and say this is a very simple quantity to get how does the the signal change with the weight we probably can get an easy one there but this one is almost as bad as the original one how does the error change with this signal okay it doesn't look like a great progress but the great progress is that this quantity will be able to be computed recursively that's the key okay so what do we have in this equation will we have the first one because if I take what is this guy what is partial S by partial W okay S is simply the sum of W's time axis so partial S by partial W is the coefficient which happens to be the X and this is the coefficient there and that is readily available so that I have okay the other guy this is a troublesome one so we're just going to call it a name and see if we can get something going for it okay and the name we are going to call it is delta okay so now delta goes with a signal okay so this there will be a delta sitting here if we can compute it and the interesting is that the derivative of the error with respect to this weight which will determine how much you change that weight because when you get the gradient you move along the direction of the gradient means in each component you go in proportion to the value of the partial derivative so since this is the partial derivative the change in the W will be the product of these two guys I'll be proportional to that one of them is X here and one of them is delta here so we'll be changing the weight according to two quantities that the weight is sandwiched between okay and that's a pretty all of those then I look at the X and the delta and the weight in between will change according okay so now let's get delta for the final layer okay why do I get delta for the final layer when we computed the thing we got X for the first layer put the input okay and then we propagated forward until we got to the output the reason we're going to get it because the mass will tell us that if you know delta earlier so this will be propagating backwards and hence the name back propagation so we're going to start with the delta at the output and it's not a surprise because I'm trying to get the partial error by partial something so the closer I am to the action to the output the easier it is to compute it and indeed for the output it will be very easy to compute okay so this is the definition of delta for any value of J and L okay and when you look at the final layer it's not mysterious it's small L equals capital L and J equals 1 I have a scalar function so that is the output layer okay therefore the quantity I'm interested in is exactly just substituting with this quantity I want delta for superscript capital L subscript 1 that's what I want to compute okay now can I compute this let's look at it okay this is E of W I think I'm differentiating what is EW okay EW is the error measure whatever you have between the value of your hypothesis that is the value of the neural network in its current state with the width frozen you apply Xn you go forward until you get the output that is H of Xn you compare that to the target output which is the label of the example Yn and that error will be your E of W the Y is it of W because H depends on W okay so that is not mysterious because H of Xn is what it's the value of the output right and that happens to be the variable in layer capital L variable number 1 that is your output okay and for example let's say that you are using mean squared error okay just for that case this can apply for any analytic error measure you put here but if you are using mean squared error this would be it okay that's a friendly quantity because now I want partial by partial I have this and this fellow is related to the thing I am differentiating with respect to this is the constant I can deal with the square so I am getting closer to being able to evaluate this explicitly okay so let's look at X the output well the output is nothing but you pass the signal through the non-linearity right the non-linearity is the tange not mysterious the signal is what I am differentiating with respect to I am almost done okay so now all I need to do is realize that when I do this because there is a chain rule and I am differentiating with respect to this and this is an intermediate quantity so I need to get theta dash okay so what is theta dash okay so what is the derivative of the tange happens to be 1 minus the tange squared this is for this particular one if you have another non-linearity you just compute what that is okay so this is good so we have delta for the final layer if I put the input get the output I go through this and I have an explicit value delta at the output so now the next item is to back propagate delta down to get the other delta okay so this is the essence of the algorithm okay so now I am taking the network but now I am taking the network I am taking one unit here okay and looking at all the units in the next layer because these guys happen to be affected by x and therefore happen to be affected by s remember delta is partial something by partial s and I want to get partial this by partial si in terms of partial by partial the s is here I am looking back so I already computed up to here and now I want to go here okay so now I need to take into consideration all the ways that this affect the output so I am drawing the relevant part of the network okay so this is the quantities that I want I want to evaluate partial e by partial so now I am going to apply the chain rule again so I will get partial e by partial these fellows which suppose in my mind I already know that's the first part of the chain then I am going to get partial this guy by partial x okay fine as long as I am making progress towards the destination I am okay by partial s okay so you go through this this is partial e by partial the final guy and this guy happened to be intermediate however the way this fellow affect the output it affects all of those guys okay so when I do that chain rule I need to sum up over all the routes that this happens through and therefore I need to sum up over all the points here for this quantity so the way e is affected by this guy is through the way e affected by this fellow through here or by this fellow through here et cetera and therefore the rule in this case would be the sum okay that's I mean looks like a very hairy one but you know no big deal now let's collapse it to something very friendly so it's a sum of something okay let's take it one term at a time okay we'll take this okay okay x i simply happens to be the non-linearity apply to this one so all I need to do is just differentiate that non-linearity and apply it to the value at hand okay so what do you get you get theta dash apply to the signal okay I can I can have that okay how about the next guy that's an easy one what is the derivative of this fellow by x i yeah this is just the sum I get the coefficient okay okay do I have all of this yes the next guy is the interesting one how do I get this well you don't get it you already have it by recursion this happens to be the old delta so now I have the lower delta in terms of the upper delta and I have the top delta in hand okay we are done we just have to keep doing this and we'll get all the delta and the form for the delta summation index j right okay and this happens to be the derivative of the tan just one minus that square so when you get one minus that square you get this and you can factor it out the rest of it depends on j and you are summing this up and you are getting this okay now isn't it lovely to have an equation like this this looks exactly like the forward pass we are taking something combining it with the way it's summing up the linearity which we did in the forward we are multiplying by this funny guy so it looks like a very much similar situation but when you are done you are going to get a bunch of deltas at every position where an s is okay and from our previous experience then we are ready to go with the delta and the x and adjust the weight that is sandwiched between them according to the arrows that are being down okay it used to go up okay so let's do this and then instead of having theta here we are multiplying by something and what we are multiplying by is this quantity okay that's what you do in the backward propagation okay so here is the algorithm refers the picture of the algorithm that's all you do you take the input you compute delta delta has disappeared for some reason okay and the delta and the x depend determine the weight in between okay so if you put the algorithm this way okay you initialize the weights and then you pick n at random that's what makes the gradient descent you do the forward run I describe you do the backward run and then you update the return the final weights and that is your algorithm okay now there are obviously all the questions determination criteria the localmina all of that that's the things we discussed in the Q&A session there is something specific here that I want to emphasize which is the initialization because it's very tempting to initialize weights to zero which works actually very well with logistic regression if you initialize weights to zero here bad things will happen so let me describe why first I'm the prescription is to initialize them why is initializing zero bad if you follow the math you realize that if I have all the weights being zero which is what that means and you have multiple layers then either the X or the Deltas will be zero in every possible weight one of the two guys that are sandwiching it will be zero and therefore the adjustment of the weight according to your criteria would be zero of the terrible coincidence that you are perfectly at the top of a hill unable to break the symmetry so you are not moving if I just nudge you a little bit you will be sleeping like there is no tomorrow but as long as you are there you are not moving pretty much like you think of a donkey that is hungry so they put two sacks of food on top of them all it needs to do is eat or eat unfortunately it's perfectly symmetric and the donkey cannot break the symmetry and it starts to death so we just want to break the symmetry so we introduce randomness we shake the food a little bit which is here to just start with a random thing choose weight that are small and random and you will be okay okay one final remark and we'll call it a day which is about the hidden layers so let's look at the network again so this is the network we have an understanding of this fellow and we have an understanding of the output and the hidden layers were just a means for us to get more sophisticated dependency okay so if you think what the hidden layers do they are actually doing a non-linear transform aren't they I have these row inputs and I am passing them through these things so I can look at these guys and consider them features okay and because they are higher order features I am able to implement a better one and this one will be features of features and so on okay now the only difference and it's a big difference between the non-linear transform here and the non-linear transform we applied explicitly in the case of linear models is that these are learned features okay so remember when I told you don't look at the data before you choose the transform and whatnot okay the network is looking at the data all it wants okay it is actually adjusting the ways to get the proper transform that fits the data okay and this is not bothering me because I have already charged the network for the proper VC okay the weights here that constitute that guy contribute to the VC dimension the VC dimension is more or less the number of weights that's the rule of thumb here okay so it is completely fine to look at the data because it's not looking at the data that is bad it's looking at the data without accounting for it that is bad and here it's built in that it's accounted for okay so this is nice because now you can see it's not a generic non-linear transformation it's a non-linear transformation it's a view to matching very specifically the dependencies that I'm after so that's a source of efficiency there okay now comes the question is okay can I interpret what the hidden layers are doing so I'll tell you a story early in my career I was doing a consulting job for a bank and they wanted to apply neural networks to credit approval okay very easy give me the data we'll do it we'll take a fairly simple network come and ask me can you please tell me what the hidden layers are doing okay so in my mind is he doubting my competence or something he wants a reassurance something like that I mean the performance is perfect and he can try it out of sample and what not but then I realize that the reason he's asking for the interpretation has absolutely nothing to do with performance okay it's legal if you deny credit for someone you have to tell them why and you cannot send a letter sorry we deny credit because lambda is less than 0.5 so that's it but the fact that you are not able to interpret what happens in machine learning is very very common go back to the movie example okay we get the factors we predict the ratings and let's say you apply this system and you keep recommending movies to someone and the person is so impressed you are recommending movies that's on the spot every time so they come and ask you how do you do it okay you tell him okay because factor number 7 is very important in your case okay they say okay great so what is factor number 7 and then you say it is lots of anyway you have no idea what factor number 7 but factor number 7 is important in your case okay very common in machine learning because you remember when the when the learning algorithms try to learn it tried to produce the right hypothesis it didn't try to explain to you what the right hypothesis is okay so let me stop here and then take questions after a short break let's start the Q&A okay so the first question is could you explain what people mean by using a momentum term in your networks okay so momentum is used as an enhancement for the batch gradient descent in order to get some effect of the second order so the idea is that if you use gradient descent is using strictly first order just the slope and if the surface is changing slope quickly which means that the second order is important okay you want to get a glimpse of that without having to go through the trouble of computing the Hessian the second order quantities so if you take what's called a momentum term which means that you take a little bit of the step that you had previously and a little some aspect of this because if the surface is curved this goes one way and if the surface is flat it goes the other way okay so I didn't I mean there are lots of heuristics the momentum is one of them for to cast the gradient descent the way I described it actually works very nicely and in all honesty if I have to go to second order I will just go for conjugate gradient because it's so principle and it really gets the bottom line so I'm not big on using momentum in my own applications but other people have found it to be useful okay so some people are asking about so the popularity of neural networks that he has got its ups and downs so what's the state of the art in neural networks research if there's any okay so initially in neural networks we're going to solve the problems of the universe so the usual hype okay in some sense it's not bad for research because it gets people excited and gets enough people to work to get the real results and then when it settles down there's a critical mass of work so I don't think this was a bad thing in hindsight but what happened is that because of the simplicity of the network and the simplicity of the algorithm people use them in many applications and it became a standard tool and there are lots of tools you will find in all kinds of software where you just apply a neural network and until this very day there are companies so they are post-research so to speak there's very little done in terms of research the basic questions have been answered but in terms of being used in commerce and industry and whatnot they are used they have very serious competitors like for example support vector machines and lots of other models but they are still in use not the top choice nowadays but every now and then someone will publish something and you do this and he will have used a neural network and got good results how to choose the number of layers and okay this is model selection so the neural network is really a class of models a class of hypothesis sets and there are obviously a bunch of things to choose how many layers and how many units per layer okay so if you look from an approximation point of view because of the sum of products in logic you can implement anything using a fairly shallow network provided that you have enough neurons in that layer but that's not an approximation question so the real question is you know how many weights can I afford and then the question of organizing them is less severe so how many weights can I afford because they reflect directly on the VC dimension and the number of examples I needed and there are actually methods that given a particular architecture it tries to kill some weights in order to reduce the number of parameters as a method for regularization and we'll allude to that when we get to regularization but basically this is a model selection a question when we get to a model section to apply the most profound of them would be validation that we will have a lecture dedicated to it okay can you why was the hyperbolic used used why is it used yes okay so I want a soft threshold and I want it to go from minus one to plus one and I want to have a nice analytic property that can differentiate it that these are basically the three reasons okay in the other case it was exactly the same I wanted something to go from minus one to plus one I wanted something to go from zero to one because in logistic regression I wanted a probability here I wasn't really interested in the continuity for its own sake there I was because it's a probability here I was interested in the continuity just because I wanted the analytic property of differentiation in order to apply gradient descent but what I care about is going from minus one to plus one which are the hard decision version will the final weights depend on the order of the way that the samples are correct they will depend on the initial condition they will depend on the order of presentation they will depend on that but that is inherent in the game we are never assured of getting to the perfect minimum the global minimum we get to a local minimum and anything will affect us but the whole idea is that you are going to arrive at a minimum and if you do what we suggested in the last lecture in the Q&A session that you just start from different starting points and have different you pick a point at random you could pick a random permutation and then go through the examples according to that permutation and then change permutation from epoch to epoch or you could simply be lazy and just do it sequentially and all of these more or less get you there we get you with different results so if you try a variety of those in let's say 100 tries and then pick the best minimum you will get a pretty decent minimum and will be fairly best in terms of independence of the particular choices that you made in any of the 100 cases could you go back to slide 12 on there so if you could review the two red flags for generalization and optimization okay so the top part of the figure showed that we are dealing with a sophisticated model because in spite of the fact that the unit of it is linear the perceptron is combining those fellows so when you have a powerful model you can express a lot of things and therefore the question of generalization comes in because if you can express a lot of things you have a big hypothesis set and then a question of zooming in and generalizations the stuff we handled in theory but the comment here is that we are going to have the VC dimension of whatever model we have and the VC just we cannot generalize but at least it's under control because we have the number that describes it in terms of optimization now it's not like I have the target written here and I'm just designing perceptrons I am giving a data set inputs outputs and I have a multi layer perceptron each of which is computing a perceptron function of a perceptron function of a perceptron function and now I want even for one perceptron that's why the optimization here is a red flag that's why we needed to go for an approximation using a continuous function where we have something like gradient descent that can work for us you mentioned that VC dimension is roughly the number of parameters so they want to yeah so they we are not going to be as lucky as the case of perceptrons of getting the VC dimension exactly okay in this case there are some analysis and because they are bound with weights in different layers and compensate for one another and there are some permutations and what not therefore they don't contribute full degrees of freedom each of them so you can upper bounded by the number of weights and lower bounded by something fairly close to the number of weights but smaller so as a rule of thumb you take it as the number of weights as being the VC dimension and that that has sort of stood the I mean if your interpretation you say yeah I understand perfectly what the what the first layer does it gives 0.3 weight to the first input and 0.7 weight to the second input and minus 0.4 weight to the third input and sums them up and then compares with the threshold which is 0.23 that if you take that as interpretation then they are interpretable but an interpretation here is that what people made that makes sense okay but what we are saying is that the the factor is relevant to the rating but cannot be articulated in simple terms that people would consider interpretation and similarly for the hidden layer here can you say what happened in the end in the bank with what explanation was taken then at the end no I can't I mean it's a private consultation I cannot comment in detail but basically the question was raised and it made the point okay can you explain again why in the past lecture you mentioned that data snooping is not a good practice okay data snooping is a bad practice if you don't account for it so I mean when we get to data snooping we will discuss it in one of the lectures we will say that you either avoid it or account for it okay the problem is that if you data snoop and you don't account for it in terms of its impact on generalization so you end up with something so optimistic you go to the bank if you do a private consulting job for a bank and tell them I have something that predicts the stock market great okay and then you give them and when they go to stock market it falls on its head and that's the problem because you thought it would generalize and it didn't so data snooping in the way I presented it was the fact that we didn't account for that we learned in our mind but we didn't account for the VC dimension of the space the damage is almost unavoidable it's a very good practice not to look at the data because the accounting is difficult in this case in the case of neural network there was looking at the data in a very prescribed way a learning algorithm was actually trying to find the weights that constitute the hidden layer so therefore it is looking at the data in abundance on the other hand the accounting has already been taken into consideration because as I mentioned the weights have been counted as contributing to the VC dimension so we know does the range of the weights alter the choice of eta which way repeat the question please does the range of the weights affect the the value of eta I mean okay so the let's say that you are making decisions okay so eventually you will take the output layer and heart threshold as if you are scaling the the weights but the intermediate weights the actual value matters because you the actual value of the output will contribute to the next layer and whatnot so you cannot just say that you know I am scaling invariant or anything like that but supposedly the learning rate was only a way to arrive at a minimum of the error function and the minimum of the error function will happen at a particular combination of the weights so it shouldn't affect it in the sense of a predictable way or a worse spot if I use a reasonable learning rate yes it does affect it but it affects it in an unpredictable way pretty much like you can say that how does the initial condition affect the result will it affect it but it affects it in a random way and you are better of just averaging over a number of cases or picking from a number of cases in order to to immunize against that type of variation is there a relation between neural networks and genetic algorithms I guess so both of them appealed to someone who is interested in a biology reflection genetic algorithms are optimization techniques based on getting a generation and having mating and keeping the good genetic properties and what not so it doesn't apply and everything in machine learning has been applied to everything so there were actually people trying to train neural networks using genetic algorithm and all the time I mean you find in the literature all combination you know therefore there is really no relationship between them okay it's more confusion so does in sample training constitute looking at the data okay so the strict answer is yes you look at the data all too well you are actually looking at the data and you are trying to minimize the performance on the data and all of that which again is fine as long as you have already put into account the limited VC dimension and therefore when you do that and you get to something you still have a guarantee of generalization from what you are arriving at to the out of sample okay so the learning algorithm looks at the data that's all it does it looks at the data okay but it's already before we even turned the learning algorithm loose on the data we have already chosen the hypothesis set and we put the generalization checks in place what do you recommend implementing the package it's it's okay honestly it's it's a borderline case for example if you are doing one I mean like a perceptron you just write it down it's so simple okay neural networks it's a little bit complicated and you will there are some sort of bugs that are typical and what not I used to have this as an exercise and then I decided that the logistics of doing it is not worth the benefit of it does analyzing performing some sort of sensitivity analysis on the weights give some some information about how the neural network yeah there are there is actually work on that there is also the questions of regularization based on that on how effective the weight is and the disturbance and what not I mean there are all kinds of analysis that are the neural network have been studied to a great level of detail and indeed the choice of the weights the range of the weights the perturbation all of these have been looked at are there other models that lend themselves more to interpretation I mean if you if you have a bunch of parameters and the algorithm going to choose them then already interpreting those parameters is not clear you can artificially put constraints in order to make sure or you can start from an initial condition that already has an interpretation and you are just fine and whatnot but that's if you are very keen on the interpretation aspect yeah okay going back to the first examples where there was logic implementation with the with the perceptions so there was a confusion that are we still trying to learn weights here or we just have them fixed or no this is was an illustration for the fact that when you combine perceptrons you are able to implement more interesting functions this was didn't touch on learning yet after we do that we found that the structure that is multilayer is an interesting model to study and from then on it became a learning question we had a neural network we are no longer going to look at target functions and try to choose the neuron we are just going to put it as a model and let the learning algorithm choose the weights which is back propagation in this case could you briefly explain early stopping okay this is okay I think it is best described when I talk about regularization and validation so it is basically a way to prevent overfitting which is the next topic so I think it will be much better understood in the context what we want to understand what overfitting is and what are the tools for dealing with overfitting regularization and validation in this case and then early stopping will be very easily explained a question on stochastic gradient in this sense so when you go through an epoch you choose randomly points only points you have not selected an epoch is one run and it's a good idea to get all the examples contribute so one way to get it to be random and still guarantee that you will get all of them is instead of choosing the point at random you choose a random permutation from one to n and then go through that in order and then for the next epoch you do another permutation and whatnot okay if you do it this way eventually every example will contribute the same but an epoch will be a little bit more difficult to define you can define it simply whether you cover all of them or not that is valid and some people simply do a sequential version no randomness at all you just go through the examples or you have a fixed permutation and you go through the examples in that order and keep repeating it and there are some observations about differences but the differences are not that profound does having layers and no loops limit the power of the neural network loops as in feedback I'm assuming okay a feedback even the definition of what function I'm implementing becomes tricky because I am feeding on myself okay so it's a completely different type which is I mean there are recurrent neural networks which actually is the model that started work in neural network and it has completely different mathematics and application domains and whatnot here you are implementing a function and it is clean enough to do it in layered weight in order to get nice algorithm like that and since we showed that you can basically implement anything you are not missing out on something by doing that one can become say that okay maybe I can get a smaller network if I can jump layers and whatnot which is possible and whatnot interesting intellectual curiosity but in terms of practical impact it has very little so in terms of the VC dimension so since it it roughly depends on the number of parameters if you had a fixed number of nodes but you arrange them in layers what do you gain or what do you lose okay if you believe in the rule of thumb and it's just the rule of thumb that is basically based on upper and lower bounds okay then if I rearrange the number of nodes the number of that the number of weights will change because the number of weights you know I see how many neurons here and how many neurons here and I apply the number and that will give me the number of weights so as long as you take your guiding number the bottom line number is how many weights did I put in the network you will be more or less okay I mean obviously you can take extreme cases where I have one neuron feeding into one neuron feeding into one neuron the example I gave last time so you have tons of weights that are really not contributing much okay but within reason if you have general architecture that are reasonable then the number of weights is the operative quantity okay very good so we'll see you next week


Michael, does the university's ranking play a role in the visa process? No, not to my knowledge. I think the visa process is done separate of the university's ranking. There are so many different rankings out there for universities. The visa process is really just based on your paperwork, making sure that you have it all in order and working with the university that you'll be attending to make sure that they do everything on their end as well. Thank you, Michael. And as Michael said, there are many different rankings. Your program could have a ranking in business or a finance, so make sure that you check different types of rankings and of course be able to explain why you want to pursue the program of study that you are going to...


In this screencast, I'm going to be showing you how you can create your own tests. Creating your own test is going to be very useful for Assignment 2 and 3. Now remember that Assignment 2 and Assignment 3 use a bunch of user space programs to do most of the testing. These programs can be found on the user line test menu. As you can see, there's quite a few programs here. Although in Assignment 2, we only use a subset of these. To know exactly which programs we used for testing Assignment 2, you can look at the Assignment 2 target file. Alright, to create your own test, first you create a directory with the same name as the test. In my example, I'm going to create one called myTest. Once this is done, you copy files from an existing test into your new directory. Now, you modify the make file to change the name of the program and the sources that are going to be compiled as part of this program.  Once this is done, we also need to change the name of the source file to reflect what we inserted into the make file. So we do that. Now, all that needs to be done is change the, modify the content of the source file to reflect whatever you want to test. In this case, I'm just going to do something as simple as print a hello world. So I'm going to get rid of all of this code and replace it with a simple. Now, by default, the OS 161 GCC has a W error flag enabled, which complains W warning flag enabled, I'm sorry, which complains when you have unused variables. So we're going to get rid of these two variables as well. Save the file and do a B make. As you can see, B make completed successfully. And if I do a B make install, this test will be installed in my root directory. Now, unfortunately, there is one additional step that needs to be done to make sure that this test is compiled every time when you do a B make in the root OS 161 directory. To do that, you go into test bin and modify the make file in there. As you can see here, it defines all the various directories into which the root B make has to go and run a B make. So what we're going to do here is we're going to add our test here. And now we should, if we run a B make here, it will compile our test as well. So this is how you create your own test inside OS 161.