In this chapter, we are going to introduce the convolutional neural networks, or CNNs, specifically for image understanding. The concept here illustrated can be easily extended to different kind of input, like audio or video. All it requires is going to be a slightly variation in the type of convolution that is going to be utilized. We will start with the rational and motivation. Why are convolutional neural networks needed in order to process images? We will see that the fundamental aspects of a convolutional neural network is sparsity, which exploits locality in the input data. Specifically, we will introduce the receptive field and the hierarchical view of deep models. The second main important aspect of convolutional neural network is that they perform parameters sharing, which is based on the assumption of the homogeneity of the input data. We will see then how a convolutional layer will perform a 3D convolution, in the case of image processing. Then we will talk about the nonlinearity layer and the decaying learning speed across the network. So we will revise the logistic sigmoid and we will introduce the rectifying linear unit. Moreover, we will introduce the LP pooling layer, where two specific cases are the average pooling and the max pooling. Then again we will see the rational and we will have some conclusions, where we will show what are the benefits of using convolutional layers and pooling layers. Then we will switch to the shell and the command line and we will install the pre-TNN package in order to have color-coded network architectures, which is going to ease our understanding of the models and will be fundamental for the next practical. And finally we will introduce the ELAB Torch 7 profiling tool. It's a tool I wrote around October 2014 and now has more than 200 commits from all the members of ELAB and others, with which we will profile a convolutional neural network in Torch. And compare the results we have computed before manually. In the field of image understanding, let's find out why convolutional neural network had had a major role. Let's start with a neural network which goes from let's say N neuron in input to three hidden layers of size twice as the input, so 2N, 2N, 2N and then capital K output. Let's compute the number of operations that a similar network may require if used for image processing. Let's say we start with an image which is a color image. So we have three planes, RGB, and which are of size 256 times 256, and then in depth we have the three color channels, RGB. So in this case N is going to be equal to 3 times 256 times 256, which I can write as 3 times 2 to the power of 8 and 2 to the power of 8, which is 3 times 2 to the power of 16. Overall we can count here 2N, 4, we can have 2, 4, 6, 7N. So we have 7N plus K neurons, which in this case is going to be 7 times 3 times 2 to the power of 16, which is roughly 1.4 million. To go from the first layer to the second layer we are going to use a matrix which has height 2N with N. Then here we are going to have a square matrix, 2N, 2N, another square matrix here, 2N, 2N, and then here something like K and 2N. So if I count how many multiplications and additions we are going to have, so we are going to have 2N squared plus 2 times 4N squared plus 2 capital K. So we can neglect this term here. We have overall 2 times 4N plus 2, roughly 10N squared, which is equal to 2 times 5 times 2 to the power of 32 and 3 to the power of 2. So this is actually going 2 to the power of 33 and this is roughly 390. Multiplication, max, which is multiplication, accumulation. Let's say we are on my MacBook Pro, which has an Intel Core i7, which runs at 2.2 GHz. So we can see how much time it is going to take to process one image, which is a color image of 256 times 256 pixels. So we have the 390 GHz divided by 2.2 GHz, which gives us 180 seconds, which is roughly 3 minutes. So we have here seen that to process this image here, it's going to take us 3 minutes just to send forward one image. Think about training where we are going to use several images in both directions forward and back propagation. This is really impossible to go this way. So what can we do? We can be smart and we can reduce the number of computations by exploiting the underlying distribution of our data. So if we actually think about what we are trying to achieve here and we are not just blindly applying a neural network to whatever problem it is, we see that our images have very distinct specific characteristics given that they are natural images, they are not random noise. So the two main points we want to keep in mind are the images highly specially correlated, means if two points on these screens are closed nearby, it's likely that they will have the same color. Most of the time they will be black in this case. But if you take two points close to some natural images, both those values will be very close in value. Moreover, in images, objects are made of parts. Let's see how to exploit these two particular aspects of images in order to reduce the amount of computation and therefore speed up the whole system. So the first thing that it's very important to keep in mind when we speak about convolutional neural network is that they introduce sparsity. Here we can see that we have a fully connected layer, L-1, which is connected to the layer L with all the connections as we have seen. Moreover, there will be the plus-1 for the bias here on top, which is connected to the three neurons on the right-hand side of layer L. The same here, we are going to have the bias on top, which is connected to the single neuron of layer L plus-1. So how can we exploit this correlation in order to reduce the number of computations? We can say that, for example, the first neuron here can simply see a subgroup of the input, because input pixels nearby will carry some kind of information relative to the object or to the location where they are coming from. But it's quite unlikely that pixels that are far away will have some kind of contribution for characterizing the specific location at which the first neuron may be looking at. So sparsity basically means that the first neuron of the layer L is going to just look at this neuron here, this second neuron over here, and the neuron over here. And of course, there will be also the bias for the same way. The center neuron will just see this guy here, this way. And again, we have the plus-1 for the bias. Analogously, we have the three connections for the last one. In this specific neural network, we say that this layer here will have a receptive field Rf of 3. That means that we'll see just three neurons from its previous layer. Let's move on and let's connect also the layer L to the layer L plus-1. So we are going to have first connection, second connection, third connection. Also here, the receptive field is 3. But the receptive field with respect to the layer L minus-1 is going to be 5. Why is that? Because the neuron at the output layer can actually see five neurons from the input space. So in this case, we have that the neurons at the intermediate layer, the red one, will just see a portion of the input space. And therefore, we'll have the notion of edge. The neuron afterwards in the higher, as we go high in the higher C, we'll see more pixels and we'll be able to have a better understanding of what's going on in the input image because they can combine the response of neurons that are in the intermediate level, which builds up to generate a more abstract and a high order representation. We can draw here, therefore, here in this direction, a global view direction. Eventually, the last neuron of this network will be able to see the whole input image and will be able to distinguish between different objects that we are going to propose to the network, for example, for a classification task. As we can see here, the number of computations is being greatly reduced. We will see later a numerical example that is going to give us some better understanding of the order of magnitude of reduction. So we said once more, sparsity give us a reduction in computations by exploiting local correlation of the data. The second main principle on which convolutional neural network base their strength is the parameter sharing. So the first one we said it was the reduction of computation due to sparsity and this one instead is parameter sharing which is going to ease the convergence of the training of the model. How is this done? How does it work? So we see basically that each of these neurons in the red layer will share all the weights for the same kind of connection. So we can represent this by drawing here, for example, in purple, this connection here. Then, for example, in blue, we have this one and then we have the last one in pink. Finally, we don't want to forget our bias, of course. Which is the same parameter associated with. In this case, each of these neurons are looking for the same kind of feature in different regions of the input. As we saw when we performed the computation of A1 here, so A1 is going to be just this partial input here. We can call it X for the moment. So we are going to have a projection of the X for the neuron one. So we can put one here. So in this case, we project the first part of the input on the specific feature theta, which is shared across the whole neurons in the hidden layers. And each of these neurons is going to look at different regions of the input space, to which it's going to still look for the same kind of target, the same kind of activation, the same kind of kernel we can call. So what we are going to do, basically, it's the projection of portion of the input with respect to the same specific parameter vector, which is a kind of feature we are going to train our system on, basically. And then we are going to take, of course, our non-linearity in order to boost the correct result and to attenuate the opposite. Parameter sharing allows us to reduce the number of parameters, which in turn helps convergence in terms of time and in terms of relative error, less time and less error. So let's see how the convolutional layer actually works. So the normal case, the generic case, we are going to have an input that I'm going to call just X, which is a 3D tensor of dimensionality. N1, 2D maps, N2 times N3. N1 maps, basically, or channels, and these maps have height N2 and width N3. For example, if we are in the input, so if we are actually talking about the real input, so image input. So in this case, N1 could be 3, which are the 3 RGB channels. And the N2 and N3 are going to be the height and the width of our image. For example, in the case we saw before, it could be, for example, 256 times 256 height and width. Then our output instead, Y, it's also going to be a 3D tensor of dimensionality N1 maps. So we have N1, 2D maps of dimension, N2 times N3. And how do we get these maps? So we also have a collection of kernels. We call them K, which is a 4D tensor in this case. And it's made of N1, 3D, N1 times P1 times P2 kernels K, K, I. Finally, we have a term B, in this case it's a vector, so it's 1D. And this is my M1 of dimensionality M1 bias term. So how do we compute Y? We said we had M1, 2D, M2 times M3, Y, I feature maps. So each feature map is M2 height times M3 width. So we have a 3D input, which is N1 maps or channels of height N2 times N3. Then the output is going to be, and again, a 3D tensor of M1 channels of maps of height M2 times M3. Then we are going to use a collection of M1 3D kernels of dimensionality N1 as the input times P1 times P2 kernels K, I. And I here goes from 1 to M1. And then at the end, we have also a term bias, 1 dimensional, which is the same size of the number of channels of the feature maps, Y, which is the bias term. Now we can see how to compute these feature maps. We have that each and every feature map Y, I is going to be equal X convolved with each of these K, I, and to which then I sum my scalar B, I. And we have this one for I, that is going to be equal 1 up to M1. And we have that this guy over here, this is the convolution symbol. And basically what this equation says here is that we project our input X onto different subspaces, K, I, and then we shift them by a specific bias term. It's very, very similar to how we were computing the weighted input Z, which was equal to my theta matrix times my activation, or X, if it's in the first layer. So as we said here, we have that each of our feature maps, Y, I, is going to be equal to the convolution between the current input X, which is the activation basically from the previous layer, convolved with each of the kernel M1 kernels for the specific layer, K, I, and then we shift by a specific bias B, I. So how does it work? This one is a 3D convolution, which can be expressed this way. So if we have, for example, an image or a feature map F of variables L, M, N, and we convolve with a G, L, M. It's going to be equal to the summation over all the variables u, v, and w of F in u, v, w, multiplied by the kernel which is flipped and transposed as we sum. So we have L minus u, M minus v, N, N minus w. If this is still very cryptic, I'm going to draw something. Perhaps it's going to be more easy to understand afterwards. So I have here my X, which we said it's N1 here, N2 here, N3 here. Inside we have our kernel, that could be, for example, 3 by 3. And this guy, as we see from this equation, is going to move this direction and this direction. From where it is, for example, in this first location, it's going to be outputting my first value. It's a 1 by 1 pixel, so this is 1, 1, 1. So then it's going to move across towards the right and down. So at the end I'm going to have something that is like this and maybe also down here. Just one here in the depth dimension is going to be of height M2 and width M3.  So that we can end up having our final output Y with M1 maps. And the height is going to be M2 and the width. As you can recall from the classical neural network case, after we compute z from the linear operation where we have the matrix multiplied with the activation from the previous layer, then we are going forward and computing the new activation as the nonlinear function applied to our weighted input. In this case, in our case, our weighted input is called feature map and the projection is done with convolution. We have now applied the nonlinear function to our feature map or our weighted input. We can start by having a look to the nonlinear function we already seen in the previous lesson. So here we can draw, for example, sigma of z and here we have z. And if this is 1, it's going to be half. Roughly if this is going to be 5 and if this is going to be minus 5, we are going to have something like and then it goes this way. And the problem that we have with the sigma is that if I compute the sigma prime in z, we have that if this was 1, this is going to be half. This is going to be a quarter. Still the same x-axis, minus 5, 5. We are going to have something that is kind of very annoying. As we can remember, every term delta in our neural network was multiplied by something here, was multiplied by the derivative of my activation. So every delta, every time we go in layer after layer, it's going to be multiplied by something that is at most one fourth when the z is close to the origin and otherwise just goes to zero. So this creates some problems because the training basically slows down across the network. The layer more close to the cost function can actually change their values quite accordingly to what gradient descent says and establish, but the layers in the lower layer, the one closer to the input, so the one with less abstraction will have some gradient that are basically very, very, very small and therefore the gradient with respect to the parameters is going to be also very small and therefore the learning appears to be slowed down incredibly for the early layers. So since we have to learn here networks that are quite longer than the, they are called deep neural network because they are deep. They are deep in length, meaning there are many layers stacked one after each other. We cannot have this problem of learning that slows down so much due to this particular effect. So we can try to remove this effect by changing our non-linearity. And we can use this one, the rectifying linear unit. As a definition, actually this is defined simply as Z plus, which is the positive part of Z. So it's completely trivial and let's say this is one and this is one. So notice that I've changed the X scale between these two charts from the left and right hand side. For the derivative, we are going to have, of course, it's not differentiable in Z equals zero, but we don't really care too much. So if this is one, we are going to have simply one here and this is zero. And we can define that, it stays here in zero at this point. In this case, we can see that the problem of having the derivative very low is not going to happen here. And we don't have this slowing down across the network due to the non-linearity. This technique was introduced in the paper of Alex Krzyzewski and John Frihinton when they submitted their, the first deep neural network in 2012 for the ImageNet Challenge. And they write in their paper, they show that rectifying linear unit helps a lot speeding up the training. The last layer used by convolutional neural network is the pooling layer. So what does this layer do and how it works? Let's go in how it works and then what it does, we explain what it does. So let's think of having here a piece of feature map, one of the M1 slices, for example, coming out from the convolutional layer that is going after the point was non-linear function. And here we may think get, for example, this first region here and compute the LP norm so that we end up with our first output here. Then we apply, for example, a stride equal to this size to the width of this running window. So our second window is going to be in this position here. And then it's going to perform again the LP norm, which is going to give me my second value here and so on. So I can keep going. Just for clarity, if we would like to compute the P norm of a vector, this is going to be equal to the P norm, P root of the summation across all components of X to the power of P. Moreover, if the P norm tends to max of the components of X, when P tends to plus infinity, max pooling can be simply seen as a specific case of the more generic LP pooling. What does it do? Basically, if we see in the dimensionality, if we apply a stride equal to the size of the kernel here, the running window, we are going to obtain an output which is shrinked in size. So if we started with m2 in height and m3 in width, we are going to get, for example, m3 half and m2 half. The number of channels is still m1. So the pooling operation works across the spatial dimensionality and reduce the number of neurons by dropping them. If we choose, for example, the max operator and therefore just the maximum activations are carried across the network and are going to be then used for later on, for later computations. Sometimes the L2 norm is used instead to average out all the activations to instead having higher robustness for noise. So there are different cases which require different kind of norm, different kind of P value. So for this reason I presented here the generic pooling LP norm pooling rather than just the max pooling, which is most commonly I think used. Now that we have seen what are the main blocks of a convolutional neural network, we can see how the initial example where I was computing the number of computations and the time to process one image can change if we do apply, if we do use convolutions. We can start here by drawing our input, which was 256, 256 and 3 in depth, our input image. We said we'd like to have twice as many neurons, so twice as many pixels in the feature map, if we can say. They are not exactly pixels, so they are activations in the feature map. So in the input space we can call each point of this image a pixel. In later layer, in the feature map they will be called activations instead. So we said 2n, therefore let's make 2n, so we still have 256, 256, let's go 6 here, still 2n, so again 6, 256, 256, then here once more. And then the last one is going to be simply k vector. So to go from the first block to the second block, let's say our kernels k, k i are 5 by 5. We said there are 3 kernels, they can be 3 if we are in the first one, or 6 if we are in the other cases here. So in this first point we are going to have, if I count the number of computations of max, we are going to have 256 times 256 times 6, that is the number of activations. Which came from a convolution of a kernel that is in this case 3 times 5 times 5. So this was the first number of computations, so for the second one it's going to be the same, but the only thing that changes is going to be this 3 that becomes 6. Then again from here to here it's going to be again 6. So I can say this one multiplied by, we said we have 2 times 2 times 1 time, so we are going to have, we can write 1 plus 2 plus 2 multiplier. So this is going to be 5. And overall this is going to be equal to 147 million max. For the final part we have the, we start, we have a matrix that is of height k, and the width is going to be 256 times 256 times 6. And which gives us k times 6 times 256 times 256. It's going to be k multiplied by 393 thousand. If we think about the ImageNet classification, we are going to have that k is equal 1000. Therefore this is going to be equal 393 million. Overall we have 540 million Mac, and if we divide by 2.2 gigahertz of my nice Macbook Pro, we get something like 250 milliseconds per image. So we went down from 3 minutes to a quarter of a second, which in my opinion is pretty neat. So we would like to keep in mind these numbers here. Can we do better? Yes we can, and we can use the pooling operation, which is going to be speeding up our system by at least 15 times. So we said we start with a cube, a data of 256 times 256 and 3 in depth. And for example we can apply now a convolution with a stride. So instead of applying the kernel every pixel and moving it on the horizontal line, horizontally and vertically by one pixel, we can move it for example by two pixels. So in this case we are going to have half of less pixels in the x direction and half of less pixels in the y direction. So we will end up with 128 pixels here and 128 pixels here. And we said we were going to double the size in depth, so again we are going to have 6 here. So with this configuration, see how many computations it's going to take. So we have 128 times 128 times 6, which is the number of activations. And then times my kernel, which is going to be 5 times 5 times 3. And this one is roughly 7 million and 300 max. And this first was convolution with stride. Then we are going to apply the second convolution, 6 in depth, 128 in x, 128 in y. And again we said 6 here and this guy is going to be the same. It's going to be the same because we may use padding, so this is not the valid convolution, but this is the same convolution if you use the MATLAB kind of jargon. So in this case we don't have to subtract let's say 4 from the size of the height and width. Since the kernel is 5, it should have been from 128, it should have gone down to 124. But then it's complicated to actually have nice numbers, so we can just pad and then we have with zero padding we are going to keep consistent the dimensionality. And we don't have to worry too much. Let's see how much this convolution takes. So we are going to have 128 times 128 times 6. And then we have our kernel which is going to have 5 by 5 by 6, actually 6 by 5 by 5. 14.7 million max. Then let's say we apply pooling. So we may start with 6, 128, 128. We end up with 64, 64. The number of computations is going to be 64 times 64 times 6 times our kernel. This is going to be for example 2 by 2 in this case. And this guy is 98000 max. So it's very small compared to the amount of computations that the convolutional layer takes. Then we have the last convolution. So we're going to have 6, 64, 64, 64, 64. And therefore the output is going to be 64 times 64 times 6 times the kernel. So it's going to be 5 times 5 times 6. And this is 3.7 million. And let's even apply one more pooling here. So we go from 6, 64, 64, 26, 6, 32, 32. Exercise let's compute the last one. 32 times 32 times 6 times 2 times 2. Which is 25000 max. At the end we are going to have the last block which is 6, 32, 32.  Here I haven't computed the nonlinear functions but the nonlinear functions, they are simply the number of neurons activations in the output layer. In the first case it's going to be 6 times 128 times 128 which is 98000. So here we have 98000 of nonlinearity, MAC. And we can see it's small enough that we don't care compared to the million of operations. So the final part we have 32 times 32 times 6 which is equal to 6000. And the final matrix is going to be k and 6000. And if k equals 1000 for the ImageNet for example, we are going to end up with 6 million MAC. If we check the summation of all the convolutions, we are going to have that this adds up to 25.7 for convolution. The last one we said it was 6 million for the linear part. So overall we end up having 25.7 plus 6. It's going to be 31 and 7 million MACs divided by 2.2 GHz. We are going to get 14 milliseconds which is roughly 17 times less than the previous number. We would like now to test and verify the numbers we just computed. Just to be sure they are reliable and let's see also how much time, how long my computer actually will take in order to process an input image in these networks that we have just shown before. Before starting with the networks, let's set up our system with some new packages. We have rocks, install, 3t, nn. Moreover we would like to get clone the Torch 7 profiling repository from the ELAB website. And there we go. So we can CD into Torch 7 profiling. And in here we see there is the profile model.lua. We can do Torch profile model.lua.h. We are going to have a help display. It shows us how to use this file moreover. It shows an example. So we are going to call th profile model, the name of the model, where we are going to define our model. And then we are going to send the specific resolution. And moreover we are going to compute MACs instead of both addition and multiplication. Let's create the model from our network. So we can go inside the folder model. Models and we can create our model file. So let's call it 3convolution.lua. So since I don't know how to do this by heart, we are going to just use the interpreter here below. So we can require nn at the beginning. And then we have, we also require pretty nn. So it true works. So we have net equal nn.sequential. And then we have our first layer. So let's copy here. So let's go here. We are going to write our first layer. So we go net add rather new layer. And then we want a spatial convolution, which goes from 3, because we were starting from input image of 3 channels to 6 channels. Using a kernel of 5x5 with a stride of 1 both directions. And let's put padding of 2 both sides. So to the left, to the right, top and bottom. So that the dimensionality of the output is going to be the same dimension of the input. Then we go net add our relu. Let's copy these two guys here. So we have first one and second one. And this one is going to be my first. So let's go with the second layer.  It's going to be from 6th channel to again 6th channel 5x5. No stride. Padding it's okay. So okay, then add another relu. Let's add these two guys. This one. And then again one more reset, right? So this one is exactly the same. And then relu. And now we can add, we had to reshape and view differently the cubic feature map into simply one long vector. So we can do this by using the view module. To which we say just resize it in one chunk only. So if we print now net, we see here it is a sequential. We can see there are seven modules in this sequential. There are three convolutions. The first one goes from three channels to six channels with a kernel size of 5x5. Striding of one in both direction and adding some padding of two pixels per each boundary. The second module is going to be non-linearity and directifying linear unit. Then we have another spatial convolution which goes again from 6 to 6. Kernel size 5x5. Stride one and padding two so we can preserve the dimensionality. Again another non-linear function. The last convolution 6265151 and 22 for the padding. Last non-linear function and then to view the 3D volume into one long chunk. So I have to add now the last linear layer. Since I don't know what's the size instead of computing by hand, I can do a trick. Let's say x is my input, so it's my torch tensor of dimensionality 3 because it's RGB. Then we say 256, 256. So if I ask what's the size when I forward to the network x. Sweet. The dimensionality is 393 whatever. So we can do the net add nn.linear which goes from this dimensionality to let's say 1000 classes for the ImageNet competition for example. So if I print the full network again here, it took some time. You can tell it's quite big as a matrix. If I print my whole network here, it's a sequential with eight layers and the last one is a linear. The last non-linear function is not required. Actually it speeds even training without the last non-linear function. This one is my third layer and this one is my output. Here we actually need to view. We need to add, I forgot, view minus 1. As we said before, we were estimating 147 million max for the convolutional part and here we have 149, so I think it's pretty reasonable. Then for the fully connected part, the linear layer, we were estimating 393 million and we have here 393 million which is also quite reliable. So overall we had 540 million and here we have 543 million. So it's very very close to the computations we have made. And overall we were estimating that CPU can perform one mech per clock cycle. So we were dividing 540 by 2.2, we were getting 250 milliseconds. Here instead we get that the forward took quite half of the amount is 129 milliseconds. And we have that this CPU, my computer, performed 4.2 giga max per second. So we have basically 2 max per clock cycle. And otherwise all the numbers are consistent. Let's make now a test for the other network, the one with the pooling. And let's see how it changes. So let's go back to our model here. So let's save this as 3-conv plus pooling. This is my first CNN plus pool. So let's put a pooling layer after this convolution and after here. And let's change the first convolution instead of having a stride of 1. Let's have a stride of 2 and 2. So we have here to modify this last layer. So let's see how to change it. So we can do torch net equal require models dot 3-conv dash pool. And this line, cool. So if we print the network, we are going to have this architecture. Where we can see we had a convolution that goes from 3 to 6 with 5 by 5 kernel. Stride of 2 and 2 and padding 2 and 2. That is basically for both directions. Then we have the nonlinearity and the new convolution. 6 to 6, 5, 5, 1, 1 to 2, nonlinearity and then the pooling. Again, one similar chunk with the convolution, nonlinearity and pooling. Then there is the view. So you net forward a torch, tensor, rend of 3, 256, 256. And then we actually ask the size of this one. We have the new number here. And we can go here and replace this one. So this is the 3-conv pool. We can go down below. Let's make it full screen. And we can call like this one instead of 3-conv. We have 3-conv pool. Everything else should be the same. Let's go. Bam. It was quite fast this time, right? So let's compare. Let's check some results. So at the beginning we were saying that we were getting the first convolution. We were estimating 7.3 million and here is 7.5. It's pretty close. Then we had 14.7 and here we have 14.8. We actually forgot to take in account the bias terms which is adding some little overhead. Then we were saying that the first nonlinearity would have taken 98,000 and 98,000 it is. Also the pooling is taking 98,000 and 98,000 it is as well. Following we had another convolutional layer with 3.7 million and here we have 3.7 million. Finally we have the linear layer was taking 6 million and 6 million it is here. We can even see in the last column the percentage of the computation which is very important. It should be always theoretically distributed across the whole network in order to have a more balanced distribution of operations and also of processing of the information. If we go down below we can see that the convolutional part takes 26 million and we were estimating 25 and 7 so I guess it's correct. Overall there is a total of 33 million and we were estimating 32. Divided by 2.2 gigahertz we were assuming one meg per clock cycle. We were estimating 14 milliseconds and my computer just finished running the profiling in 11 milliseconds. And this is pretty I would say accurate estimate we got on paper so far. Now we could hypothetically write down the network that is using only linear layers but we can see that it's not going to even be loading into memory. But just for exercise we can do this. We can do linear for example and we have my huge MLP multi-layer perception or a classical neural network. So we can write a local n equal 256 times 256 times 3 and then we have our local net equal nn sequence. And then we have the first layer we were saying net add we add nn.linear which goes from n to n and then we are going to do net add nn.radial. And then we have the second layer which is very similar to the previous one but here we have from 2n to 2n. And the third layer which is going to be identical in this case third layer and then the output which is going to be net add nn.linear of 2n. And then we go into the output which is 1000 and then at the end we have the return net but this one won't even load into my computer's memory so you are free to try it yourself. I won't already crash my computer previously so I won't do it again. And that's it for today. Stay tuned because in the next video we will see some pretty standard architectures from the first one like the Lynette and the AlexNet up to the most recent GoogleNet and ResNet and why some specific architectures have been chosen and what are their good points and what we can learn from this.


Personal notice, dangerous my stock and trade. If the job's too tough for you to handle, you've got a job for me, George Valentine. Write full details. Well, Company of California invites you to let George do it. In just a moment, we'll begin tonight's adventure of George Valentine. But first, here's more proof that heavy-duty RPM motor oil gives extra protection against wear and repair. A fleet of taxi cabs used to average only 35,000 miles between overhauls. Now, using heavy-duty RPM, these same cabs travel over 100,000 miles before overhauling. And that's your proof that heavy-duty RPM motor oil cuts engine wear and greatly increases the time between overhauls. Gives top protection and top economy of operation the kind you want in your car. So next time, be sure you get heavy-duty RPM at any independent chevron gas station or standard station, where they say, and mean, we take better care of your car. Tonight's story, Chance and Probability, a transcribed adventure of George Valentine. Dear Mr. Valentine, Chance and Probability, a harmless-sounding phrase, isn't it? It's had a horrible influence on my life ever since I first heard it. You're in a beautiful South American town that has pink houses, incredible sunsets of liquid gold, and caesars. The fanciest gambling can see no outside of Monte Carlo. Oh, yes, it also has a murder that everyone, but me, is willing to forget. Mr. Valentine, I need help from a fellow American, someone I at least know by reputation. So please, please use these airplane tickets and come to me. Anxiously, Mary Marston, Kara Professor LaRue, 25, Avienta del Sol, Pulizone. P.S., the words Chance and Probability refer to a gambling system which can't fail to make anyone a million dollars, except that its possession is an invitation to death. Wow, that was quite an afterthought, eh, Bruxy? Yes, George, and those tickets aren't all at Mary's house. Here. Wow, five golden yellow chips with the name Caesar's, and a royal crest no less. The fanciest gambling casino outside of Monte Carlo. Five $100 chips as a retainer, and the tickets are on a flight leaving tomorrow. I'm going right home and packed. Eh, but Bruxy, to go all the way to South America, just like that. Oh, yes. And all the things we have to do here at sea. Oh, yes. But after all, Mary did say the place has incredible sunsets. The golden yellow color. Of these incredible chips. South America, here we come. You've done sing to cheer one of the very best, Professor Marcel Leroux, at your service. Well, that's fine, but Miss Marcelle. Ah, you, Miss Marcelle, all you have the born grace of the natural dancer. Well, I get along on the dance floor. You get along? Oh, no, just get along. No, I can't teach you to dance. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marcelle. Miss Marston, you have come to talk to her about the system, no? I mean, that would be the only reason you would come all the way from the United States, no, mademoiselle, no? Ah, secrecy.  I'll keep that in mind. No, I will not let anyone else ever get it first. Please, remember that.  Ah, boy, boy, they're welcome to Polyzone. Welcome, Monsieur, but we'll say... His name was Bill Shields. He lived with the professors, too. He was the only friend that I had here. From what you told us, he must have been a pretty wonderful man. He wanted me to go back home. I didn't know he left me all his money until the police found his will. And somebody killed him at Caesars. Yes, because Bill had the system. Do you really believe there is such a thing, Mary? Well, he won $10,000 in one week, and he could have won more. But he said that would be enough for him. Enough for me to get well and strong again. How long have you been down here? Well, almost a year. I was singing with a show that went broke here. And then I got sick. But one for Bill, I... What was Bill doing in Polyzone? Well, he'd been an engineer on some kind of a construction job. But then he got interested in the system. This business of chance and probability. What was that? Well, Bill said the answer to any system had to be in mathematics. He always had me getting books on mathematics. Chance and probability. Then he succeeded. Mr. Valentine, I can't use any of Bill's money until I find out who killed him. You can understand that, can't you? I'll do what I can. What police say about all this? Well, the night that Bill was killed, there was a robbery at Caesars. They say the robbers killed Bill just sort of accidentally. I see.  He's a copper would be interested in doing that. Oh, and somebody might have tried to get the system from Bill and failed. Has Bill ever written this system down, Marie? No. No, not that I know of. Has he ever explained it to you? I'm afraid I wouldn't be smart enough to understand it. Anyway, now you feel that nobody will do anything to find out who killed Bill, is that it? Mr. Valentine, you can get all the facts from the police and from Cesar Capo. They'll be delighted to give you the facts. But that's not enough. I must know who killed Bill. All right, Marie. I already saw the fabulous pink houses of Polyzon. This evening, we'll see the sunset. And maybe tomorrow we can complete the points of interest and look for the murderer. Say, Mr. Valentine, I assure you we are doing all we can. Yes, so I've heard, Lieutenant D'Costro. Not anything you can do to help, Senor. We would be honored. Thanks. How about this Cesar Capo? A fine man. Fine. Everyone in Polyzon seems to be an exemplary character. That is the way it would appear, Miss Brooks. Exactly. Still, a man is dead. And the system... No, that is something I would not venture an opinion on, Senor. Didn't you ever see Mr. Shields play at the casino? Oh, see. And so him win again and again. Wow. Could have been just a run of luck. That could happen. But before each play, I even saw Mr. Shields figure out things on a piece of paper and always burn the paper with his cigarette lighter. That could have been just window dressing, eh? I beg your pardon. Never mind. Now, about that robbery. A very unusual thing to happen in Mr. Capo's establishment. Oh, I'm sure. It is our theory that poor Mr. Shields was merely in the way of the thief. And so if we should ever catch the thief... A rather doubtful possibility, eh? Frankly, yes. You see, our police force is not large. And there are so many different ways to get in and out of Polyzon. But naturally, if there is anything we can do to cooperate... Thanks. Thanks very much. I think I've had all the cooperation I can stand. And, uh, just one more thing. Yes? Should you come across the late Mr. Shields system? The one you're not even sure exists, Lieutenant Necastle? It would be more pleasant and more profitable for you, Mr. Valentine. If you were to get in touch with me first, good day. But you have me wrong, Mr. Valentine. Mr. Shields' death was a very unpleasant shock to me. Yeah, you must have cried all that night. No, I am serious. You must admit your grief does seem hard to believe, Mr. Capo. Caesar, dear lady, a much nobler name. Well, the fact remains, Caesar, that Shields won a great deal of money here at the casino. And presumably, he could win as much more as he wanted. However, that is just the point. Mr. Shields was a gentleman. The night he was killed, he came here to say goodbye. He apologized for winning the $10,000 at my expense. Said it was all he needed. And ventured the gas that I would survive. And as I see, you have quite well. Yes, this is a beautiful place, isn't it? You ought to see it in the evening when the players in full swing. It is truly worthy of that. Caesar, I know. But I still don't get what you're driving at about Shields. This is an honest casino, Mr. Valentine. I do not have to cheat. But obviously, I cannot survive a truly genuine system. Mr. Shields is dead, remember? I repeat, I had nothing to fear from him anymore, but whoever has this system now. Has someone else been very lucky around here lately? No, but I think that somebody will someday. Why? Needless to say, I watch Shields every move during the play. One evening, in the excitement of winning, he dropped one of the pieces of paper on which he was making his calculations. The paper was an unusual shade of yellow. Hmm, Shields' system was the talk of the town. How many people were trampled in the rush? As a matter of fact, only one other person noticed it. A shifty individual around town called Nico. I wasn't fast enough to stop him before he reached the door. The penalty I've overweighted. I'm sure you had a talk with the impetuous Nico later. I did, in private. But he said he couldn't make any sense of the figures on the paper, so he destroyed it. Oh, a likely story. Precisely. Especially since Nico has lately been seen spending more money than he ever had in his life. And more especially since Nico is now nowhere to be found. So you figure Nico sold the precious piece of yellow paper and was told to make himself scarce? Mr. Valentine, I will pay you handsomely. I must know if the systems that exist are no hazard. I already have a client, C.C. But you're welcome to tag along if you'll help me. Of course. All right, I want to know all the details of the robbery. How do I get them? I will arrange for you to speak to Clara Sebring, my manager, and Freddie Griggs, the cashier. Is this loyal Valentine? No, I merely think my employee is dramatizing himself again. Meaning what, Sebring? There's no such thing as a winning system. It's a tune we've heard before. It's usually followed by another tune. When you find it, remember me first. I've been a gambler for many years, Miss Brooks. Gambling's my whole life. And I never found a system. As you notice, I'm an employee of Caesar the Magnificent, who gambles not at all. The morals are obvious. So it is, and amen. But at the moment, I'd like to know about the robbery. Well, it took place just about here at this roulette wheel. The lights suddenly went out. Aren't there extra ones? An emergency lighting system? Most gambling places have one. The switch to the auxiliary lighting system is near the cashier's cage. When Freddie turned it on, he found it had been tampered with. Oh, that was convenient. By the time some other lights went on, the croupier was neatly knocked out. The money was scooped up from the racks, and in the excitement, Mr. Shields was shot with a gun, which must have had a silencer. The lights were tampered with, huh? An inside job, wouldn't you say, Sievering? I would say so, but Caesar shies away from the ideas if it were a reflection on his own honor. The men in Caesar's chosen can do no wrong, and that sort of thing. Again, I get the idea you're not very fond of Mr. Capo. Fond of him? Better than that, I admire him. His colossal ego. He actually has that phony royal crest on every handkerchief he owns on every cigarette he smokes. So obviously, the stolen money incident had to be put on a grandiose basis, including the Nico incident. Now, if you'd like to see the cashier. Yeah, lead on, Sievering. Yeah, I'd give ten years of my life for a system like that, Miss. Always knew somebody would be lucky and work one out. And the man who did wasn't very lucky, it seems, sir. Oh, yes, sir. I see what you mean. Freddie. Yes, sir? You can turn the emergency lights on right here near the cage, right? Right, oh. In fact, I tested them myself just before the casino opened. Mr. Capo was right with me. The old thing is a ruddy mystery. Yeah. But later, when the lights went out, didn't you do anything? Governor, I have a game leg, see? Got it in an accident. That's why I took this cashier job. So I wasn't going to run anywhere. Besides, it might have been a trick, so I do just that. Did you leave the cage? You got it, Miss. So I just stayed in here in the dark, sat here with a gun. A gun with a silencer? Silencer, sir? Why? No, just an idle question. Yes, sir. Uh, did you notice Nico here at the casino on the night Mr. Shields was murdered? Nico? You know him, don't you? Just enough not to give him credit. He hasn't been around in days and days. No idea where I could find him. Oh, no, sir. George, I think Lieutenant Necasters is looking for you. Oh, good. Maybe he can help us find Nico. The only lead we seem to have. Hello, Lieutenant. Mr. Valentine, I was very anxious to find you. Yeah, what's up? It seems we have had another murder. Committed by the same man who killed Mr. Shields. The bullets match. Who was killed? And that is the mystery. A very unimportant man. A man who cannot have any connection with this case at all. A man called Nico. In just a moment, we'll return to tonight's adventure of George Valentine. I'm sure that you've heard me say before that any gas can be made to stress one performance feature at the expense of others. But Chevron Supreme gasoline gives you not one, not two, but all eight high performance qualities in correct balance. And it's true. But the one real test for any gasoline is the way it performs in your car. That's why I also say that you'll never know real driving pleasure until you try a tank full of Chevron Supreme. Those eight high performance qualities make sure that Chevron Supreme performs best under all driving conditions. Quick starting, fast warm up, smooth acceleration, area blending, anti-knock, taper lock prevention, full power, and economy mileage. Those are the qualities you get in every gallon of Chevron Supreme.  Remember, Chevron Supreme can make your driving a pleasure instead of a chore. So next time you need gasoline, stop in and say fill her up with Chevron Supreme at any standard station or independent Chevron gas station where they say and mean we take better care of your car. Chance and probability. The heart of a winning gambling system that brought luck to nobody. Not to Bill Shields, the man who invented it. Not to the shifty little South American Niko who tried to cash in on it. Both were murdered. Yes, you're up to your neck in this very strange case that brought you thousands of miles to the little town of Polisarum. And even if your name is George Valentine, you begin to think that the chances for success are small and the probability of you getting a bullet next is great. Still, you stick to your job, which at the moment is again talking to the huge and egotistical owner of the casino, Cesar Capo. The murderer killed Niko so that he would be the only one who had the system. Now he am at his mercy. Oh, cut that out, Cesar. This place isn't worth a cent of a man can come in and win as much as he wants anytime. Two men are dead. That's a little more important than the money you might lose. I built up this casino from nothing. I built it beautiful and strong because I am Cesar Capo and I am not going to see destroyed. I am... I... Yeah? You're what? I don't know what I can do. Okay, then listen to me and try to help. Yes, sir. Who was here the night Shields was shot? Was Mary Marston here? The American girl? No. Professor LaRue? Let me think. Yes, he was here, but not that he had any money to play with, but he was here. And the lieutenant? The caster? Yeah. He was here, but the... One more thing, the emergency lights. You were with the cashier when he tested them. That is right. They worked. Have you any idea how they could have been tampered with? No, I would swear it would be impossible for anyone to do that without Freddie seeing him. Impossible. Impossible, but it was done. Yeah. The murderer doesn't have to come here by himself. Ever. Yes. Can have dozens of people come here. Use the system. Ruin me. Me, Caesar, Capo, they must... Freddie's address. Huh? What? Freddie Griggs, the cashier. What does he live? Carl Seabrig must know, but why I don't see... I want Freddie's address. And an oversized portion of good luck. It's overdue. Which one if Freddie decides to come back here? After all, this is his home and climbing in through a window, isn't it? He's having dinner, so stop worrying. Besides, what are we looking for? Not sure. Good. That always helps. I'm hoping it might be money. Lots of money. The money stolen from the casino? Yeah, it could be. Maybe we ought to believe, Caesar, that it was impossible for anyone to monkey with the lights. And it would have to be Freddie himself. Surely he wouldn't keep the money here. Couldn't very well take it to the bank. Still it's sort of in the dream department, isn't it? I guess so. Shirts, socks, handkerchief. What is it, George? A piece of yellow paper. A very unusual shade of yellow. The system. A piece of paper that Nico sneaked out of the casino. And on which Bill Shields figured his place. Let me see. The square root of C3D over A plus B equals the square root of C3D parenthesis A plus B over parenthesis A plus B times 2. Real fascinating. Well, it must be the key to the system. Uh-huh. But George, what's it doing here at Freddie's? If Freddie bought the piece of paper from Nico, then he must have killed him and Shields. That's what bothers me, Brooksy. How come this paper was left where it could be found? Freddie's too smart for that. Yes. Well, what do we do next? Come on, I'll get the lights. At least we can leave through the door. Well, the police can make Freddie talk. And if he knows anything about this piece of paper... Stop, Brooksy. Stay down. See what was? No, no, too dark. I just turned and moved. I'll find out. You never get him, George. Please don't. I guess you're right. The murder is getting more and more desperate, darling. Let's leave this case in the Castro. Not in your life. We still have this piece of paper. And a client who has a conscience and a heartache. Come on, Brooksy, we gotta see people. I just got here to the casino, Valentine. First one. Uh-huh. Caesar around, Sebring? Oh, no. He doesn't come down this early. He's probably home getting his daily manicure and shave. Freddie, he here? He should be here any minute. All right. Took a little walk after dinner, I did. Always like me constitutional. You don't like to be walking alone. You don't have a lot of people watching what you do. Yes, miss. I do like my solitude. Chance to do a little thinking. Meditate under the stars and let the bomb of the evening...  Nice to talk to you. So you were here in your apartment all evening, Caesar?  Have your nails manicured with a barber here? No, no, not tonight. I was here all alone. Even Caesar has his moments of loneliness. But it was terrible, Mr. Valentine. Calm down, Maro. Oh, I was just to call the police. Oh, Mary, poor Mary. Well, what about her? Makes sense, will you? I was in the other room teaching a group how to do the Charleston. And I heard Mary scream. I ran to her room and she was gone. Romance this way, isn't it? Yes, miss. Yes. And then one of the students ran over to the window. There is that big garden you cross before you get to the street. Yes, yes. Yes, we know. And the students saw a man dragging Mary across the garden. And she was struggling with him. You mean it couldn't see who the man was? A kidnapping. Oh, yes, yes. Yes, that is right. Well, the room sure is a mess. Mary put up quite a fight, all right? And I think I know who she fought with. George, who was it? Get Lieutenant N'Costrow Angel and meet me at Freddy's place as soon as you can. Everything is different, I swear to God. I was right about you, Sebring. Please stand there, the girl.  Move. Okay. The gun makes you a much bigger man than me. So you'll get no argument, Buster. He said it was going to make me tell him about the system. Even if he had to torture me. That's what I'm going to do. And then you'll have to go. With Valentine joining you later. The police will say Freddy killed not only Shields and Nico, but the two of you as well. After all, I already planted the paper. You're wasting your time, Sebring. Mary doesn't know the system. I don't really. I don't. I must contradict you. Shields was your friend, Miss Marston. Good enough friend to leave you all his money. You got him all those books. Oh, he confided in you all right. No, no, he didn't. When I get through with you, I'll know for sure. No. There's a matter of curiosity, Valentine. How did you know that I took the girl here? Well, if I'm going to satisfy your curiosity, we can at least be civilized and have a cigarette to get in. I can offer you a good American brand. Later. Start talking. Okay. Of all the people in this case, only you, whose passion is gambling, had no interest in the system. Insisted that you didn't believe in one. That was strange to say the least. Perhaps I did overplay my hand a little. Help me. Please help me, Mr. Valentine. Point two. After somebody took those shots at me, I heard him run. Certainly it couldn't have been Caesar, who waddles, or Freddie, who has a bad leg.  Point three. Freddie couldn't have dragged Mary across LaRue's garden. And Caesar, who only wants the system destroyed, would have killed Mary in a room. Not taking any chance of being seen. So again, it had to be you. Yes. Yes, it was me. All along it was me. But why? Why did you kill Bill? I took advantage of Freddie's rather obvious maneuver with the lights and his stealing the money to kill Shields and confuse the motor. But he never harmed you. I was sure I'd never get the system from him. But with him dead, I could solve the piece of paper I bought from Nico. Or as a last resort, get the system from you. And I will. Now. The paper was a fraud, wasn't it? A problem in algebra, that's all.  Cigarette? No. Then let me. And you can have this in the face of me. Ah! Cellophane burns even better than tobacco.  No, Mr. Valentine. No second chance, Buster. Except for me. Mr. Valentine. Oh, Mr. Valentine. Okay, Mary, okay, it's over. And I think I hear the lieutenant. As polite and vague as ever, I bet. And almost too late. With crisp fall days already making an appearance, the battery in your car will be getting more and more of a workout as winter approaches. So better take time now to be sure it's in good condition. Your car savers are equipped to offer complete battery service. They'll be glad to test your battery and, if necessary, give it a recharge. But don't wait until it's too late. Stop in soon for a battery check-up at any standard station or independent chevron gas station where they say and mean we take better care of your car. It's nice to be going home with you. You'll get well and strong again, Mary. Bill would want that, and you'll forget all this. Sure, sure. And the sunsets back home are pretty nice, too.  I won't even try to thank you, Mr. Valentine. Well, that's the smartest thing I heard anyone say today. George. Angel. I did find a system of weather as Lieutenant Nacastro said it was just a sensational run of luck. What do you think, Mary? I don't know. We'll never know. Now? That's right. Maybe that's best. Mr. Valentine. Caesar the resplendent. I have a present for you. A token of my gratitude. Cut things. The most beautiful, the most expensive, and made just for you so personal. Wow. Thanks. That's nice. See? Well, the crest. Of course, to let you remember that once you held Caesar. Goodbye. Bon voyage. Goodbye. Company of California on behalf of independent Chevron gas stations and standard stations throughout the West. Robert Bailey is starred as George with Virginia Greg as Brooksea. Let George do it was written by David Victor and directed by Kenneth Webb. Virginia Eiler was heard as Mary, Ted Corsia as Caesar, Harry Bartel as Carl, Eric Snowton as Freddie, and Larry Dobkin as Nacastro. Music by Gaylord Carter. Your announcer, John Heaston. Listen again next week, same time, same station too. Let George do it. Let George do it is heard overseas through the worldwide facilities of the Armed Forces Radio Service. This is the Mutual Don Lee Broadcasting System.