Hello everyone, I'm Kevan Kamali. This is the part three of the Deep Learning Tutorial series on convolutional neural networks. So as before, if you want to get familiar with Galaxy Platform, Introduction to Galaxy Analysis is a very good introductory tutorial. As for this specific CNN tutorial, I'm sure you've gone through part one and two of this Deep Learning series on feedforward neural networks and recursive neural networks. There is also an Introduction to Deep Learning, which is useful to review. So the questions we're going to answer in this tutorial is what is a convolutional neural network and what are some applications of CNN? And our objectives are understand the inspiration behind convolutional neural networks and learn the CNN architecture, learn the convolutional operation and its parameters, learn how to create a CNN using Galaxy's Deep Learning tools, and finally solve an image classification problem on MNIST digit classification data set using CNN in Galaxy. So the last two steps are part of the hands on part of this tutorial, which will be done separately. So what's a convolutional neural network or CNN? So there has been an increasing popularity of social media in the past decade or so. And as a result, image and video processing tasks have become very important. Previous neural network architectures, for example, feedforward neural networks, they could not really scale up to the image and video processing tasks in hand. So convolutional neural networks are specially tailored for image and video processing tasks. So this is a feedforward neural network. It has an input layer, an output layer, and a hidden layer. This is a fully connected network in that every node is connected to every other node in the next layer. Every connection has a weight that must be learned by the learning algorithm, the back propagation learning algorithm. So what's the limitation of a feedforward neural network? We're going to illustrate that with an example. Let's suppose we have a 64 pixel by 64 pixel grayscale image. So each grayscale pixel is represented by a value, which is usually between 0 to 255, whereas 0 represents black, 255 represents white. And any value in between represents different shades of gray. So since each grayscale pixel is represented by one value, we say the channel size is one. Channel size could be more. For example, it's 3 for color images. We'll get to that later. So an image is represented by 64 times 64 times 1 values, which is 4096. That's basically we multiply the number of rows by the number of columns by the channel size. So the input layer of a feedforward neural network then would have 4,096 nodes. Now let's assume the first hidden layer has 500 nodes. And our feedforward neural network is dense that is fully connected. So every node in the input layer, and there's 4,096 of them, is connected to all the nodes in the hidden layer, which is 500 of them. So the number of weights that we have to learn for just between the input layer and the first hidden layer is 4,096 multiplied by 500, which is over 2 million weights. So these are only the number of weights that we have to learn between the input layer and the first hidden layer. In complex problems, we usually have multiple hidden layers. So as you can see, this compounds the problem of having too many weights. So what is the problem of having too many weights? Well, first of all, the learning is more difficult because the search space's dimension is increased. So we're searching for the optimal weight combination in a search space that's huge. Also, the time and resources for doing such a learning task could be prohibitive. And finally, we're increasing by increasing the number of parameters. We're also increasing the likelihood of overfit because our data amount is fixed. So if we have a higher number of parameters with the same amount of data, we're more likely to overfit. So the problem is further compounded for color images. As I mentioned, for base scale images, we can represent the value of each pixel by one value. For color images, each pixel is represented by three values, red, green, and blue, or RGB color mode. So because each pixel is represented by three values, we say the channel size is three. And the image is represented as a 64 by 64 by three matrix. And we have 12,288 values, which is basically the number of rows multiplied by the number of columns, multiplied by the number of channels. So the number of weights is if we have a hidden layer, after the input layer, we would have 12,288 nodes. And let's suppose we have a hidden layer with 500 nodes similar to the base scale image network. Now the number of weights is 12,288 multiplied by 500, which is 6 million. If you go back, for base scale, it was 2 million weights. For color images, it's over 6 million weights. So the problem is even more, it's a bigger problem if we have a color image. So clearly, the feed for our neural net force cannot scale to larger images. So 64 by 64 is not a realistic image size. Usually images are bigger. So furthermore, we have more weights. So another problem with feed forward neural networks is that we represent a 2D image as a 1D vector in the input layer. So any spatial relationship in data is ignored. So a lot of times, like different parts of an image that are closer to each other, there's a relationship between what's in there. But if you represent this as a vector, you're basically losing all that spatial information. So we clearly need something better than feed forward neural networks for image processing. And that's what convolutional neural networks are. So how did convolutional neural networks came about? So in 1959, Hubble and Weizel did an experiment to understand how the visual cortex of brain processes visual information. So they recorded the activity of neurons in visual cortex of a cat while they were moving a bright line in front of the cat. They noticed that some cells fire when the bright line is shown at a particular angle or location. They called these simple cells. While other cells fired when the bright line was shown regardless of the angle and location. So these cells, they seem to detect a movement. They called these the complex cells. It looks as if complex cells receives input from multiple simple cells. And it looked like there was a hierarchy. So Hubble and Weizel won the Nobel Prize in 1981 for their experiment. Inspired by complex simple cells, Fukushima proposed neocognitron in 1980. It's a hierarchical neural network used for handwritten Japanese character recognition. It's the first convolutional neural network, if you will. And it had its own training algorithm. So in 1989, Locoon proposed CNN, which was trained by back propagation. So back propagation is the algorithm used for feedforward and recurrent neural networks. And now, 1989, Locoon proposed a CNN that was also trained by back propagation. That's great. Because all these different architectures are basically trained with the same algorithm. And Locoon won the Touring Award years later for his contribution to the field of neural networks. So CNNs became popular when they outperformed other models at the ImageNet Challenge. ImageNet Challenge, it's a competition in object classification and detection. There are hundreds of object categories and millions of images. And the competition has been running annually since 2010. So notable CNN architectures that won the ImageNet Challenge are AlexNet in 2012, ZFNet in 2013, GoogleNet and VGG in 2014, and ResNet in 2015. So AlexNet was the first CNN architecture that won the ImageNet. And it did significantly better than not only other models, but also the models from the previous years. So that started this convolutional neural network craze, and everyone started doing research and improving CNNs. So a typical CNN has four layers. There's an input layer, convolution layer, followed by pooling layer. And a fully connected layer at the end. So we may have multiple convolution layer, pooling layers, and sequence. And we'll get to that and show you an example. So in this tutorial, we'll explain a two-dimensional convolutional neural network. But we have to understand the same concept applies to one or three-dimensional convolutional neural network. So the input layer, let's say we have a 28 by 28 base scale image, unlike at feedforward neural networks, we do not flatten the input into a 1D vector. Inputters were presented to the network in 2D as a 28 by 28 matrix. This makes capturing spatial relationships easier. We don't lose any spatial information by converting a 2D matrix into a 1D vector. So that's good. We basically resolve one of the limitations of feedforward neural networks. So the convolution layer is composed of multiple filters or kernels as they're called. So filters for 2D image is also two-dimensional. And let's say we have a 3 by 3 filter, nine values in total. And values are randomly initialized to something, for example, between 0 and 1. There are different initiation techniques. So convolution is you place the 3 by 3 filter on the top left corner of an image. You multiply those nine values by the nine image pixels and add them up. And then you move the filter one pixel to the right and you repeat. And when you get to the top right corner, you start from the left-hand side and you only move the filter one pixel down. And you repeat this until you get to the bottom right corner of the image after which you're done. So let's explain this visually. So here we have a 4 by 4 image and the blue. And the dark blue is a 3 by 3 filter. So the 3 by 3 filter starts in the top left corner right now and then it moves to the right and then down and right. And then we're done. So when you go to the top left corner, the nine values of the filter are multiplied by the nine pixels that it is basically covering. And we add them up. So there are nine multiplications and then one addition and then we get one value. So as you can see, we repeat this four times and we get four values. So the green is the output of the filter being applied, a 3 by 3 filter being applied to a 4 by 4 image. The green, it's like, it's a 2 by 2 image. So the filter or the convolution operator has multiple parameters. There's five of them, the filter size, padding, sprite, dilation and activation function. And we're gonna go through all of them one by one and provide an example. So filter size can be 5 by 5, 3 by 3 and so on, larger filter size should be avoided because filters are basically weights. And if you have a large filter, you have more weights and as we know, we don't wanna have too many weights for all the problems. Increased learning time, increased computing capacity, neither for the learning and more likely it would have overfit. We should generally avoid even size filters. Odd size filters are preferred because they have a nice geometric property in that all the input pixels are around the output pixel. You can't have, you don't have this property for an even size filter. So odd size filters and small size filters are the preferred one, probably 3 by 3 would be a good example. So here in this example, we applied a 3 by 3 filter to a 4 by 4 image and we got a 2 by 2 image. So the size of the image was reduced. Sometimes that's what we want, but sometimes we wanna keep the size of image as it was, we wanna keep the size of the output image same as the size of the input image. That's why padding comes in. So if we want to keep the image size the same, we can use padding. We've had the input image in every direction with zeros before applying the filter. If padding is one by one, then we add a zero in every direction, up down, right, left. If the padding is two by two, then we add two zeros in every direction, up down, right, left. So here we have a five by five image in blue and we have a padding of one by one and we have a 3 by 3 filter and the result is a five by five output image. So the size of the input image or the size of the output image is the same, is five by five. Our filter size is three by three or padding is one by one. So stride is how many pixels we move the filter to the right or down? Stride of one, we move the filter one pixel to the right or one pixel down. Stride of two, we move the pixel, we move the filter two pixels to the right or down. So here's an example. This is a stride of two. So as you can see, we move the pixel two to the right and then two to the down. If you compare this to here, the filter, which you can see down here, moves one pixels to the right or down. Whereas here, it's like a jump of two. We could have stride of three and so on. So that's another parameter. As you can see, having a stride of more than one reduces the size of the output image. So next parameter is dilation. So when we apply a three by three filter, the output is affected by pixels in a three by three subset of the image. So dilation is when we want to have a larger receptive field. We want a larger portion of the image to affect the output. So if we set the dilation to two, instead of a contiguous three by three subset of the image, every other pixel of a five by five subset of the image affects the output. So let's see how that works. So here we have a three by three filter. So nine values affect the output pixel, but instead of these nine pixels in the input that affect the output being contiguous, they are, because the dilation is two, there's one gap between them. So as you can see, there's a gap here and here. So instead of the input receptive field being three by three because the dilation was two, now it's five by five, but not all the pixels in that five by five affect the output. So we've increased the receptive field. Sometimes that's desirable. That's another parameter we can play with. So the next parameter is the activation function.  So one of the preferred activation functions for convolutional neural networks is RELU. So RELU is very simple. If you have a negative value, it replaces it with zero. If you have a positive value, it leaves it as is. So here's one example. Let's say we applied a filter to an image and we got what's on the left-hand side. Now we apply RELU activation function. So anything negative is replaced by zero. Anything positive or zero, we leave as is. So let's say we are applying a three by three filter to a five by five image and the filter is being applied to the top left corner of this image. As you can see, there are nine values highlighted in blue in the top left corner of the image and the filter itself has nine values. And these nine values have to be multiplied by each other and then added up. And how it works is for the first row of the filter, you multiply zero by zero, one by two, and minus four by one. You repeat the same process for the second and third row. For example, in the third row, minus one is multiplied by one, four is multiplied by zero, three is multiplied by two. And if you add all these numbers up, you get a value of five. You get the value of one pixel. So if this filter scans the image, moves to the right, repeat, moves to the right, repeat, moves down and across until you get to the bottom right corner of the image, you're gonna get nine values as the output. So let's say, so this would be applicable if your image was grayscale and only your channel size was one. But if your image is color, then you have the channel size of three. Showing how, so if you have a channel size of three, your filter also has a channel size of three. So that would be three dimensional. And unlike this one, it cannot be displayed in 2D, in a 2D graph. So what we do is we have, we represent the three channels in the input as three separate matrices. And we represent the three channels in the filter, again, as three separate matrices. And this filter applied to top left corner of the image results in the value of five for channel one. This filter applied to the top left corner of the image results in the value of four for channel two. And this filter applied to the top left corner of the image results in the value of 10 for channel three. Then what we do is we add these three channels up And we get a value of 19. So this is a 3D representation of the same process. So just so you see how it works. And as you just saw, output of a multi-channel 2D filter is a single-channel 2D image. But nothing stops us from applying multiple filters. And for example, if the input image is 28 by 28 by 3, number of rows, number of columns, number of channels, and we apply a 3 by 3 filter with 1 by 1 padding, we get a 28 by 28 by 1 image. So we go from channel size 3 to channel size 1. However, if we have 15 filters, we're going to go to 28 by 28 by 15 output size. So the number of filters allows us to increase or decrease the channel size. That's very important. So the next component of a convolutional neural network is a pooling layer. Pooling layer performs down sampling to reduce the spatial dimensionality of the input. This decreases the number of parameters, reduces the learning time computation involved, and also reduces the likelihood of overfitting. So most popular type of pooling is max pooling, where you have a 2 by 2 filter with a stride of 2, and returns the maximum value as it slides over the input data. So this is an example of CNN. You have the input on the left-hand side. You have a convolutional layer with ReLU, multiple filters. And then you have max pooling. And then you repeat this three times. Eventually, you get to a fully connected network. And you have your outputs. Let's say you're trying to detect images 0 to 9. So you get a probability distribution on images 0 to 9. This one is obviously a 0, hopefully. We'll get the highest probability for the image 0. So typical convolutional neural network has several convolution plus pooling layers. And each is responsible for feature extraction at a different level of abstraction. So for example, the filters, the first layer detects lines, maybe horizontal, vertical, diagonal. The filters in the next layer detects shapes. It's a combination of lines. And the filters in the next layer detects a collection of shapes, like objects. So filter values are randomly initialized. They are learned by the learning algorithm. So CNN not only does classification, but it also does automatic feature extraction. And this is a very important feature. This distinguishes CNN from other classification techniques such as support vector machines. And this is part of a reason that CNNs are so popular. Because feature extraction is a difficult problem. If it can be automated and optimized by neural network and the classification is also done by neural network, then that would be great. So the last thing in a convolutional neural network is a fully connected layer. It's the last layer in a convolutional neural networks. All the nodes from the previous layers are connected to this fully connected layer. And it does the classification. Maybe we have a couple more connected layers. So the MS data, it's a data set of handwritten digits, 0 to 9, obviously, composed of a training set of size 60,000 and a test set of size 10,000 images. Digits have been size normalized and centered in a fixed size image, 28 by 28 pixels. It's gray scale. So each pixel is represented by a value between 0 to 25, as we discussed before. So MS data set is a standard image classification data set. It's used to compare various machine learning techniques. So that's a nice thing about having a standard data set because different techniques can be compared. So we define a CNN and train it using MS data set training data. The goal is to learn a model that, given an image of a digit, we can predict whether it's, we can predict the digits, whether it's 0, 1, all the way to 9. We then evaluate our trained CNN on a test data set and we plot the confusion matrix to evaluate our classifier. So references, as before, they are mentioned in the tutorial. The tutorial, you go to galaxytrainingattraining.galaxy.org. Go to statistics on machine learning and click on the deep learning part 3 convolutional neural networks. If you go all the way down at the bottom, actually there's a link here, you see all the references for this tutorial. And as before, the Galaxy training material can be found at this link, various topics, tutorials, contributors. For any help, please go to helpgalaxyproject.org or use the Gitter Chats. And for the events, you can go to galaxyproject.org and see what events are being advertised. So what's left is the hands-on section of this tutorial in which we implement a CNN using Galaxy's neural network tools and we train it using the MNIST data set to classify images. So that should be done shortly in a separate video. Thank you. Hello, everyone. We're going to be going over the hands-on section of the convolutional neural network tutorial. So as always, you can go to galaxytraining website at training.galaxyproject.org. Then you scroll down to statistics and machine learning and scroll down to deep learning part 3 convolutional neural network CNN. So if you click on the slides, we're going to do section 5 and 6 here. So we're going to go to get data section. We're going to upload the training and test set files to Galaxy. And what we need to do, so what I'm going to do is I am at usegalaxy.eu. I've logged in and I'm going to create a new history. Let's give it a meaningful name, GCC 2021 CNN. And then I am going to copy these training test files, the URLs from the tutorial. Then I click on the upload data. I click on paste fetch data and I paste them here and I click start and close. So this is going to take a few minutes to upload because these are relatively large files. So the jobs have started. So I pause the video while the files were being uploaded and may take a few minutes. So we have to rename all these files and make sure they are all of type tabular. So what we do is we get rid of the URL and the file name and the extension. We save it. Check the data type. It is tabular. So we do the same thing for the other three files. Check the data type. It's not tabular. So you click on this new type dropdown, type tabular, select it, and then click on change data type. And that takes care of that. So we start on the third one, the data type. Here it is tabular and the last one. And this needs to be converted to tabular. So change data type. So this basically takes care of the get data section of this tutorial. So I'm going to be following the steps in creating the deep learning model architecture and so on from the tutorial. So there are multiple steps. This steps creates a deep learning model architecture. This one creates a deep learning model. Then we do training and evaluation, model prediction, and the visualization. So these, I follow the steps from the tutorial. I have the tutorial open on a separate window. So I can look and make sure I do all the steps correctly. What you could do is you could download the workflow. If you go to the top of the tutorial, you can download the workflow. If you click here, you see the workflow, you can download it.  You just, what's it called, download it and then import it. So that would take care of everything. You can run the workflow on the data set that we just uploaded. But what I'm going to do is I'm going to basically do all the steps that the results in that workflow that's saved there. So let's go back. So I'm going to go back to the classification of MNES. So I'm basically following these instructions and the, in the use galaxy EU page. Okay. So first we need to do, it creates a deep learning model architecture. We have a set of tools here. We can search for them here. I just know them. I know that these tools are in machine learning under ML and statistics. So if you, statistics and visualization, sorry. So you see the header here, statistics and visualization on the left hand side. There's statistics, machine learning and graph display data. So the created deep learning model architecture is here. So I'm going to find it. So here it is. So it is going to be a sequential model. The input shape is a 784. This is because we're using the MNES dataset, which is images of digits zero to nine.  So 28 by 28 pixel, that's a 784 pixels. So that's the input shape. And what we're going to do is we're going to select a reshape layer here. And we're going to change the shape into 28 by 28 by one. So our input data, the image is represented as a vector, one dimensional vector. We're going to reshape it into 28 by 28 by one, which is the number of rows, number of columns and number of channels. Because this is a grayscale image, we only have our channel sizes one. One value between zero to 255 can represent a grayscale value with zero being black, 255 being white and anything in between with shades of gray. If the image was color image, a color image, then we had red, green and blue values that the combination of these three colors can represent any color. But we have a grayscale image here. So it's going to be a reshape layer and the target shape is 28, 28, one. We're going to insert another layer. It's going to be a convolutional layer and a Conf2D. So it's a 2D convolutional layer. We have a two dimensional image. The number of filters is 64. The kernel size is three. So we have 64 filters in this layer and filter sizes are three by three. The activation function, usually for CNNs, we pick ReLU and we're going to make sure that our image size is the same after the filters are applied. And the way you do that is you specify padding equals same. So we insert that layer two. Now the next layer is going to be a max pooling layer. So if we go to pooling max pooling 2D, that's what we have and it should be a size two by two. We're going to leave everything else as the default. Do another insert. Now we have, so usually in convolutional neural networks, we have a convolutional layer followed by a max pooling layer and we could have multiple of those. So we just added one of those convolutional layer followed by max pool layer. Now we're going to repeat another convolutional layer followed by a max pool layer. So here, we're going to have a 2D convolution again. Number of filters is 32, kernel size will pick three. And let me look at the tutorial notes and the activation function we pick ReLU again and that should be it. And the next one is again, every convolutional layer is followed by a max pool layer. So we go to pooling max pooling 2D and it's two by two. We're good. Another layer. So this is, we've already, I think added like five layers. So you, this is obviously not a big model but sometimes you could have dozens of layers hence the name deep learning, which means there are a lot of hidden layers in the neural network. So after the second max pooling, we're going to flatten everything. So we're going to use a core flatten.  And that's that. And the final layer, we have a core dense which is a fully connected layer. And because we're predicting images, sorry, digits between zero to nine, we need to have 10 units here. And each of those, we're going to use the softmax activation function. So the output of those 10 units, their probabilities over those digits. So the probability, the digit that has the highest probability is the prediction of the network. So I think we're good in terms of the architecture. I mean, the number of layers, number of nodes and all the other types, node types, activation types, et cetera of the network, we can click execute. So that should create the model config, which is you can download it as a JSON file if you need to. So the next step would be creating a deep learning model. And it's the same thing, you know, if you search, you could search for it. I just know where the location is. So I'm going to go down to statistics and visualization, machine learning. And we're going to go to create deep learning model with an optimizer, loss function and fit parameters. So we click here and I'm looking at the tutorial notes again. So the choose a building mode is a build a training model. The selected data set containing my model configurations in JSON, it's the output of the current job, which just started. So we leave that as is. We do classification or regression, we're doing classification. So we leave Keras GClassifier as is. The compile parameters, the loss function because this is a multi-class classification, we're going to be using a categorical cross entropy, not binary cross entropy, but that's for the case when our classification has only two classes. So let's find categorical cross entropy. The optimizer, the default should be Adam. We pick that and the metric, it's accuracy, that's fine. Number of epochs, we're going to set it to two. This is basically how many times we use the whole training set to train our model. So if it's two, that means we use every training example in the training set to train our model twice. And the batch size, we set it to 500. And that means, let's say our training example, our training set has a size of 60,000. We don't want to update the weights in our network after 60,000 training examples have been presented to the network. That's going to be too slow. So what we do is we update the weights in our network after batch size, which is much smaller than the training set size. So after 500 examples are presented to the network, we calculate the gradient or the derivative of the cost function relative to each weight. We average those over the 500 values that we got and then we update the weights. Sorry. Okay, there was a problem with running the Keras model config job. After we tried a few times, I think this was like probably the server was being maintained or something because this runs on the Galaxy EU server. It's pretty late, close to midnight, I assume over there. So anyway, I reran the job and it succeeded. So the next step in the tutorial is we just created architecture. We have to create a deep learning model and we go again to the tools pane, scroll down to statistics and visualization here, machine learning, if you scroll down, we can find the create deep learning model with an optimizer loss function and fit parameters. So I click here, we leave the choose a building mode as build a training model, the default for selected data set containing model configuration, the JSON file, that's the result of the previous step, which is pre-populated. We need a classifier because we're classifying MNIST images because it's a multi-class classification. We need to select a categorical cross entropy optimizer. We use Adam and we leave the metrics as is. Accuracy is fine. We set the number of epochs to two. That means how many times each training example the data set is used for training the data. So in this case, twice. The batch size, we set it to 500. So after 500 training examples are presented to the network, we calculate the derivative of the cost function relative to every weight. For all those 500 examples, we average them. So that way we know the step size for updating the weights and biases in the network. So I think this is ready. We're just gonna click execute and wait for this to complete. Okay, so the model builder succeeded as well. So if you look at the tutorial, the next step is a deeper in training and evaluation. Basically, we're gonna train our model and evaluate it using the training data. So again, in the tool pane, you scroll down to statistics and visualization under machine learning. We're gonna be looking for deep learning training and evaluation, which is right here. We leave, select the scheme as train and validate. Next, choose the data set containing pipeline estimator. Pipeline or estimator object is a result of previous step. That's job number 10, which completed right here. We'll leave it as is. Then we go select input type. Our data is tabular and we're gonna select our training examples here, X train and doesn't have a header. We want all columns and the labels are in the Y train file, which is here. It doesn't have a header. Again, we care about all the columns. So train and validate, tabular data, X train, all columns, Y train, all columns, I think we're good. So we're gonna execute this and wait for it to complete. Okay, so the training completed. We get three files as output. One is the result of the evaluation. The other one is the model and the third one is the weights of the model. So going back to the tutorial, the next step is model prediction. So we used some training data to train our model that now we're gonna use that model on the test data and see what the model predicts. So let's do that. That's going to be a model prediction. Again, it's the same thing on the left hand side. We scroll down to a statistics and visualization on the machine learning. We're gonna look for model prediction. That's model validation. This is model prediction. So first it says choose the dataset containing pipeline estimator object. That's pre-populated with the correct file. That's number 12, job number 12 right here. And the next is choose a dataset containing weights for the estimator of what that would be, number 13 weights trained on the dataset. So I'm gonna pick number 13 here. We're going to do prediction. So we leave this as predict. Our data is tabular. Our data is X test. Doesn't have a header. And we're interested in all columns. So we're gonna execute this and wait for it to complete. Okay, so model prediction completed. If you view it by clicking on this icon, you see that for the 10,000 test examples, we have predictions. So the first one is digit seven, second one is digit zero, digit one and so on. So going back to the tutorial, the next step is basically visualizing how our classifier performs. So we have to go to machine learning, visualization extension. So let's go there. Again, we scroll down to the machine learning statistics. Let's see if it's under graph display or, sorry, I have to look this up. So the machine learning visualization extension is under machine learning. So we're gonna click here, expand that. We do have machine learning visualization extension here. So selecting a plot type, we're gonna pick a confusion matrix. That's what's used for classification. Select data set containing true labels. This is for test data. So it would be Y test that has the labels. Data set doesn't have a header. And we're interested in all columns. And so select data set containing predicted labels. This is the result of the prediction, which would be a number 14. And so the correct one is pre-populated. Does the data set have header? It does. So I'm gonna change this to yes. And let's see, I think that's it. So plot title is confusion matrix between true and predicted labels. So we're gonna click on execute and let this finish. Okay, so our job completed. You can click on this icon to see the confusion matrix. So going back to the tutorial, we can see the confusion matrix as a table that describes the performance of a classification model. It basically lists the number of examples that were correctly classified by the model, true positive and true negatives. It also lists the number of examples that were classified as positive that were actually negative. We call these false positives or type one error. And the number of examples that were classified as negative that were actually positive. We call these false negatives or type two error. So what we do is usually given a confusion matrix, we can calculate precision and recall. So precision is a fraction of predicted positive that are actually positives. That's equal to true positives divided by two positives plus false positives. Recall is a fraction of two positives that are predicted. That's equal to true positive divided by true positive plus false negatives. So after we calculate precision and recall, one way to deal with just one number is to combine those two via F score, which is basically a harmonic mean of precision and recall. So this is a formula for harmonic mean of precision and recall. And if you look at this confusion matrix, the rows are the true class labels. So we have 967 plus two plus one plus eight plus two digits of zero and 1132 plus one plus two digits of one and so on. The columns are the predicted class labels. So column zero means that we predicted 967 zeros that were actually zeros. And same thing for column number one, I'm sorry, same thing for digit number one all the way to digit number nine. So the way, so true positives are these green diagonal points. These 967 zeros, true zeros were predicted as zeros. 1132 ones were predicted as ones. 1002 twos were predicted as twos and so on. But what about what's on the first row and first column? So these are the predicted classes, the columns. So these were predicted as zero, but they are not zero. 967 of them are, but four plus two plus eight plus seven plus two are not. So for example, these four were predicted as zero, whereas they're class two. These two were predicted as class zero where they're class five and so on. So these are the examples that were incorrectly predicted as zero, they're false positives. So if you add these values up to seven, eight, two and four, you get the false positives. So true positives is 967 false positives. If you add up these numbers and then false negatives are actual zeros that were not predicted as zero. And those would be all the other entries on the first row. So two plus one plus eight plus two zeros, true zeros were not predicted by our classifier. So that's our false negative. So we have true positive, false positive. True positive is the green circle here, 967 false positives. You add everything up in the first column, false negatives. You add everything in the first row and then you can plug them in into these formulas, calculate precision, recall and F score. And we can repeat the exact same thing for all the other digits. So you just have to do the second row and second column for precision and recall and F score for, you know, digit one all the way to digit nine. So as you can see our true positives are large, like 967 out of maybe less than a thousand. So that's pretty good. So that means this is actually a heat map. So the darker, the diagonal elements get better. And actually we're doing pretty well here. So in this tutorial, we explain the motivation for convolutional neural networks. They are really good for image processing and they have fewer parameters than feed forward neural networks. We explain their architecture.  It has certain parameters. And then we have some pooling layer, like filters or convolutional layer followed by pooling layer. And we can have multiple of those in sequence to do basically feature extraction, automatic feature extraction, which is very, very powerful. This is an excellent feature of neural networks. So finally, we use the Galaxy's neural network tools to solve a image classification problem using CNNs on MNIST dataset, which is a standard dataset for image classification. So this concludes part three of the neural networks tutorial. This hopefully will give you an introduction to various neural networks that are in use. And if any of you have any questions or if you want pointers to more advanced tutorials or topics, feel free to contact me. I also hope to expand on these tutorials in the future. One idea is to come up with a tutorial for autoencoders and variational autoencoders. Those are non-supervised machine learning techniques that via neural networks, and they have a lot of applications in different domains. Okay, thank you, take care and bye.


How's it going fam? Anthony Johnson here today from 21 Studios here at CPAC 2021. I'm with anti-communist documentary filmmaker and author Trevor Loudon. Trevor, how are you doing today? Doing great Anthony. Always a pleasure. Thanks for having me on. Absolutely. Yeah, we got to meet a few weeks ago at a local Republican event and I really liked your talk. I really appreciated it. Your savage. I was impressed to be frank. Yeah, I'm not here to muck around, mate. I'm here to lay it out. I don't come all this way to pussyfoot around. So for the fans of my channel who don't know you, can you tell them kind of what you're famous for and what you're known for with documentaries and everything happened with cancel culture? Yeah. Look, I've done several documentaries, I've written several books, I've pretty spoken to 500 audiences around the country and what my message is this is that America is the number one stumbling block to world socialism, world revolution and the world revolutionary movement which is now driven by the Chinese is very, very close to achieving their aims and they have done that by infiltrating US politics, they really control the White House now. They have done it by infiltrating the education system and a very big part of that has been to destroy the American family and destroying the American father, you know, even things like the Boy Scouts, you know. Having that upholds masculine values, the American family, patriotism, courage has all come under massive attack by the left and we've got to understand this is not just because people have a chip on their shoulder, this is an organized revolutionary program. Karl Marx said the big enemy to the communist state is family. If you can destroy family, you can communize everything and so that's why the American male boys and men are under attack by the courts, by our culture and the military. You name it, everywhere you look, men are under attack and we've got to understand that in the context that this is a revolutionary program to bring this country to its knees. Yeah, so with a very long term dark goal, what is that goal? Can you describe it a little more? Is it global governance, one world government, worldwide communism? What is it? Well, it's worldwide, worldwide communism, you know, and some people might characterize it as a sort of elitist, globalist movement. It is a world where very few run everything, where the average person is just a cog in the machine, has no dignity, no freedom, no prosperity.  That's the ideal world for the communists. You work, you provide for the state and when you're no longer useful, you just get turned into soil and green or whatever you get to fleece.  That is the ideal communist world. You know, we believe in a world where the individual is sacred, we believe in liberty, personal responsibility, that the family is the bedrock of our society and that every family needs a strong father and a strong mother and that pair, that team working together produces fantastic kids. That's our world and that's worked pretty well for a long time, right? Did it not build America? Did the American revolutionaries have to stop for, to find secure zones on their way to fight the British? Did the American soldiers who fought the revolution, were they unsure about their masculinity? Did they dress up in women's clothing when they went to fight the British?  That's what created this country. So in your view, the founding fathers and the founding generation were a great example of masculinity. Is that correct? Look, absolutely. You know, for thousands of years we've glorified the masculine virtues.  The ideal of the strong father who would protect his families, teach his boys to be a man, teach his daughters to be a lady, to be ladies. That was the ideal for thousands of thousands of years. It's only in the last 50 years when the radical left have got a very strong foothold in the judicial system and the education system that we've even begun to question those values. Like if I told the young man who stormed the beaches at Normandy that in 60 or 80 years their grandsons wouldn't be sure if they were men or women, that they'd be ashamed of their masculinity, that being a man was somehow toxic, they would not have believed you. They couldn't have begun to believe you. And so that has shown how much the radical left, how much damage they've done to the American family and the American male and American womanhood too, by the way. We've all suffered from this. Nobody's gained but the revolutionaries. But we need to understand this is not an accident. This is not a conspiracy. This is just a program by the left to transform society, to recreate it in their image, and they know the main impediment to that is a strong American family. So that's what they do. I'd say to clarify for audience, your view is that the war on men, family and masculinity is rooted in communism and Marxism. Look, absolutely. You know, who started the radical feminist movement? No, I'm not against women voting. I want my daughter to have every opportunity that's open to her. But the feminist movement was never about that. It was about dividing men from women, about breaking down families, making women ashamed to look after their families and cook a meal for their husband, you know? So smash the patriarch and smash the father. The patriarch is a good thing. I'm not saying that men should be tyrants, but everybody knows that a family is best when you've got a strong man who can protect his family, protect his wife, whose wife looks up to and has a wife that he respects and is very strong in herself. That's the ideal. But they have worked to weaken the man so women don't respect men and then they don't respect themselves. The kids grow up without role models. They don't know. Young men get into gangs and prison because they don't know what being a man is.  Well, there's screa. There's scream bad, too, that they're toxic and they're villains and they're demons. That's what they're taught today, right? And all they can do is go away and hide in their basement and play adventure games on the computer because at least then you go out and do some stuff. You do some stuff that men should be doing, you know? You're mentioning values and ideals here for masculinity and femininity and family, but it also feels to me growing up as a young American man, I'm 32 now, there's almost been like this massive rejection of common sense masculinity, common sense family values. So is it also a rejection of common sense, not just ideals? Yeah, well look, look, common sense, it means when you can look at reality and make accurate decisions, where you've got a common sense, where we have a common agreement on what real things are and what real values are and what achieves good results. It's common sense that kids do better. It's common sense that kids do better in an intact family. It's common sense that honesty is the best policy. It's common sense that you protect your family, that you should work, that you should teach your kids work ethic and honesty and moral values. That's all common sense because it's common people have accepted those values for a very, very long time. So if you want to transform society, you have to take what is common sense and turn it into toxic. You have to make it somehow bad.  How insane is that? All of those things benefit everybody, but the war on men and patriotism and all of the values that go with that have so intensified that what once would have seen completely common sense. It's common sense we have a border. It's common sense that families should stay together. It's common sense that we have two genders, right? Two genders, you know, that's all common sense. Thirty years ago, if I told you that we're going to be questioning things like there are two genders, that that would be a serious legal question and a serious philosophical question. You know, I thought I was a nutcase because that was common sense and it's not common sense now because you've got so many people driving an agenda to make that seem weird. You know, the lunatics are running society now. They are agenda driven. They are bitter and twisted. See, this is an interesting thing. Some of you may have heard of Tammy Bruce. She's a lesbian woman out in Los Angeles who was a radical feminist. Now she's a conservative, right? And I asked Tammy, when you're with the left, the radical feminists and whatever, what made them like they are? And she said every one of them was a damaged person. They'd come from a split up family. They'd been bullied. They were misfits in some way or another. Now we all have our own personality problems, right? None of us are perfect. Trauma, we all have difficulties, but we accept personal responsibility. We sort ourselves out and make ourselves better people. The left sees it differently. They want to make us like them. They are misfits, so we all have to become misfits. We believe in personal responsibility. We have faults. We sort them out. We have difficulties with our spouse. We sort them out like adults. We come to a resolution. The left believes in splitting, dividing and forcing people to adopt their values.  You look around society now. You look at the left. We have been forced into sickness like we have never seen before. And that's hurting our kids. And if we won't stand up to protect our kids, we don't deserve to be parents. We'll put one of my final questions. You're actually from New Zealand, not from America. I am. So I want to ask you, how does the state of masculinity outside of America, but in Western nations, Canada, Australia and New Zealand, what have you seen and is it similar to America, further ahead, a little bit behind? How is it over there? I would say it's a little bit ahead in my country. I'm talking 15 years ago when my boy was about four or five. I went and picked him up from his preschool one day and the teachers up right told me off because he had picked up a broom and was pretending it was a gun and was pretending to shoot his friends. And I said, what's the problem? And they said, well, you can't have violence. And I said, well, look, what do you do? How do you handle that? And they said, well, we get him to pretend the gun is something else. And I said, well, you mean like a bazooka or something? And I said, well, I'm going to walk out of this room with my boy and we're going to machine gun each other all the way across the car park to the car. I want my boy to be a man. I want him to learn to be a fighter, to stand up and protect himself, protect his family, protect his country. That's what I want. I admire that in a man. So look, every Western country is suffering from this. This is not just America. In fact, it's probably not as bad in America as it is in many other countries.   Well, America hasn't got that bad yet, but that's where it's hidden. Yeah, not if not over my dead body. Well, good for you. That's a pretty masculine thing to say. Thank you. And that's not toxic. That's great. I admire that.   This is wacko stuff, right?  I'm not a perfect father. I know I can see the mistakes I've made, but I love my kids, and I want to teach them the values that I was brought up with, because they were successful for me. And we see them all around us. We see what made America great. America was made great because you had brave men and women who came all the way over the ocean for religious liberty, to make a new life and a new land where they could live and breathe free. You got the best men and women of Europe to settle this country, and they built the most fantastic country the world has ever seen that has done more to protect other countries, to spread freedom, to protect people from tyrants than any other country. And now we've seen the radical left come and destroy all of it, if they can. We cannot allow that to happen. Our history, our present, our future, all of it, yeah? All of it, you know? They are erasing our past, because if you don't know your past, you don't know your virtues. You don't know what's right and what's wrong. We've got kids now who come out of school thinking America is toxic, America is evil, America was nothing but slavery, America was rich white men who got rich ripping off everybody else. They don't see that America was the first country in the world that made a country based on individual liberty, that created more prosperity than any other country, that lifted the world out of poverty, right? That lifted the world out of poverty, that stopped the communists, that stopped the Nazis, that stopped, that built churches, that built schools, that built missions, that built clinics and countries all over the world. This great benefit to the world and this is all at risk of being lost because we are being cowed down by radical leftist nutcases who try and force their views down our throat and have us doubting our own common sense. So my last question is circling, circle back, right? Circle back, girl. Let's circle back to feminism a little bit. Can you talk to me about the more direct connection between feminism and communism and how dangerous is feminism for women today? Like obviously they've had the right to vote in our country for a long time. They seem to be, you know, a lot of it then is aggressive though against men and fathers. So what's the danger to women and what's the connection between communism and feminism? Look, most of the leading feminists of the early 20th century were actually communist party members. You look at Gloria Steinem. She's a member of Democratic Socialist America. She's a Marxist. You look at a lot of the other famous right feminist writers and what they were all either Marxist or communist party members. You look at the founder of the gay rights movement, Harry Hay. He was a communist party member. You look at Black Lives Matter today too, right? Black Lives Matter is a pro-Chinese communist operation. It's run out of the Chinese consulate in San Francisco. And they have on their own website that they want to destroy the nuclear family. So they are revolutionaries.  To be treated fairly before the law. To be treated fairly before the law. All of that I support. But I don't support the feminist movement, which is essentially anti-male. It's pro-communist and it wants anti-father and wants to destroy the family because one of the number one planks in the communist manifesto was the abolition of the family. And so you've got to view feminism, the LGBTQ movements and the anti-male movement as all aspects of the communist movement. They are part and parcel of the communist movement. But we are so gulled by the media in this country we wouldn't recognize communists if they had a hammer and sickle tattooed on their forehead. You know? They're just progressives. They're just a little bit liberal. No, they are communist revolutionaries. Black Lives Matter is not a movement about the rights of black people. It's a movement about using and exploiting racial differences in this country to destroy this country. It is founded by 100% feminists. People seem to ignore this. Three lesbian feminists. Patrice Collures, Opal Tometi, Elysia Gaza, all lesbians, all radical communists, all communists, they have ties to China. They have ties to former East German communists, Vietnamese communists and Venezuelan communists. And you have people taking the knee to these people. You know, respecting these people, these people are revolutionaries. There's a great man, David Horowitz, who was a communist who became a conservative. And he would always say, when you look at these movements, the LGBTQ movement, the environmental movement, the anti-family movement, they don't believe in their own movements. He said, the issue is never the issue. The issue is always the revolution. The feminists don't believe in the feminist movement. Black Lives Matter doesn't believe in the black movement. The feminists don't even care about women, basically. No, they don't. They care about the revolution. They're just using women as canon fodder. Like a smokescreen. A smokescreen. Get the woman to hate the man. Get the woman to break up the family. And they'll achieve our revolutionary purpose. And we'll tell them, if they stay home and look after the kids and cook meals for their family, they're slaves. They're slaves to the patriarchy. They might be perfectly happy. They might love their husband. They might love their kids and feel very fulfilled. But they tell them they're slaves to the patriarchy. So let's have a divorce. Get the father out of things. Spend the rest of your life in misery.  Drowning a cat hair in cheap wine, yeah? Drowning a cat hair in cheap wine. It's a very good way of putting it. Look, there's a lot of women in this country who invested in the feminist movement of the 1960s. They never got married. And now they're looking at their friends who have husbands and children and grandchildren. And they are very, very bitter, lonely women.  My last question is going to be, tell me about your documentary. Was it banned on Amazon? I heard some that had millions of views. What's the story with that? I got a documentary out there, and I really do urge people to watch it. It's called Enemies Within. Now it's political. It's basically showing the Marxist and Islamist infiltration of the government, particularly the Democratic Party, not exclusively though, to control the Democratic Party. And so it was on Amazon Prime. It was seen three million times and they took it down two weeks before the election and won't put it up again. You can still buy the hard copy DVD on Amazon Prime though.  Exclusive to Amazon DVD? Look, I think it's on some other things like Zulu. I think it's on some other things as well. You can get Enemies Within, Trevor Loud and you look at it. You see Enemies Within the church looking at the massive Marxist infiltration of even the evangelical churches now. I have a lot of pastor friends you'd like to meet. Well, look, they're all about Black Lives Matter. They're all about the old Christian values, the old testament stuff, all of the old... This is the other thing. Christianity used to be a very masculine religion. They used to sing songs like Onward Christian Soldiers. Patriarchal, right? Now Christianity has been feminized. It's all about these pastors with their skinny jeans and their wispy beard saying we love you, doesn't matter what you do. Come to church. We're not going to talk about the old testament stuff or any of that. We just talk about the gospel of love. Well, that is the feminine love, which is okay, but you've got to have the masculine love too. What do you do in coming to church when you've beaten your wife? What do you do in coming to church when I saw you smoke and dope the other day on the corner? What do you do in coming to church when you behave like a sinner all week long and you come to church and you think that's okay? Sort your life out then come to church. That's masculine love. And that is love. That's missing from the churches these days. That calling to account. You know, any young boys being brought up by a mother who will love them like anything but she can't provide that discipline and that fear of God that a father can say. She over-caught them. They over-caught them and they do. They turn them into little girls. They can't say you wait till your father comes home sonny and your boys because they don't have that leverage anymore and that does not good for the boy and it's not good for the girl. It's not good for the mom. There are women out there who do a great job of raising the kids. Don't get me wrong. But I guarantee every one of those would acknowledge they would like to have a father in the house because that would make their job a lot better and they know would be better for their kids. Statistics back this up 100%. It's common sense. It is common sense and there's plenty of empirical evidence that kids do way better in a two parent family when the man, especially when the man is very clear on his identity is willing to be involved willing to discipline the kids where necessary and the mother and the father work in partnership where they have a clear role. You go into a family with a very weak father and a very domineering mother that's not very good either. It's unhealthy. No it's not healthy and a woman likes to respect her husband. She doesn't like to boss her husband around. This is radical talk Trevor we can't have this man. You've got to go to the feminist gulag. This wouldn't even be questioned 20 years ago. I'd be some wimpy mainstream guy 20 years ago. Now what I'm saying is you can even joke that it's sort of a bit radical. But it's not radical. It's just common sense. We've got to stand for common sense. We've got to acknowledge the values we've brought up with and realise they are the truth. They are good. And just give this feminist anti-male LGBTQ stuff no respect whatsoever. It's Marxist it's revolutionary. It's nothing to do with the rights of gay people. It's nothing to do with the rights of women. All it's designed to do is wreck the family so the revolution can unfold more easily. That when you understand that it makes a lot more sense what is happening. Trevor, I appreciate your time. Last question. Where can people find you?  Surprise surprise. So you've got to put TrevorLoudon.com in the search bar at the top of the page then you'll get it. They can get enemies within on Amazon. You can get it through my website too and you'll get it a lot quicker and you're not paying Amazon to basically wreck your life. So support local support patriots you can get my movies get my books on TrevorLoudon.com Thank you very much, Trevor. It's been an awesome interview. Yes sir, thank you. See you next time guys. 21 Studios been Anthony Dream Johnson. Adios.


The tax season is over, but it's interesting to note that the IRS recently released some guidance talking about how it plans to treat the taxation of transactions that take place using Bitcoin or other cryptocurrencies and electronic or digital media of exchange. Now, in this guidance, the IRS has basically announced that it intends to treat Bitcoin as property, not as currency, and hence there'll be capital gains type treatment accorded to Bitcoin transactions. So absent some sort of intervention by Congress, this is sort of the rule or the tax law as it pertains to cryptocurrencies at the moment. Now, this raises a whole host of interesting economics questions in the areas of barter and exchange and currency versus other forms of property.  Now, take the case of the farmer who produces a bushel of wheat and takes it down the road to another farmer who has mules for sale and exchange is the bushel of wheat for a mule because the farmer grows the wheat would rather have a mule than the additional bushel of wheat for his own personal consumption. Now, this is a pure barter transaction. It takes place without any money, yet both parties to the exchange feel like they're better off afterwards or we can assume sort of a priori that they wouldn't have entered into the transaction. Now, this is an awfully tough way to run an economy because of the concept of a lack of coincidence of wants. In other words, if the only farmer nearby doesn't have mules but has nails or gunpowder and that's something you don't particularly need, you have a lot of costs involved in trying to find someone who's got something to exchange with you. So of course, money arose from a pure market need and over the ages it developed and different kinds of money arose, oftentimes silver or gold. But nonetheless, money solves this problem of lack of coincidence of wants and gives us a way to specialize and conduct business. Now, when it comes to the tax angle or the IRS perspective, it's interesting to note that the treatment of a barter transaction where one farmer gives his a bushel of wheat for the other farmer's mule is treated the same as though this transaction had been conducted in cash. In other words, if the wheat growing farmer takes his wheat to an exchange and gets US dollars in cash for that wheat, he pays income tax on that cash, then he takes it and buys the mule with it. Now, the cash he uses to buy the mule may or may not be deductible as a business expense but nonetheless, all of this is firmly a taxable transaction from the perspective of the IRS. Well, with barter it's no different. There are specific rules in the tax code and in the tax regulations that apply to barter transactions and so this idea that when the farmer trades wheat for a mule, he's better off, the IRS says, well, that's an accession to wealth and hence you have to pay an income tax on that. And this comes from a very famous tax case in the 1950s known as the Glenshaw glass case. So the point here is that we cannot depend on technology somehow or different forms of trade to get us out of taxes because conceptually the idea of what constitutes taxable income in our society and frankly in most societies with most tax authorities around the world is any sort of increase in your personal wealth. So when we go back to the Bitcoin or cryptocurrency issues, this new IRS guidance creates an incredible burden in terms of compliance costs. Bitcoin or other cryptocurrencies will now be treated in effect like a capital good. So this means just like with stock, you have to track your tax basis. So if you or I go out and buy stock in Apple and we all wish we'd bought more, let's say we bought a certain block of stock in 2000, another block in 2005 and another block last year in 2013, well, it's all had different prices. We track our price and then when we sell the stock, we pay capital gains tax or have a capital loss based on the price difference between what we paid for it and what we later sell it for. Well, now the IRS is basically saying that Bitcoin and other currencies are going to be digital currencies are going to be treated in much the same manner. So that means that every time you acquire one of these, you're going to have to track your tax basis and every time you purchase something, in effect, that could be in the IRS perspective, a taxable event. And if that comes down to using crypto currencies for everyday transactions, buying gasoline, buying coffee, buying groceries, you don't use your app. You don't exchange Apple stock for groceries, but if Bitcoin is treated the same way, for example, it's going to create a tremendous compliance burden. So in conclusion, I would just say that as Austrians and as libertarians, we tend to encourage and champion any attempts to rest control of money away from governments, from politicians, from central bankers. We want to see money as a market commodity and a market phenomenon that has nothing to do with the state. But this recent IRS guidance is just an example. It's really just the tip of the iceberg of how those same governments and central bankers and political class see things like Bitcoin. So we all need to understand that technology alone can't save us from the state and its taxing. The only thing that can save us is the actual elimination of those taxing authorities and better ideas among the population about the proper role of government.